Over the past few months, we’ve been thinking a lot about the benefits and challenges of evaluating the impact of services and activities on loneliness in older age. We’ve had the privilege of talking to a wide range of organisations – from small befriending schemes to large housing associations – about impact measurement at two recent workshops, based on our new guidance: Measuring Your Impact on Loneliness in Later Life. 

What are the benefits of using a loneliness scale?

Loneliness impact measurement is not easy. So we’ve written this blog to reflect on some of the things we’ve heard over the past two months. This includes the benefits and drawbacks of using a scale, and suggests some solutions from services about how to deal with some of the challenges that can arise during monitoring and evaluation of loneliness ‘interventions’.

The 70+ attendees at our two workshops were asked to share why they were interested in using a scale to measure the impact of their services on loneliness. Scales were felt to:

  • Produce consistent and comparable results
  • Help to clearly demonstrate the difference you’re making, by collecting information at the “start” and “finish” of a service
  • Show what is working and what is not working, and make changes
  • Help make the case to funders – particularly those who like quantitative research techniques in evaluation and recognised tools or scales
  • By protecting anonymity, encourage people to take part in an evaluation

What are the drawbacks?

We also thought about some of the main problems that can arise when using a quantitative measure of loneliness. These included:

  • A scale only gives you a snapshot of someone’s emotions and experiences – and might not represent how they feel overall
  • Lacking depth and “rich” data – closed questions cannot uncover the reasons underpinning someone’s responses or loneliness
  • Needing to train people asking the questionnaires, to minimise bias or leading respondents
  • Interpreting the results properly – without care and/or expertise they can be misleading
  • Finding it difficult or lacking time to ask the questions and analyse the results
  • People might find the questions intrusive or distressing
  • Different funders have different tools and targets that they want to be recorded
  • People may be reluctant to answer honestly – for fear service being taken away
  • A scale (particularly a longer one) may be too intrusive for lower level support services

Finally, a lot of attendees pointed out that their services are delivered by volunteers and staff who may not want to do the evaluation, and questionnaires can be perceived as “onerous”.

Overcoming challenges

We asked attendees to think about how to overcome some of the challenges and drawbacks of using a loneliness scale. This blog cannot full represent the discussion that went on – but here are a few thoughts shared.

Problem: scales only provide a ‘snapshot’

  • Ask the questions regularly, e.g. when you first meet someone, then at 2 or 3 monthly intervals
  • Use both qualitative (e.g. open-ended questions or observation) and quantitative research techniques in your evaluation
  • Ask people more in-depth questions after they’ve been attending your service for a while , and then use case studies to complement the ‘snapshot’ data
  • Remember that you’re measuring a general “direction of travel” and cannot be 100% correct all of the time

Problem: people might find questions intrusive or distressing

  • Provide training to the people inviting people to take part in the evaluation or asking questions directly – get them to practice with each other and talk about what to do if someone looks or becomes upset
  • Make sure your staff have a blurb that explains why the questions are being asked – e.g. required by the funder – and that people can stop answering the questions at any point
  • Explain that the evaluation results will benefit other people in the future, if they provide feedback
  • Make sure there is support immediately after the questionnaire – for example, make sure the interviewer or someone else is available to talk through any concerns or emotions raised

Problem: applying the scale regularly is difficult and time-consuming

  • Embed the evaluation into project delivery – for example, finding opportunities to ask evaluation questions when you’re having your first meeting with someone or make it part of existing activities
  • Support your staff to manage their time and talk openly about workload pressures
  • “Streamline” different evaluation tasks so that activities are not duplicated, too onerous or take too long
  • If have resources, ask respondents to give answers to the questionnaire on a tablet or computer so that the results can be analysed quicker later without additional data inputting