THE DOS AND DON’TS OF SOCIAL IMPACT MEASUREMENT #1
INTERVIEW WITH DATA MANAGEMENT EXPERT MEGAN O’NEIL-RENAUD
Originally published on WhatAMission.com on 10th August 2017 - 9 min read
As social entrepreneur, sooner or later you come across the question how to measure your impact. Social impact measurement is not only important to demonstrate the effectiveness and impact of your practice, it also helps to attract funding and motivate staff and volunteers. Our online editor Vanessa Faloye interviewed Megan O’Neil-Renaud, manager of Social Enterprise and Social Finance at Pillar Nonprofit, on that important topic. Here you find the first part of the interview.
Something that I think is being seen a lot in the field of social innovation or a conversation that is being had a lot more now is about measuring impact and how we’re doing that. From your perspective, what are the shifts in trends of impact measurement?
Well I would say there’s a big push for using impact measurement towards systems change. Sobefore was all about bums in seats (and I still preach this all the time)… so you get a grant, they want to know how many people showed up, how many times did you run your programme etc. The government is a little slower but there’s that shift that says well how do we interpret that data or are we even measuring the right thing? I love big data, I like looking at numbers and then use those numbers for something, not just recording. Not just ‘government says I have to check off that 40 people came, I did two presentations, and that’s it. That’s meaningless. So I see that push to make meaning out of it.
How do we measure the reach and impact of social media and multimedia (for example television, radio etc) in terms of usefulness and not just engagement?
For example doing a radio programme or social media campaign. So not only how many people clicked and engaged with the content but its usefulness for those people.
So, for me, if I was measuring the impact I want to see, just like a business, how it translates into sales. I want to see an increase in sales, I want to see that I do a campaign and I get an x percentage increase in people coming to my product. For example, I have a friend that’s an inventor that invented a craft beer tap that measures and then reports via an app how much beer at exactly what time beer comes out of that. So they can measure to the minute when their super bowl ad launches, how that translates to sales. So that’s how they’re capturing data, we need to do that. We need to capture a campaign in terms of revenue because that is ultimately how our social enterprise is going to survive.
And if it’s more for a charity organisation and they’re doing a campaign that focussed on behavioural change for example? Here the metrics aren’t sales so what would you say about that?
This is where it gets harder. That’s really hard to measure. It would be the conversation, the dialogue that starts. If you’re looking for behavioural change you need the dialogue about the change first. So it wouldn’t just be a retweet, but rather you would see other people talking about it. That’s big campaigning where you need specialised hashtags or something to identify that the dialogue has started.
How can we measure preventative social impact? Services that prevent things like crime or drug use, again perhaps through campaigning or ads. How do you measure the prevention?
That’s again really hard because correlation is not causation so I actually encourage every agency to take that correlation piece and run with it a little bit. So when you’re starting a dialogue and suddenly crime goes down, I think that if we the people in this ecosystem have to say ‘hey wait a minute crime was at 25% and now it’s at 22% and we’ve been campaigning all year, is it possible we’ve had something to do with that?’ Without such a big research project like multi-year studies – nobody has the resources for that. So it’s really keeping an eye on where you’re at. Know where you start. So if you’re going into a campaign to change mental health outcomes or to change participation in the mental health conversation, know where you’re starting from by doing your research.
I can tell from your language that you’re saying ‘maybe it was us’ and you’re not making any overarching grand statements that lack evidence. I think for social entrepreneurs who are trying to write grant applications this is something to watch out for.
When you’re writing a grant application, you’re projecting this impact down the road and then you can track this if your grant is 3 or 5 years. At the end of 3 years you can say, we noticed this trend in general terms. It’s all about creating that dialogue around the impact, lots of people will say ‘well how do you know?’ You don’t. But you can ask the publicly ask the question ‘could it be us?’ You don’t know that it was, but you also don’t know that it wasn’t.
So in that sense it’s very important to have an almost 360 panoramic view of what is going on your community ecosystem or space?
Yes you have to keep talking to the other players in your ecosystem – even if you’re working in food, work with the anti-poverty people, work with mental health, work with community living. Just keep talking to all these people so you know what’s going on. You never know what it might lead to, at least a great collaboration. It’s important to support each other and then you know where you started.
Where can you begin to start calculating the effect your project is having on an overarching system like healthcare for example? How can you find statistics to quantify the prevention your project is having?
That is huge. That’s where you need academic institutions. So you’re best bet is to count the minimal steps that you’re doing and then do a literature review. Go into a library at a university, get access to their databases. Not a blog search and not Google. Some Google Scholar is ok. And do a literature review. Take a look at all these other papers, at least 15 showing how for example a homeless initiative will improve overall health of a community. You don’t need to reinvent the wheel, that’s why government pays these researchers. Run with the research. Particularly government research, use their own data for them. There’s tons of papers out there and so much work already done on it. Then take the incremental pieces and map it from your project onto this literature review. For example, I read these 5 papers and they said that if the community takes steps ABCD then the outcome will be XYZ.
How do you measure long-term behavioural change in a short-term way?
I would measure coming in. So if you’re running a workshop or an educational programme or anything that is typically getting behavior change, measure people coming in before you say a word to them or even before they show up to the workshop. And even if you have forgotten in the beginning, ask them: what were you like before you heard about this and give a ranking from 1-5. How do you feel your knowledge was on a scale of 1-5? And then how do you feel your knowledge is now and what can we do to help? And then at the end of your project or near the end, ask them: how do you feel now? What was most beneficial? What will you do going forward? Will you share this knowledge? So, for example, someone who has gained knowledge who plans to share it has typically changed behaviour. So just track it. And don’t forget to track years down the road, so ask them if you can contact them in three years. Follow up and see what they’re doing.
What have been the common challenges, mistakes, or more creative ways that you have seen entrepreneurs measuring impact?
The common mistake is not measuring it at all. I find that it astonishing that people don’t even measure. So social entrepreneurs who have a small product that feeds back into a programme, but they’re not doing any impact measuring. For example, a battery regeneration company that had no impact measures – I mean this business is saving tens of thousands of kilograms of lead and sulphuric acid in the environment and not to mention third world countries and child/maternal health and all those these other pieces that were never thought of. The entrepreneur just regenerated batteries and kept them out of the landfill. So get into it, ask why, be a critical thinker. What does this mean…? What does that piece mean…? What does it mean when we grow gardens that people get to eat from? Well what does that mean for the environment…? What does it mean in terms of what the government is going to like to hear: like waste water runoff or heat sinks or greening the downtown or tourism or dollars so that we can translate a garden into dollars for government. Really unpack it and keep asking why until you can’t ask why anymore. And go back to your mission, your vision and your goals. What is the goal of this project and is the impact related to this goal. If it isn’t part of your mission then you’re measuring the wrong thing. So always go back to your goal and to your mission and how does this impact measurement relate. Why does it relate, where does it relate? What piece of it relates?
Of course I coach people to look at bums in seats but I really want people to look deeper.
Megan O’Neil-Renaud is the manager of Social Enterprise and Social Finance at Pillar Nonprofit. She helps social enterprises move beyond idea stage and into launch then sustainability. She leads data management forthe social enterprise and finance clusters while working on the front-line with some pretty incredible social entrepreneurs. You can get in touch with her via LinkedIn.