How do we measure progress in collective impact, when most measures are lag indicators?

Perspectives from the field: A conversation about collective impact and collaboration from Australia and Canada

Over the last month, Liz Weaver, Vice President, Tamarack Institute and I have been engaged in an email exchange puzzling out the answers to many wicked questions about collective impact and community change. Below is part II of V, where we discuss how to ensure a community agenda not just a shared agenda. You can view Part I here.

How can we measure progress when so many of the measures or indicators are lagging indicators?

MAX HARDY:

This is a really important question to consider, especially in the early days of an initiative. I’m a big believer in the power of appreciative inquiry; and the saying ‘What you focus on will grow.’ The question then becomes more specific; what kind of behaviours and practices would we like to see more of?

We need to be wary of indicators that measure a range of things. For instance, simply measuring, ‘What percentage of meetings does a certain organisation or individual attend?’ will give you some data, but not information. If someone attends very regularly it may not necessarily tell you whether they are committed. It could mean they are there because they don’t trust others at the table and they want to see what is going on, for instance.

So, we need to pay attention to, and yes, measure, more than simplistic data. Perhaps it would pay to measure things like, ‘What additional information is being shared to help our collective endeavours?’ ‘How much are we all learning from each other?’ ‘To what extent are we following through with commitments we each make in between meetings?’ ‘How much do we enjoy, or at least look forward to, our gatherings?’ ‘To what extent are we confident we are working with our community of interest rather than just designing another way of delivering services?’

These questions are not always that easy to measure, but they might be easier than people think. For instance, noticing the body language as people arrive, and how willingly people stay longer to have a one on one conversation, may provide really useful information about the level of enthusiasm.

Reviewing progress will be much richer if we talk about how much we are learning from each other and sharing with each other, than simply counting bums on seats. If we focus on some of these things we are likely to measure our progress in a more meaningful way; and if we focus on these things they are likely to grow.

How about you Liz?

LIZ WEAVER:

Lagging indicators are data points that are a few years old. This type of measure often is generated through a national survey. In Canada, every five years there is a national survey that Statistics Canada conducts which looks at several population indicators. The data is then analyzed and released a few years or more later. This data is helpful, and can be useful to determine population trends over time. But, the lag time is an important factor to consider. If your initiative is only a three-year initiative, national statistics data may not be carried out with the type of regularity that is required and therefore may not be relevant to your effort.

Some collective impact efforts also consider leading indicators. A leading indicator is a measurable economic factor that changes before the economy starts to follow a pattern or trend. Leading indicators are used to predict changes in the economy, but they are not always accurate.

In the case of leading and lagging indicators, the group must determine the relevancy of the indicator to the context of the initiative. Sometimes there just aren’t the right set of indicators in place and that is when the evaluation questions that Max suggests may be more helpful to determine how the initiative is moving forward. For shorter term efforts, focus on a small set of indicators but also ask questions to determine how behaviour is changing by key stakeholders over time. These short-term behaviour changes have the potential, if supported and encouraged, to lead to major impacts over time.

Do you agree with their perspectives? Do you have other questions they should consider? What are the challenges you are facing as you engage in collective impact? We invite you to add your comments and join the conversation here.

Click here to view the publication.