Impact and evaluation of public engagement

Some outcomes from the workshop on impacts and evaluation.

• There's a lot to learn from reviewing evaluations already done (meta-evaluations). Unfortunately, too often evaluations are done as a check-marking and this needs to be avoided. It's important to take up the lessons learned; RCUK is doing a bit of this by reviewing evaluations.
• Categorising types of impacts and benefits is a critical step in evaluating public engagement.
• There are then stories and matrices that can be developed based on the type of impact being explored.
• Experience shows that quantitative indicators can be hard to find, difficult (and expensive) to measure; qualitative indicators have value. One of the greatest challenges with them, though, is making qualitative evaluations systematic; how to make generalisations and themes emerge from storytelling and case studies etc.

Top tip: Be clear on why you're doing evaluation (while recognising that motivations may change over time)

• How to select measures which reinforce or triangulate other measures or data - need to be creative, but this is worth thinking through.
• Important drivers for impact evaluation are RCUK and other funding councils.
• Need to find ways to ensure it fits your own purposes too.
• Need to be creative in finding ways to record.
• An important starting point for making use of evaluations is a desire to improve your practice. Ultimately, evaluations (at least formative ones, not after activity is over) should inform your practice.
• Evaluations must be simple and not a difficult add-on; efficiencies should be a cornerstone. Participatory methodologies are worth exploring.
• Self-reflection is also a valid tool in evaluation; how did doing something change you and your thinking on an issue, a methodology, etc.? This might be as valuable as any impact evaluation directed to recipients of your activity. For example, if your practice or insight changes as a result of doing something.
• Seeing and quantifying impacts is extremely difficult/unlikely/not realistic with most short-term projects and research. Too much time needs to pass, too many attributing other factors.
• Choosing the right time to evaluate is also key; to get valid feedback (i.e. choosing to gather responses immediately following an event, vs a week or a month or more later).
• Making meaning of results can be challenging; support in understanding the process of drawing conclusions would be helpful.
• Despite usually messy and less than rigorous methods, evaluations that seek trends or generalities are still important. A basic sense of costs vs. benefits of doing something (vs. not doing it) can be really important.
• Evaluating inputs and processes can be really helpful, and more realistic sometimes than looking for long-term impact.
• There's lots of expertise in evaluation out there, just as there are in public engagement. Researchers can't be expected to become experts in this area and so need to have help to connect to appropriate resources and people. This doesn't have to be expensive ongoing evaluation support, just resources, possibly a workshop, etc.
• Language used in evaluations is important - can isolate others.