Making MEL easier

Making MEL easier

by

Monitoring Evaluation and Learning, often called MEL for short, is the practice of collecting and analysing data on a project or program. MEL enables project progress, impacts and performance to be measured and lessons to be generated for improvement of future work.

So many organisations with which we collaborate or provide support in Community Works struggle to achieve an effective approach to MEL. It is either something they feel ill-equipped to do, a reporting burden or a reluctant add-on to projects that are already underway. Many teams rarely appear that they are comfortably and competently managing MEL as a key part of their work.

For MEL to be truly integrated to project work, this situation needs to change. Part of the problem is that MEL is a professional discipline all of its own and one that is akin to research. But most program staff do not have a research background. When educators or health workers or community development practitioners take up a position with an organisation, they are often not acquainted with MEL because it has not been part of their formal training or even their previous experience. Faced with the complex task of defining indicators to measure, collecting data relevant to each and then analysing the data, it is not surprising that they appear overwhelmed. This is especially true where organisations do not already have their own data collection processes and systems.

So, what can be done? Here are suggestions that draw on our experience at Community Works:

Bring in community voices

Most staff working in community or non-government organisations, or for directly for government agencies, are accustomed to speaking regularly with the users of services or with participants in projects. Part of their job is to build trust and rapport with people in the community.

To make MEL easier, these strengths ought to be harnessed. In other words, some of the data collected should bring the perspectives of the community into the data by asking community members what their experiences have been since the project started. A small number of well-designed interviews and focus groups can provide high-quality data and rich insights into projects and programs.

 

Choose indicators with care

Some changes are complex to measure and may require extensive work to develop indicators and then work out how to collect meaningful data. Examples are community resilience or effective governance.

Rather than trying to design a framework for MEL that encompasses the full complexity of a subject, it may be valid to choose aspects that are more measureable and to focus on those. In the case of resilience, a measureable component is the level of confidence that people in the community feel about the steps they will take when a future emergency occurs. For governance, a key element could be the development of the skills of the chair and other members of a community board.

Use Theories of Change

A Theory of Change (or ToC) describes the key cause-effect relationships that underlie a development intervention. The discipline of focussing on the basic rationale for a project or program can help us decide with greater clarity exactly which data we need to collect.

Working with project teams, I have frequently concluded that more time devoted to discussing a ToC and less time spent on brainstorming indicators will lead to a better plan for MEL. Developing a ToC may be yet another discipline for an overstretched project worker to learn, but the benefits justify the effort. The discipline of developing a ToC can be valuable for anything from a single project activity, like a women’s  group or a learning activity at a drop-in centre, to a major program scheduled over several years.

Less can be more

A few times recently I have been asked to review logical frameworks that represent the designs of projects or programs. Sometimes the column that includes indicators is very long. In other words, there are too many indicators. The impression is that the project team has thought of every single measure that could be applied but nobody has then culled the list to one that is manageable. One to three indicators per activity is usually enough for a logical framework. For a project or program as a whole, then between four and eight indicators ought to suffice.

 We must always keep in mind that practising MEL costs time and money. Focussing on a smaller number of indicators is more efficient and less overwhelming for a team to manage than trying to apply a comprehensive set of measures that cover every aspect of the project. There is little to be gained from adding more indicators if the existing ones already enable effective MEL.

Direction of travel

For positive changes that can take years to realise, such as changes in attitudes and beliefs, it is unrealistic to expect a project of one, three or five years in duration to achieve those changes. In these situations, MEL should therefore concentrate on indicators that represent positive steps towards those longer-term goals. We call this the direction of travel. For example, for an individual or a group of people to show greater awareness of the health effects of smoking is an important step towards a change of attitude towards smoking.

Coupled with use of a Theory of Change, being clear about the direction of travel is valuable in reducing unrealistic ambitions of MEL, making it less daunting and more effective for even small organisations with limited resources to use effectively.

Rather than MEL being a burden for many teams and therefore done either poorly or not at all, it can be made easier. As a result, the value of doing it becomes apparent again. With data on the difference that projects and programs make, we are in a better position to know whether they are actually working. And we also learn why that might be the case and how to improve our work in the future.

‘We are not researchers’ – Making M&E accessible

‘We are not researchers’ – Making M&E accessible

by

‘We are not researchers’

Making M&E accessible

Steve Fisher

The increasing need for evidence of the impact of investments in social development, health and education programs places pressures on organisations and their staff. This is not unreasonable, but for many people, monitoring and evaluation remains an obscure subject, distant from the day-to-day activities of working with clients.

Community Works often runs introductory training workshops on monitoring and evaluation for health workers, NGO board members or groups of development professionals seeking to strengthen their work in this area. Sometimes participants make clear their difficulty with the subject. ‘We are not researchers’ they say. ‘Çollecting and analysing data is not something we have done before’. Or, they sometimes imply, is that the reason we became health or community workers in the first place.

We have learned from these experiences. As a result, we try to develop materials and deliver training sessions that follow four key principles:

Modify language

The phrase ‘impact assessment’ often throws people off course straight away. We have found that ‘measuring change’ is a good way to talk about monitoring and evaluation because it leads to a conversation about what are the specific changes a project or program intends to achieve and how can we best measure those changes. In social development, usually the answer is to ask people good questions. The same applies to other terms, such as data (equals information).

 

Desmystify the subject

Monitoring and evaluation is research. But that doesn’t mean the subject has to be clouded by research process jargon. For example, data analysis can be broken down into a process of grouping responses to interviews or surveys into key messages, trends and ideas. We often suggest that teams measuring change present all their data on the wall or on a big table, so everything can be seen together. That makes it easier to spot key insights and other information.

Focus on the process

When an electrician or a plumber comes to my house to repair or install something, the technical language they use is often a barrier to me understanding what they are going to do. We have addressed language above. But not knowing the process also leaves me clueless about their work. The same applies to monitoring and evaluation. In supporting better practice, one of the most useful ways we have found to support people is to set out the steps required. We sometimes present the process visually, as a flights of stairs for example. Once people can see the steps, the whole process seems more manageable.

Build on the strengths of participants

Often, qualitative information is best collected through, for example, interviews and focus groups. It sounds obvious, but the backgrounds of certain professional or community groups can be very good preparation for this kind of work. For example, health workers are often skilled in putting people at ease and asking questions in a supportive and encouraging way. This means they are often very effective, once given the chance to practice and guidance on the way to facilitate a focus group, for example.

In our training programs, we like to introduce practice early on. We find that participants who might have approached M&E with indifference or fear, become animated and interested when they see how effective they can be in collecting information through talking to people.

Minyerri; what a community arts centre is for

Minyerri; what a community arts centre is for

by

We have been working with the community of Minyerri recently through a project managed by Enterprise Learning Projects (ELP) through The Smith Family. The work is part of the Australian Government-funded Communities for Children program.

I visited the community in August during a period when the Arts Centre was being revitalised and relaunched, so it was an exciting time as the building was repaired and improved. A beautiful sign was created by the women and children at the Centre (see the photo).

At this time, the activities of the centre were also being planned, which is why I was there. My job was to help the people running the Arts Centre and ELP work out a way of planning and measuring the results of the work they would be doing at the Centre into the future. We call this a Monitoring and Evaluation Framework, which is a grand-sounding name for a table with three columns and which follows the logic below:

Discussing this framework with the women at the Arts Centre led to one important question; what is the Minyerri Arrts Centre for? There are many answers to this question. As we talked about it, the conversation produced a set of outcomes that define what they are trying to achieve. Outcomes include sharing of knowledge between older and younger generations in the community, developing new skills, building self-esteem of children and young people and strengthening connections that they have with their history and culture.

These are early days but the Arts Centre is becoming a place for many activities to take place. Drawing and painting are part of the plan, especially as after-school activities. One lady is making cushions there. Minyerri is also the home of Gulbarn Tea and so the collecting and packaging of tea also takes place there. More information on Gulbarn can be found here: https://gulbarn.com.au.

The official opening day was 1st October. The children formed their own mini catering and events organisation, doing all the promotional material and distribution around Minyerri. They prepared wraps and popcorn and sold food and art to the community. It fitted in with one of the outcomes in the framework: ‘The community, especially parents, will know about the good things that kids can do at the Art Centre, that will help them grow up well’.

I will be visiting Minyerri again in February, this time to help work out ways to collect information on the outcomes being achieved. No doubt we will review the framework in the light of experience so far at the Arts Centre. I am looking forward to being there again with Minyerri people and the ELP team.

Countering violent extremism through social resilience

Countering violent extremism through social resilience

by

It is September 11th, and I am steeped in literature relating to counter-terrorism. Community Works has recently engaged in a new project examining ways of building social resilience to prevent violent extremism, and, importantly, how to approach evaluation and impact assessment in this rapidly emerging area.

War and Peace by Jason Edmiston

A shift to prevention

Governments around the world are struggling to figure out how to prevent the disillusionment of the post-911 landscape from breeding future terrorist activity. The importance of research into countering violent extremism (CVE) has gained traction in recent years as increasing evidence demonstrates that hardline approaches to counter-terrorism can actually exacerbate the threat. At the same time, there is a surge in recognition that preventing new recruits from resorting to violent extremism requires a drastically different skillset from conventional approaches to counter-terrorism that focus on detection and interception of attacks.

Early approaches to CVE focused on broad attempts to ‘counter the narrative’ of radical ideologies, but it is now understood that ideologies are only a small part of the problem. More and more, analysts and policy-makers are recognising the need for social approaches rooted in local contexts. Growing emphasis is being placed on finding ways to build social resilience among individuals and communities to resist the pull of violent extremism.

Context, contact, and complexity

It occurred to me today that, like the broader field itself, my own approach to studying CVE is rooted strongly in the history of what happened on this day fourteen years ago. At that time, I was living abroad in Denmark as a volunteer at a residential school for at-risk youth. One of our students, Habib, had recently arrived from Afghanistan. He was almost fifteen years old, spoke little Danish and even less English. His father had vehemently opposed the Taliban, and their attempts to punish his family had driven them to flee. Soon after I first saw those planes slam into the skyline of a city I love, in the country where I grew up, Habib came to me and apologised with great sincerity. My response was astonishment: ‘No Habib! What happened today has nothing to do with you!’

But of course it did, because my government’s response was to launch an attack on Habib’s country. When he heard the news, he collapsed in tears and repeated adamantly ‘If they stop the Taliban then it will be worth it.’ This was no light statement: aside from his parents and siblings, Habib’s whole family, his entire history was in Kandahar, one of the first cities to be bombed. Whereas I had been able to pick up the phone and hear almost immediately that all my loved ones were safe, it would be months before Habib would learn the fate of his family. Fourteen years later, there are far too many young people facing similar situations.

The nuances that unfolded in this exchange with Habib have made it impossible for me to think of the ongoing wars in the Middle East as a matter of ‘us’ and ‘them’. Although our countries were at war, it was plainly clear that he and I were not enemies and no amount of propaganda could convince me otherwise. The psych-social processes at play here fascinated me, and my research in the years that followed explored how interaction between people from different cultural backgrounds relates to peacebuilding. The social psychology of peacebuilding now plays a central role in my investigation of innovative methods for countering violent extremism.

The demand for research on CVE

Community Works has recently conducted a review of the literature surrounding CVE. A strong theme to emerge from the review is the demand for more rigorous studies, especially to develop effective methodologies for social approaches to countering violent extremism at the local level. Specifically, the literature points to the need for:

  • Collaborative, transdisciplinary approaches – most studies of CVE have arisen from security-focused fields; there is a strong need for input from a range of perspectives including psychology, sociology, education and public health;
  • Focus on stakeholder ownership and empowerment – there is strong theoretical support for the importance of empowering communities through CVE initiatives, but very few programs employ participatory approaches;
  • Studies that focus on women and girls – the vast majority of research on CVE has focused on men and boys, with very little consideration of the strong potential role of women and girls
  • Innovative methods for impact assessment – although governments are increasingly investing in social approaches to CVE, there remains no clear idea of how to gauge which programs are achieving impact;

In partnership with LaTrobe University, Community Works is currently developing a longer-term research project with the intention of filling these crucial gaps. The project will draw heavily from key theoretical frameworks emerging from the social sciences, such as intergroup contact theory, moral inclusion, and social cohesion. The central aim of the project will be to assist a local CVE initiative in developing ethical and effective methods for measuring and leveraging their impact.

Source: Holmer, G (2013), ‘Countering Violent Extremism: A Peacebuilding Perspective’, United States Institute of Peace Special Report.

Learning development practice

Learning development practice

by

The current design of the Masters program on Management in Development

The current design of the Masters program on Management in Development is in its third year and is being coordinated by Steve Fisher. The focus of the program is effective development practice and it is offered as five one-day participatory workshops on aspects of the subject:

It is always interesting to observe which elements of the program appear most engaging and relevant to students. Sometimes they can be surprising, such as the preparation of a proposal for an imaginary Men’s Shed project in Carlton. Using the outline below, four groups each wrote a section of the proposal.

We then read the whole document out as a single description of the need, design implementation and intended outputs and outcomes. This was the final part of a day of learning about the subject of communicating on projects, especially to attract resources for them. It was a strangely uplifting experience. I compiled the final version into a two-page concept paper.

Another part of the program that always proves interesting is on innovation, focussing on four aspects of the subject:

  • Conceptual frameworks (new ways to think). Example: Social inclusion
  • Process improvements (new practice). Example: Participatory impact diagrams/outcome mapping
  • Technical change (new technology). Example: Use of mobiles for development purposes
  • Organisational change (new ways to organise). Example: Social enterprise and social franchising.

Following success with a marketplace approach to learning in an event for over 100 participants on water and sanitation in Melbourne last year, we used the same method for this work. An advantage was that it enabled students to move around between subjects across a one-hour period, learning from and contributing to each. The end results were impressive and I will feature them in a future blog on the subject of innovation.

My appreciation to the excellent and dynamic student group of twenty-four people, representing ten countries. Many thanks also to Maria Rodrigues for her research support and to Ingrid Horton for design work on the program.