Friendly feed-back on our quasi-experimental evaluation

The German national centre for crime prevention (NZK) has published a synthesis of studies on efforts to prevent violent extremism among youth. Our 2015 study on the HEROES project in Berlin (Raab & Stuppert 2015 - unfortunately Wolfgang Stuppert is misspelled in the article) has been lauded for (1) our  definition of a highly relevant central evaluation question which the evaluation could actually answer with methods that we described transparently, and (2) the strengthening of the internal validity of our findings by the use of propensity score matching in quasi-experimental research.

Those who wish to read the full NZK text (in German only) can find it here. A summary of our study is available from this page of the Berlin Senate (in German, too); just download publication BFG Nr.57/2015.

Choosing appropriate evaluation methods

Barbara Befani, a reliable source of high quality information, has held a talk at the Centre for Development Impact on choosing appropriate evaluation methods. Listen to her presentation here; it comes with a slide show that is best watched while listening to the talk.

The spreadsheet which helps choosing appropriate evaluation tools is available from the BOND website here.  Enjoy! And contain your disappointment if you realise that most methods won't work with the weak data and skimpy resources available for your evaluation.

Evaluation Norms and Standards - now in six languages!

The United Nations Evaluation Group norms and standards for evaluation are now available in all UN languages - Arabic, Chinese, English, French, Russian and Spanish. 

Read them - spread them - use them! You can find them here.

In recent years I have been involved in large reviews of evaluations in the development sector, studying a couple of hundreds of evaluation reports. Many of the evaluations would have been vastly better if they had complied with, or at least been inspired, by evaluation norms and standards. I don't say that every evaluation must attain the quality of rigorous scientific research - in the opposite, time and resource constraints usually don't permit rigorous research in the context of evaluations, anyway. But you can still try to make it useful. 

At least, evaluators must make sure they respect the rights and safety of people who participate in the evaluation - i.e. ethics. Otherwise, the harm caused by an evaluation might outweigh its benefits.

A must-read for evalutors, campaigners and everyone: Beware of confirmation bias

Followers of this blog know that bias is a recurring topic on these virtual pages. Evaluators must be aware of bias to produce robust work. After this year's elections and the disturbing success of "alternative facts" and "post-factual" phenomena, the topic's importance has been acknowledged way beyond research circles, entering public debates.

Bias affects everyone's judgment - and that includes ourselves! A good way to deal with bias is to know about it, to seek out opinions that conflict with ours (i.e. mine, yours, everybody's), and to keep thinking.

A February 2017 issue of the New Yorker aptly summarises a few recent books on confirmation and "myside" bias. Find it here and read it - it may change your way of seeing things, and help you understand why others may sometimes seem terribly obtuse.

And if you like the idea of a song about the types of bias that may occasionally cloud your mind, have a look at this earlier post.

 

Getting those evaluation reports right

The other day I tried to remember how many evaluation reports I reviewed or used over the last ten years - probably a couple of hundreds. An astonishing number of reports felt like a a waste of time (and money). Anyone who is in the international development business knows how extremely diverse evaluation reports tend to look in this sector - from clear, concise and comprehensive to inintelligbile, interminable and still incomplete.

It is super-frustrating when - for example - you carry out a meta-analysis of evaluations and you have to discard half of the reports because of their poor quality.
 
Now, the great Better Evaluation site has published a checklist for evaluation reports, which lays down the characteristic of the ideal evaluation report. Find a link to download it here (a new page opens and there you'll click on "View Resource" to get the checklist). It is an annotated checklist with tips on how to write well. A wonderful resource. Use it and spread it to everyone who writes evaluations!

By the way - and I believe I have posted that earlier - Better Evaluation also offers guidance for managing evaluations and TOR writing. An important resource for those who commission evaluations. Because even the best-written report will be of little use if it answers the wrong questions, or if it can't answer the TOR questions because the resources at hand don't match the expectations. 

All that research for NOTHING?

A very accessible video by 3ie starts with evidence from a recent study by the World Bank. It has found that a third of the research reports published by the World Bank on its website have never been downloaded by anyone and 87% never referred to in any subsequent research. This is extremely wasteful. Remember, our tax money pays for such research.

So the big problem is maybe not that too few research reports and evaluations are published - the big problem is that people don't use them, because they don't look for such evidence, or they find the reports too hard to understand. The latter can be dealt with by writing in a more accessible style. Finding time to think and explore may be more complicated.

New Resource on Women, Land and Corruption

Transparency International has just published a guide which is the fruit of a three-year process I have had the privilege to support with my facilitation and writing. 

The guide builds on the experience of Transparency chapters in Ghana, Uganda and Zimbabwe, who have recognised that they can work more effectively on land and corruption issues if they involve more women and take into account the needs and interests of women and men. It is called Gender-Responsive Work on Land and Corruption: A Practical Guide; if you click on the title you'll be taken to the link where you can download it. (At the time of writing this post there is a typo under a caption; try to spot it before they correct it!)