Come and see me @evalfacil.eu

My blog has moved to my website evalfacil.eu. Come and look up my latest posts under the "blog" tab over there, at evalfacil.eu!

Moving to evalfacil.eu

In its 13th year (can you believe it ?!), this blog moves to my new website, on a dedicated blog page. Over the coming months, I will review the content on developblog.org and move some favourites to the new virtual space. And then, in a year or two or three, I will close down developblog.org, this very page you're on right now.

I hesitate to praise Blogger/Google but I can't help being grateful for such a low-threshold space to publish content, keep a (now defunct) library of links to interesting sites, share information on my activities and so forth. It has been fun to check the analytics, realising that my posts have attracted readers from all world regions, from Iceland to the Pacific Islands! 

There are much better-resourced online libraries on monitoring and evaluation now. But I will continue writing, on my new blog, because blogging is an excellent way to organise one's thoughts. Please do pay a visit to evalfacil blog and if you have followed me here, start following me there! There will also be an option to share comments.

Many thanks for reading me!

 

Diskussionsforum: Evaluation und Wissenschaftlichkeit (in German)

Am 17. September ab ca. 13:45 veranstalten wir - Bernward Causemann, Ines Freier und ich - ein Diskussionsforum bei der 24. Jahrestagung der Deutschen Gesellschaft für Evaluation (DeGEval). Als erfahrene, mit-allen-Wassern-gewaschene und mit wissenschaftlichen Methoden vertraute Evaluationsgutachter:innen werden wir das Spannungsfeld beleuchten, das sich zwischen der Standardisierung und Normierung von Evaluation (im Sinne einer auf diese Weise verstandenen Wissenschaftlichkeit) und den Ansprüchen an die Nützlichkeit von Evaluation aufgebaut hat, und vertiefte Diskussionen zu diesem Thema moderieren.

Auch dieses Jahr findet die Tagung online statt; es besteht die Möglichkeit, sich für einzelne Tage anzumelden. Unser Diskussionsforum ist Teil der Sitzung D7 (Blitzvortragssession und Diskussionsforum). Mehr dazu auf der Konferenz-Website. Wir freuen uns auf eine interessierte Teilnehmerschaft! 

Finally! Thoughtful guidance on applying the DAC criteria

Long-awaited new guidance on applying the evaluation criteria defined by the Development Assistance Committee of the Organisation for Economic Cooperation and DevelopmentD) (OECD-DAC) is finally available in this publication! Long-awaited, because evaluators and development practitioners have grown desperate with assignments that are expected to gauge every single project against every single OECD-DAC criterion, regardless of the project's nature, and of the moment & resources of the evaluation. This new, gently worded document is a weapon evaluators can use to defend their quest for focus and depth in evaluation.

Those who commission evaluations, please go straight to page 24, which states very clearly: "The criteria are not intended to be applied in a standard, fixed way for every intervention or used in a tickbox fashion. Indeed the criteria should be carefully interpreted or understood in relation to the intervention being evaluated. This encourages flexibility and adaptation of the criteria to each individual evaluation. It should be clarified which specific concepts in the criteria will be drawn upon in the evaluation and why."

On page 28, you will find a whole section titles Choosing which criteria to use which makes it clear that evaluations should focus on the OEC-DAC criteria that make sense in the view of the needs and possibilities of the specific project, and for the evaluation process. It provides a wonderful one-question heuristic: "If we could ask only one question about this intervention, what would it be?" And it reminds readers that some questions are better answered by using other means, such as research projects or a facilitated learning process. The availability of data and resources - including time - for the evaluation helps determine which evaluation criteria to apply, and which not. Page 32 reminds us of the necessity to use a gender lens, with a handy checklist-like table on page 33 (better late than never).

About half of the publication is dedicated to defining the six evaluation criteria - relevance, coherence, effectiveness, efficiency, impact, and sustainability - with plenty of examples. This is also extremely helpful. Each chapter comes with a table that summarises common challenges related to each criteri on - and what evaluators and evaluation managers can do to overcome them. It also shows very clearly that lack of preparation on the evaluation management side makes it very hard for evaluators to do a decent job - see for example table 4.3 (p.55) on assessing effectiveness. 

The document is a bit ambiguous on some questions: The chapter on efficiency still defines efficiency as the conversion of inputs (...) into outputs (...) in the most cost-effective way possible, as compared to feasible alternatives in the context" (p.58), which makes it extremely hard to assess the efficiency of, say, a project that supports litigation in international courts - interventions that may take decades to yield the desired result. However, the guidance document states that resources should be understood in the broadest sense and include full economic costs. On that basis, one can indeed argue, as Jasmin Rocha and I have on Zenda Ofir's blog, that non-monetary costs, hidden costs and the cost of inaction must be taken into account. Yet, table 4.4 on efficiency-related challenges remains vague (p.61). Has anyone read the reference quoted in the table (Palenberg 2011)? I did and found it very cautious in its conclusion. My impression is that in many cases, evaluators of development interventions are not in a position to assess efficiency in any meaningful manner.

On the whole, I would describe the new OECD-DAC publication as a big step forward. I warmly recommend it to anyone who designs, manages or commissions evaluations.

 

Für Deutschsprachige: Online Moderieren - Lebendig und Produktiv

Mein online-Workshop zu online-Workshops ist am 26.Mai online! Mehr Informationen und Anmeldungsmöglichkeiten gibt es beim PME-Campus. 

Apologies to those who don't speak German - my first workshop on online facilitation will be in German. But if it works out nicely, I might offer sequels in English and in French! A bon entendre, Michaela

Join my workshop on online facilitation in PME

After a year of lockdown-induced life in cyberspace, web-based workshops have become a routine in planning, monitoring and evaluation (PME). Workshops are about exchange, about developing something together. But often, I have witnessed online workshops that were so virtual you hardly noticed the participants. Seemingly endless pages of screen-shared text being read out, word by word, in a soothing voice. No breaks. Confusion about technical refinements, links posted to inaccessible clouds. The loneliness of the person who finds herself alone in the main channel, without any pre-assigned breakout group... 

But there are also online workshops that actually work, that engage us in an exhilarating process, and that produce results. They can be more efficient than real-life workshops. And they surely save enormous amounts of CO2 and travel costs. Even though many miss the informal encounters at the coffee machine, over lunch, in the bathroom line (we're in 2021 and people identified as male still enjoy better acess to toilets than the rest of us) -  I suspect that online workshops are here to stay. 

It takes deliberate planning, adaptive pacing and plenty of participation to make an online workshop work. After more than a year of developing, facilitating and documenting a range of workshops on different platforms, I am distilling key insights into a short workshop for German speakers - online, of course! Not the technical stuff - the providers' video tutorials take care of that - but key principles and ways to apply them. I'd be delighted to meet you. More details in German) are available here. 

Gender equality in organisations: a good resolution for 2021

Gender equality is a key element of sustainable development – as illustrated in the Sustainable Development Goals (SDGs), which weave gender across virtually all 17 SDGs. It makes sense that 'mainstream' organisations, which are not specialised in promoting gender equality, have developed gender policies and related activities. Where are they at, and what should come next?

More handy tips for videoconferences

An addendum to yesterday's post - ICA:UK, a reliable source of materials and training on highly participatory facilitation, has summarised 10 principles to prevent online fatigue. I've been using all of them. They work. 

At any rate, avoid text-filled slide shows with voices droning on in the background! Visual aids are great, but if you just show pages and pages of text that you read to your audience, they'll end up muting you and joining a different event on their other computer. I admit that was what I did in a recent conference full of half-hour text-rich presentations by invisible voices. I couldn't help it. Since the different conference followed the same mode, I still felt I would have been better off reading an article, in my own time, at my own (rather energetic) pace.

Easy socialising in tight video conferences

How to recreate a sense of a "real life" team event in a video conference? In real life (IRL, as nerds put it), people usually linger near the coffee/tea kitchen or in the hallway for a quick chat - one reason why it tends to be so hard to get participants back from "real" breakout rooms. 

"Random" virtual breakout rooms - if they don't come with too burdensome assignments - can recreate this atmosphere.

Like most facilitators I know, I have facilitated more video conferences in 2020 than ever before. I have discovered that participants tend to hijack virtual breakout rooms: Before getting started on the small group assignment, they'd have an informal chat on totally different subjects. Or, in other cases, they'd get the assignment done as fast as possible so as to spend the rest of the small group chat on their own agendas.

Know what you need to know

Evaluations often come with terms of reference (TOR) that discourage even the most intrepid evaluator. A frequent issue are long lists of evaluation questions that oscillate between the broadest interrogations – e.g. “what difference has the project made in people’s lives” – to very specific aspects, e.g. “what was the percentage of women participating in training sessions”. Sometimes I wonder whether such TOR actually state what people really want to find out.

I remember the first evaluation I commissioned, back in the last quarter of the 20th century. I asked my colleague how to write TOR. She said, “Just take the TOR from some other project and add questions that you find important”. I picked up the first evaluation TOR I came across, found all the questions interesting and added lots, which I felt showed that I was smart and interested in the project. Then I shared the TOR in our team and others followed suit, asking plenty more interesting questions.

I wonder whether this type of process is still being used. Typically, at the end, you have a long list of “nice to know”-questions that'll make it very hard to focus on questions that are crucial for the project.  

I know I have written about this before. I can’t stop writing about it. It is very rare that I come across TOR with evaluation questions that appear to describe accurately what people really want and need to find out. 

If, as someone who commissions the evaluation, you are not sure which questions matter most, ask those involved in the project. It is very useful to ask them, anyway, even if you think you know the most important questions. If you need more support, invite the evaluator to review the questions in the inception phase – with you and all other stakeholders in the evaluation – and be open to major modifications.

But please, keep the list of evaluation questions short and clear. Don’t worry about what exactly the evaluator will need to ask or look for to answer your questions. It is the evaluator’s job to develop indicators, questionnaires, interview guides and so forth. She’ll work with you and others to identify or develop appropriate instruments for the specific context of the evaluation. (The case is somewhat different in organisations that attempt to gather a set of data against standardised indicators across many evaluations - but even then, they can be focused and parsimonious to make sure they get high quality information and not just  ticked-off boxes.) 

Even just one or two evaluation questions is a perfectly fine amount. Anything more than ten can get confusing. And put in some time for a proper inception phase when the evaluation specialists will work with you on designing the evaluation. Build in joint reflection loops. You’ll get so much more out of your evaluation.

My first hackathon #EvalHack

The International Programme for Development Evaluation Training (IPDET) has moved its 2020 summer school on-line. As an extra, it has opened up its online evaluation hackathon to external people - like me. I have taken up the invitation, registering for my first evaluation hackathon. Our group challenge was to develop a prototype for efficiency measurement. This is what we have come up with: Evaluation United. If you like it, scroll to the bottom of that web page and LIKE it; we're in a competition! 

Five tips for remote facilitation

Despite the risks and uncertainties associated with independent consulting, I have never felt as privileged as I do now, living in a country with a highly developed, accessible health system, working from my customary home office, and equipped with a decent internet connection and the hardware needed to stay in touch with friends and colleagues. The crisis has been an opportunity to develop my remote facilitation skills. Before, I facilitated the occasional "real-life" workshop in a video conference room with participants in other locations joining us via Skype or the like. I have shared that type of hybrid experience on the Gender and Evaluation community pages. Now I have gone one step further, facilitating fully remote workshops from my home office. I mean interactive workshops with some 5-20 people producing a plan, a strategic review or other joint piece of work together - not webinars or explanatory videos with hundreds of people huddling around a lecturer who dominates the session. To my delight, virtual facilitation has worked out beautifully in the workshops I have run so far. Good preparation is a key element - as in any workshop. I have distilled a few tips from my recent experience and from the participants' feedback.

Facilitating with care in lock-down situations

In the current situation of (near-) lock-downs, interviews, meetings and group discussions are moving on-line. And so are workshops, with tutorials and useful advice on running on-line workshops popping up all over the place. If, as a facilitator, you want to make sure all workshop participants make the best of a virtual workshop, the challenges of moving on-line are not just technical. When everybody has to work from home, your workshop enters the participants' households. To express it in terms of systems thinking, your workshop is not confined within the borders of office spaces and working hours anymore; it intersects with a number of household systems:

International Evaluation in Times of the New Coronavirus

What does the surge of SARS-CoV-2 (the scientific name of the new coronavirus) infections in parts of Europe mean for international evaluation? Can we, as evaluators, join the soothing voices of those who say, the current common flu epidemic has killed many more people and there is no reason to change anything in our lives? I don't think so. I would like to remind all of us of the Do No Harm principle: Research ethics require us to carefully weigh the potential benefits of undertaking research (at a given time) against the potential harm associated with it. We can relax about ourselves but we must not endanger others. International evaluations can also be done without international travel.

Proud to be Rosa Marina Flores Cruz's mentor

Rosa Marina is the third mentee I present on these pages. Although our mentorship has formally ended, we're still in touch. Rosa has kept me up-to-date on her engagement for indigenous people's rights around the world - most recently at the Indigenous Women in Community Leadership gathering in Antigonish, Canada (November 2019). As with the previous posts, I am presenting Rosa in a written interview. 

(1) What are the main issues that you are currently working on?

I am an Afro-indigenous activist and researcher from the Isthmus of Tehuantepec in the State of Oaxaca, Mexico. I work on topics like rural feminism, environment and energy, and the autonomy and rights of indigenous peoples. I have worked in training projects for community health and human rights promoters, and for the defence of the Right to Free and Informed Prior Consultation. 

Photograph by Shirley Kimmayong