Quick evaluation: What a difference a couple of days make!

You know those evaluations that come with dozens of questions on a whole complicated international development programme (or even set of programmes), to be answered within one week of desk work, one week on-site and one week to tie it all up? They are still around and they are not about to disappear. I used to hate them: The time frame makes it near-impossible to draw reasonably rigorous samples of respondents (for interviews or for a survey) and there is so little you can do and see in a week at the programme site. What can an outsider find out in one week that an insider doesn't know yet? After having worked on a couple of "quick evaluations" in recent months, I have adopted a milder stance. They can generate useful insights. But how? Here are a few tips.

This is written from an evaluator's perspective, but I address those who commission evaluations, too: 

• Cluster and prioritise the questions and draw a sample of projects, in dialogue with the client. This may sound banal - but anyone who has tried knows how hard it can be to get people to shed some questions. Even with a clear sense of priority, you won't get "scientific evidence" within that time frame, but the fewer questions you work on, the deeper you can go. Gathering data against, say, 30 evaluation questions on 40 projects run by 30 organisations would mean asking thousands of questions, also because it is not enough to gather data from just one source per project. But you can fit only about ten questions into an hour's semi-structured interview or a couple of dozens into a ten-to-fifteen-minute standardised questionnaire. 
• Work in tandems! Well-informed external opinions are even more valuable if they come from more than one person. In two of "my" recent quick evaluations, I - the evaluation specialist - was paired up with a regional specialist. We did some work together, some in parallel, and discussed our findings every day - the result has been so much richer than what a single consultant can produce in so little time. If the budget does not allow for joint field work, you can still organise a desk-based peer review which would mean just an extra couple of days' honoraria.
• Put in three more days on site. Whatever the original "quick" time frame is, three more days make a huge difference. You need a day for inception on-site and at least a day, better two, to analyse initial findings and discuss them with the people who run the programme. That will take two to three days off your week - adding just three days to the overall exercise will double the time for data gathering and analysis.
• Mix your methods, even if there is no time for "proper" sampling and statistics. If your main instruments are interviews and group discussions, look at the budgets and the spending as well - seeing where the money goes tends to yield valuable insights, too.

And, dear consultant, keep a close watch on the time you spend on quick evaluations. I think it is OK to put in some unpaid hours. But not days or weeks.