Classism in evaluation design

Individual interviews for "important persons", focus groups for "beneficiaries", right? Wrong!

These days I have been reviewing evaluations of projects supporting survivors of traumatising human rights violations in countries that are not quite at peace, or even still at war. One would think that in such circumstances, evaluators would be particularly respectful and careful with their interlocutors, avoiding questions and situations that would make them feel uncomfortable, trigger difficult emotions or cause a resurgence of their trauma. In some cases, the opposite is true:

Some evaluators asked people to talk about their traumatising experience in group discussions with five to ten persons - neighbours or strangers, people who were brought together in a one-off two-to-three-hour meeting only because the evaluators needed data from "beneficiaries". To obtain data from project managers or local officials, the same evaluators tended to prefer individual interviews. I see an implicit message here: People in positions of power deserve more individual attention than simple users of project services. Is that really what we want, when we evaluate projects that are supposed to strengthen people's confidence and empower them to transform their lives, contribute to change in their societies and make this world a better one?


The problem is not unique to human rights and service-related projects. I have seen evaluations of rural development programmes where "beneficiaries" were mainly interviewed in groups - for instance, in the convenient setting of an agricultural extension class. It is not only an issue of respect, or lack thereof; it is also a methodological problem. In group interviews, people speak not only to the person who conducts the interview, but also to everybody else who sits in the circle (or around the table). As a result, they are likely to speak in ways and about things they consider acceptable in that group setting (social desirability bias) - not necessarily about their true thoughts and feelings. Focus group discussions are not a good instrument to learn about personal thoughts and experience.

But they can be an excellent instrument for questions that are less personal, for instance, to map actors in a field the participants are familiar with, to learn about local social norms, or to get different experts' views on a certain topic. For instance, when a project is about health services, it can make sense to run focus group discussions with health providers: They can explain the situation in their sector, sketch typical processes, discuss together where exactly the project fits in and what contributions it may have made, and so forth. 

I would like to come back to the point of respectful interviews, especially when interviewees are survivors of traumatising violations. I did find one excellent example: The researchers designed questionnaires and interview guides that kept people from digging too deeply into difficult memories. They gave survivors a few days to think before they consented to be interviewed, and offered them the choice of the interview setting - a counselling centre, for instance, or a secluded hotel in a pleasant area. They provided breaks and meals, a couple of nights' accommodation if needed, as well as a post-interview check-out with a psychologist - all that to make sure any distress caused by the interview could be dealt with. Coincidentally, the researchers worked in a European country. There is no reason why one shouldn't work that way in Africa or Asia, is there?