Blog

Stay up to date with Giving What We Can's blog. Featuring the latest news about our community, how to donate effectively, charity evaluation, community events and more!

Giving What We Can's blog, featuring the latest news from our community.

Donate

Pledge

footerNavigation

About us

Contact us

Our team

Our values

Our members

Our events

Transparency

Our mistakes

Our impact

Donate to Giving What We Can

Careers

About

Our best charities

Take a giving pledge

Get involved

Make a bequest

Videos, books, podcasts

FAQs

Sitemap

Learn more

Press kit

Media coverage

Case studies: People who pledge to give

Media

Footer Navigation

DONATE

topNavigation

Best charities

High-impact causes

Why give to funds

How much to donate

Why donate publicly

Recommendations

Our research & approach

Comparing charities

Evaluating evaluators

Our approach

Giving guides

Videos, books, podcasts and essays

Hesitations about charitable giving

How rich am I?

Can money buy happiness?

Resources

Research

Team

The 🔸10% Pledge

🔹Trial Pledge

Company Pledge

Further Pledge

Bequests

Why pledge

Pledges

Newsletter

Podcast

Videos

Connect with our community

Sharing our ideas

Events

Giving games

Charity elections

Pledge drive

Fundraisers

Community

Donate now

Manage my donations

Tax deductibility

Australian page

My pledge

My donations

My income

My impact

Top Navigation

William MacAskill

Some General Concerns About GiveWell

Note: This article hasn't been updated recently. We're keeping it up for the historic record of the debate, and because people still seem to refer to the arguments.

In a previous post, I explained why I was surprised by GiveWell's recent recommendation of GiveDirectly. In this post, I'll take the opportunity to explain some other general concerns I have with GiveWell. As before, I'll note that these just represent my view, rather than those of GWWC as a whole, and that such criticism should be taken as a sign of respect for an organisation which I generally think very highly of. We are making these suggestions in part because we draw heavily on GiveWell's own research and want it to be as compelling as possible.

I'll also note that many of these concerns can also be applied to GWWC itself. This list therefore also represents a list of improvements we would like to make to our own research and publications in future.

Different countries vary dramatically in their rate of economic growth. In 

, India's economy grew by 7.8%, whereas Namibia's economy grew by just 3.8%. In the long-run, insofar as your donation generates some jump in the productivity of the labour force, compounding effects will cause it to snowball into substantially more income if used in India than in Namibia. On this basis one might argue that donations to India constitute a better 'investment' than donations to Namibia. On the other hand, one might reason that, because we should expect people to be richer in India in the future, they will get less benefit from any extra income in the future. The economic trajectory of the country that you try to help could potentially be a 'crucial consideration' concerning which charity you should give to, but it hasn't yet been addressed by GiveWell.

This is not even considering the magnitude and persistence of an intervention's impact on economic productivity. This is another crucial consideration which we understand GiveWell has looked into and found very difficult to answer.

As far as I know, GiveWell haven't commissioned a thorough 

 of their recommendations. (GiveWell have done some external evaluations in the past, but mainly in 2010 or 2011, and mainly done by volunteer undergraduates or recent graduates.) This surprises me. Whereas businesses have a natural feedback mechanism, namely profit or loss, research often doesn't, hence the need for peer-review within academia. This concern, when it comes to 

, is even greater. If GiveWell's analysis and recommendations had major flaws, or were systematically biased in some way, it would be challenging for outsiders to work this out without a thorough independent evaluation. Fortunately, GiveWell has the resources to, for example, employ two top development economists to each do an independent review of their recommendations and the supporting research. This would make their recommendations more robust at a reasonable cost.

To put it simply: If GiveWell's top recommended charities are so good, why hasn't their room for more funding been used up?

, because the best ones tend to be snapped up quite quickly. But they have provided very public information on their top-recommended causes, some of which have been on the list for years. And we know that major philanthropic foundations (such as the Gates Foundation, and CIFF) are aware of their research. So it's surprising, in their view, that the room for more funding for their top-recommended organisations wasn't used up rather more rapidly. This either suggests that their model of how philanthropy works is wrong, or that their recommendations aren't believed by the big players in the 'philanthropic market'.

Even if we accept that their model of an efficient market in philanthropy is inaccurate, and so could attribute the failure of big foundations to fund their recommended charities to irrationality or different values, there remains cause for concern. GiveWell functions in an advisory capacity to GoodVentures, which has very similar aims and attitudes to those of GiveWell. GoodVentures' endowment could easily use up the room for more funding at GiveWell's top charities. So it raises the question why GoodVentures doesn't donate to GiveWell's top charities: even if not exhausting their room for more funding, at least using up a considerable proportion of it. The most plausible explanation in my mind is that GiveWell thinks that GoodVentures can better spend its money at a later date after further research. In which case, it's hard to see why individual donors shouldn't save and donate later, or even try to give to GoodVentures. Perhaps that's wrong, but it's worthy of discussion.

GiveWell claim that they want to maximise good done per dollar invested, but they haven't explained what they think 'good' consists in and why. Do future people count? If so, how much, and how far into the future? Do they have a discount rate? How much priority should they give to the worst off? Is an infant death worse, or better, than the death of a 20 year old? How do they weigh improvements in health against extensions in health? What do they think is the relationship between economic growth and wellbeing?

These are just of theoretical interest - varying any of the parameters could lead to different charitable recommendations. For example, when they discuss AMF's cost-effectiveness, they focus on under-5 mortality. But they don't discuss the fact that most people think that an infant death is not as great a loss as a young adult death; an intuitive view that has good theoretical grounds. (This is something that will likely be taken into account in the next generation of the DALY metric, in the third edition of the Disease Control Priorities Project.) Depending on how, exactly, one thinks one should weigh saving infant lives against saving adult lives, this issue could make AMF several times less cost-effective relative to SCI than it is currently judged to be. This one methodological issue could therefore dramatically alter their recommendations.

To their credit, GiveWell has moved towards helping people to tailor their choice to their values, by comparing the cost of bed nets delivered, children dewormed, and cash delivered. This can be done using spreadsheets and explanations provided in some recent 

. In its current form this requires more work than most donors would be willing to engage in. Making this easier, and guiding people through discovering the implications of their moral values would be worthwhile improvements. This kind of tailored advice is only possible for the minority of charities GiveWell has looked into in great detail, which means it is still important to be explicit about the value judgements that generated that shortlist.

Once they allow the moral view or 'judgment calls' of the donor to be an input, providing a #1 #2 #3 ranking of charities becomes misleading. For example, I wrote previously that I was surprised by their recommendation of GiveDirectly. Most of my surprise would have been alleviated if, rather than ranking them #2, they gave them a conditional recommendation: "If you have such-and-such moral view, then GiveDirectly is an excellent option." Indeed, there are many reasonable views based on which giving to GiveDirectly could be the best option. Perhaps you do not want to be paternalistic towards the poor, even if being paternalistic would do more to improve their wellbeing in the long run. Or perhaps the reason you give is because you think you should redistribute your wealth as a matter of justice, and it's not your right to say how that money is spent. In either case, it's very useful to know that one has a good option to put wealth directly into the hands of the poor. But, if that's the motivation for recommending GiveDirectly, ranking them above SCI and below AMF disguises the fact that it is being judged on a different set of criteria.

Bastian Stern

The $/DALY Illusion: Part I - Comparing Improvements

Imagine the following situation: Two of your friends, Jack and Jill, are each about to donate $1000 to two different charities. You know that both of them are on their way to their closest letter box, with the intention of posting their cheques to those charities. Unfortunately, neither of them has heard of the idea of cost-effective giving, and you know that you only have time to call up one of them convince one of them of directing their donation to a more cost-effective charity instead before they have reached the letter box to and it is too late to do that. You know that is costs $1000 to avert a DALY by donating to Jack's preferred charity, A, and that the best charity you could convince him to donate to instead, B, would allow him to avert a DALY for only $50. Jill's preferred charity, C, allows her to avert a DALY for $20, but you know of an even better charity, D, that would allow her to avert a DALY for only $10, and you have reason to believe that you could convince Jill to donate to D instead.

A tempting first guess is that you should call up Jack: The difference in A's and B's cost-effectiveness measured in $/DALY (1000 $/DALY- 50 $/DALY=$950 $/DALY) dwarfs that between C and D (20 $/DALY-10 $/DALY=10 $/DALY). (Call this the "linear method" of evaluating the relevant differences.)

A tempting second guess would be to look at what we might call the proportional improvement in cost-effectiveness of the changes from A to B and from C to D. On the most natural understanding of this approach, the proportional improvement between A's and B' cost-effectiveness measured in $/DALY is (1000 $/DALY - 50 $/DALY)/(1000 $/DALY)=95% and the proportional difference in C's and D's cost-effectiveness measured in $/DALY is (20 $/DALY - 10 $/DALY)/(20 $/DALY)=50%. Again, this would seem to strongly suggest that calling up Jack is much better than calling up Jill. (Call this the "proportional method" of evaluating the relevant differences.)

In fact, simple calculation shows that both lines of reasoning are mistaken. By donating the $1000 to A, Jack would avert 1000/1000=1 DALY, whereas donating to B would allow him to avert 1000/50= 20 DALY. This means that calling him up allows you to avert 19 additional DALY. In contrast, donating her $1000 to C would allow Jill to avert 1000/20=50 DALY, whereas by donating to D she could avert 1000/10=100 DALY. This means that calling her up allows you to avert 50 additional DALYs. Accordingly, calling up Jill allows you to have more than twice the positive impact!

What went wrong when we made our first and second guesses, which turned out to be so radically misleading? The answer is that when you are interested in how much good you can do by redirecting a given amount of money between different causes, you are interested in how many more DALYs you can avert by giving to one cause rather than another, i.e. differences in their cost-effectiveness reported DALY/$.

Where "x" is the cost-effectiveness of charity X reported in $/DALY, and "y" the cost-effectiveness of charity Y reported in $/DALY, the improvement in cost-effectiveness in moving from X to Y reported in terms of DALY/$ is given by (1/x)-(1/y), or, equivalently, by (y-x)/xy (which, as you can check for yourself, delivers the correct result in the case above, namely that the improvement in moving from C and D is 50/19≈2.6 times greater than that between A and B.)

This also shows what was wrong with our first two guesses. Our first guess was to evaluate the improvement from X to Y's cost-effectiveness as y-x (which I called the linear method), our second guess as the proportional improvement between y and x, i.e. as (y-x)/y (which I called the proportional method). However, these latter two functions evidently fail to be related in a strictly increasing way with (y-x)/xy (i.e. fail to be related in such a way that whenever (y-x)/xy increases, these latter functions increase, too), which is why both the linear and the proportional methods give the wrong answer.

The linear method misled us because it simply ignores the denominator of (y-x)/xy, xy, altogether. The proportional method misled us because it omits part of the denominator of (y-x)/xy, namely x. Both methods systematically overvalue improvements in cost-effectiveness between less effective charities (like A and B), and, by the same token, systematically undervalue improvements in cost-effectiveness between especially effective charities (like C and D).

Note that had we reported the cost-effectiveness estimates in DALY/$ from the outset, these complications would not have arisen.

Also note that there are cases in which straightforwardly taking the differences (i.e. using the linear method) does tell us what we are interested in. For instance, imagine that both Jack and Jill want to avert exactly one DALY (rather than spending a fixed amount of $1000), and your primary interest is to advise them so that they save as much money as possible. In that case, it really is the case that x-y is what you should be interested in. By averting 1 DALY by giving to B rather than A, Jack can save $950, and by averting 1 DALY by giving to D rather than C, Jill can only save $10, which would suggest that, in this altered scenario, you should call up Jack.

However, whilst this latter sort of situation is what we typically encounter when making day-to-day purchases (e.g. when you want to buy exactly one chocolate bar, and you are interested in comparing how much money you can save by choosing between various shops), it is clearly not the sort of situation donors are typically faced with in the context of charitable giving. Donors are usually interested in the differences in how much good they can do with a given amount of money, rather than the differences in how much money they can save for a given amount of good they want to do.

In the next post, I will discuss complications the $/DALY metric gives rise to when we talk about cost-effectiveness averages.

[1] The phenomenon I am about to describe is analogous to the well-known MPG Illusion: People tend to be misled about the value of improvements in car fuel efficiency when efficiency is reported in miles per gallon (instead of a unit that allows us to state efficiency in terms of volume per distance traveled, e.g. liters per 100 kilometers). Reporting efficiency in terms of MPG also gives rise to complications associated with evaluating average efficiency analogous to those I discuss below.

[2] The Disability-Adjusted Life Year (DALY) is a metric that is commonly used to quantify and compare the benefits of different interventions. DALYs are a measure of lost health due to illness. One DALY represents the equivalent to losing a year of life at full health. Each illness has a disability weight assigned to it, representing the amount it reduces a person's quality of life. For example, a year of life spent with blindness counts for 0.6 DALYs.

[3] I take this to be "most natural" given that basing one's comparison of two improvements in cost-effectiveness on the comparison of the corresponding reductions in cost per unit effect (which this "proportional method" amounts to) appears to be the best analogue of a mistake some people make when asked to compare the improvements in car fuel efficiency reported in MPG. In the latter case, people sometimes (mistakenly) think that, where the improvement comes about by moving from car A to car B, they need to calculate the "proportional improvement" given by (Efficiency(B)-Efficiency(A))/Efficiency(A), and compare improvements by comparing "proportional improvements". 

Julia Wise

The Way It Was

In thinking about problems that currently affect developing nations, I try to remember that the United States 

Malaria once plagued the American south and Midwest. It's the reason English colonists abandoned the Jamestown, Virginia settlement for somewhere with fewer mosquitoes. In 1946, the Centers for Disease Control were formed to fight malaria. Five years later, malaria was eradicated in the United States. (

, Laura Ingalls Wilder describes her family's experience with the disease in 1870: 

Laura tried to get up, but she was too tired. Then she saw Ma's red face looking over the edge of the bed. Mary was all the time crying for water. Ma looked at Mary and then she looked at Laura, and she whispered, "Laura, can you?" "Yes, Ma," Laura said. This time she got out of bed. But when she tried to stand up, the floor rocked and she fell down. . . . She knew she must get water to stop Mary's crying, and she did. She crawled all the way across the floor to the water-bucket. There was only a little water in it. She shook so with cold that she could hardly get hold of the dipper. But she did get hold of it. She dipped up some water, and she set out to cross that enormous floor again.

It drives me crazy when people say child mortality is saving us from environmental disaster. Do they wish that for their own families? Do they feel they have too many siblings, too many cousins, and we really ought to get some contaminated water or intestinal parasites around here to deal with the problem?

No one wants that for their family. We want low child mortality and reasonable family size. Developed nations made that transition, but it took a while.

Hans Rosling does an excellent job at making public health statistics understandable to non-statisticians. I recommend his talk on "

", especially for its comparison of child mortality rates across nations (at 8:40). Rosling notes that his home country of Sweden had a high child mortality rate in 1800, twice as high as anywhere now. Over time, with better education and better public health, the rate declined to its current low level. Most countries are currently on this journey of decreasing family size 

 child mortality, many of them progressing faster than Western countries ever did. (The site 

 lets you play with the charts yourself.)

While it's clear that large family size is correlated with child mortality, I'm not as convinced as Rosling is that lower child mortality is the main 

 of smaller families. I gather it's a combination of fewer child deaths, female education, access to birth control, and urbanization.

The Ingalls family were typical 19th-century Americans - they had five children, one of whom died in infancy. I come from a typical 20th-century American family with two children, both still alive. I'm 

 we made this demographic transition. My hometown no longer has malaria. I'm educated, I drink clean water, and I control my own fertility. When I choose, I will raise one or two children who will probably also be quite healthy.

We've come a long way. Now I want this life for everyone.

Rob Wiblin

Can we evaluate political advocacy?

', which advocates for greater government spending to tackle tuberculosis (TB). Research volunteers Nick Beckstead, Rossa O'keeffe-O'donovan, and Gordon Irlam of the 

Back of the Envelope Guide to Philanthropy

, all deserve a great deal of credit for their help with this work. We also appreciate the assistance of the folks at RESULTS, in particular Jennifer Maurer, for providing detailed feedback on an earlier version. Whether or not RESULTS ultimately beats our current recommended charities, they are working hard to make a difference on crucial issues in global health.

This is the first time we have released our preliminary research on a topic in this way, and quantifying the good done by political advocacy raised novel challenges for us. We not only have to establish whether the advocacy efforts have generated additional spending, but also try to work out what the additional money has ultimately been spent on and whether it is likely to have worked. Unlike when we consider the Against Malaria Foundation, we cannot just request the relevant data from the charity delivering the programme. Nonetheless, I am excited that we are stepping beyond the more familiar bounds of direct health interventions. Many members have requested that we do so, and if we can identify the right campaigns to get behind, political advocacy offers the potential to move large sums towards programmes that really work to help the world's poorest people.

We are hoping that posting this document will encourage people with experience in the following areas to talk with us:

• Evaluating the effectiveness of political advocacy;

• Experience with RESULTS/ACTION or similar organisations, and their influence in Washington;

• Understanding of bilateral US aid for TB, and the funding mechanisms used by The Global Fund and GAVI;

• Understanding of the cost effectiveness of further spending on TB diagnostics, treating multi-drug resistant strains, and patients with both HIV and TB.

If you know about these issues and are willing to help, please email 

, or Skype me personally at robertwiblin.

We will continue to try to answer our remaining questions and build our model over Christmas and New Year to reach a decision about whether to recommend RESULTS in January. In the meantime, I will continue to update the version available on the website as we correct problems and improve our model.

Without further ado, here is the executive summary. Happy reading!

RESULTS is a charity that advocates for improvements to public policy to reduce poverty. This report focuses on ACTION USA, a project of RESULTS, which has appeared to achieve significant policy successes in the past, through a campaign for increased spending on the treatment of tuberculosis (TB). We believe that RESULTS' past cost effectiveness in terms of years of healthy life for each dollar spent has been high, possibly higher than our current top recommended charities. It continues to advocate for greater spending on TB, and now also vaccinations, both areas which include some very effective projects as measured by the years of healthy life (QALYs) they generate per dollar spent.

Unfortunately, the evaluation of advocacy charities is very difficult. It is not usually possible to confidently attribute the passage of a piece of legislation to a particular organization. The addition of further links to the chain between your donation and the ultimate outcome you want (e.g. additional people being cured of TB), creates the opportunity to generate 'leverage' as you potentially influence the way billions of dollars are spent. On the other hand, it means you cede some control over how that money is spent. It also exposes you to the risk that the impact of your donation will be nothing, either because the legislation you desire fails to pass, or because one of the organisations through which the funding passes choses not to, or could not, make effective use of the additional money. Finally, politics is a strategic and rapidly changing activity which can make it hard for outsiders such us Giving What We Can to accurately model what is occurring.

While we may recommend RESULTS, or a similar charity, in the future, we currently have insufficient information to do so with confidence. We are releasing this draft in order to solicit information which can help us answer the remaining questions, which are critical to our estimate of future cost effectiveness. In the meantime, we see ACTION is a highly promising option for those who seek to help people in developing countries through political advocacy.

• How large was ACTION's role in achieving the observed policy successes?

• What are additional appropriations in the US being spent on, by the time funding is passed through to health providers? How effective are those activities at improving health, in QALYs/$?

• How valuable is the government or private spending crowded out by any resulting appropriations?

• What is the marginal effectiveness of RESULTS' advocacy in the USA in the event of an expansion of operations?

• Are there strategic considerations (for instance, re-categorisation of spending, or misleading attributions of credit) that could be causing us to misunderstand what is going on?

Letting Myself Care

Recently I was talking with friends about what role emotion should play in giving decisions. Some said we shouldn't let emotions muddle our thinking.

We're emotional creatures (some more than others). Personally, I'm pretty high on the bleeding-heart scale. One thing I loved about Oxfam is that they do a great job documenting their work in articles and photography, so I could feel really connected to the people they work with. (

 is also very good at this-ed.) I put their pictures on my wall. I wanted to see their faces."But I want to have an emotional connection to the people I help," another said. I found myself agreeing with both sides.

Of course, I don't want to choose a cause based on how emotionally connected I already am to it. I don't want to pass up an excellent opportunity to help because I felt more pulled to a less effective cause. When it comes to choosing a cause, I want to act based on 

, not sentiment. At the end of the year I'll reevaluate which charities to support, and I don't want to be muddled when I do that.

But once I've chosen a cause, I give myself permission to fall in love with it. What I care about, after all, is not the cause itself but the human lives at stake. (For you, it might be animal lives as well, or the lives of people not yet born. But you get the idea.) So I look for a way to connect. (For example, seeing how DtW help the individuals they treat 

 posts photos of most of its bednet distributions. It's a way to document that they're doing what they say they are - but for me, it's also a chance to see the human face of the work they do. The work I help them do.

Seeing her face is a way of overcoming the blind chance that put me in the US and her in Mali. It's a way of making us neighbours.

It's hard to feel warm and fuzzy about our donations going to dozens of distribution points around the world. We didn't evolve to feel an emotional glow about numbers. It can feel like we're sending money into some kind of oblivion. It's hard to feel motivated about that.

That's why I imagine it going to her. I imagine how glad her family will be to see her reach her eighth birthday, eleventh birthday, twentieth birthday. I imagine all the things she can do now that she's not sick. That kind of glow is what my heart was made for.

Sebastian Farquhar

Christmas gifts, goats and cash-transfer charities

It’s getting closer to Christmas, and we’re running out of time to get presents for friends and family. It can be hard to work out what presents people will actually enjoy. An increasingly popular option is to make a donation on behalf of someone else as a present.

When it comes to presents, it seems like a thoughtful gift is better than a gift certificate, and a gift certificate is better than a thoughtless gift. It’s a lot like that for charity gifts too. Your best options are likely to be donations towards 

. And in last place will be the unasked for gift of a goat.

What was originally a fairly uninspired set of options for charity gifts has since turned into a wide range of charitable choices for the holidays. Oxfam alone lets you select gifts from 

But the old stand-by is sending farm animals to developing countries. This seems unlikely to be motivated by effectiveness. I googled “Give a _____ for christmas” for a bunch of farm animals and wrote down the estimated number of search results.

Now, it’s possible that goats are just much much more valuable to rural people living in developing countries than any other sort of animal. Alternatively, it’s possible that the alliteration of “g” in “Give a goat” makes for catchier headlines. I don’t know for sure, but I have my suspicions.

There are also some decent reasons to think, as 

donations of livestock might be a particularly ineffective way to help the developing world.

 They seem to be quite a lot like cash transfers, but with a bunch of extra question marks.

(which is currently GiveWell’s second recommended charity after AMF)

 you empower people to make a decision about what they want most and buy it. That is, cash transfers are like the gift certificate of the charity world.

There are some real problems with cash transfers. For example they can lead to resentment and arguments when some people receive the transfer and others don’t or have distorting effects on local economies.

But with animals you lose most of the benefit of cash-transfers - you’re just deciding for people what you think they’ll want. You also avoid many of the benefits of the best non-cash transfer charities. Buying someone a goat for Christmas is like getting someone chocolate. You didn’t ask them what they wanted, you don’t really know them too well personally, so you just got the first thing on the shelf that looked like a present. The difference is that chocolate won’t poop on your floor.

But that doesn’t mean we have to just cave in and do the charity equivalent of a gift certificate. What’s the analogue to a really nice thoughtful gift that suits someone perfectly?

It turns out, you can get a thoughtful gift and donate to the most effective charities at the same time! Giving What We Can has it all. 

 lets you make a donation to one of their top recommended charities.

Adam Casey

Does Oxfam Help?

Oxfam is placing itself at the forefront of the push towards evidence-based charity. This is a very encouraging sign from one of the world's largest non-governmental organisations.

In general, one problem that charity evaluators like us encounter is that not enough studies have been done to measure the effect of different projects. The main reason cited by charities for this is that properly conducted 

 (RCTs) are prohibitively expensive. We are regularly asked what our view is of Oxfam, and have previously had to say that we just don't know, because not enough information is collected to assess their impact. This is why we are very glad to see Oxfam releasing data from its new Effectiveness Review.

In September 2011 Dr Karl Hughes published a 

 asking how charities could demonstrate they were being effective without bankrupting themselves. This year Oxfam answered his question by publishing the first results from a new "Effectiveness Review". Oxfam randomly selected a small group of interventions and studied them in depth to get a sense of how effective "typical" Oxfam work is.

Oxfam is currently running 362 "mature" projects. Of these they selected 26 spread across all categories of intervention Oxfam is involved in. 12 community based projects were analysed quantitatively with RCTs and 9 policy interventions were analysed by qualitative methods. Humanitarian projects were analysed in terms of how well they kept to recognised quality standards; 5 interventions were analysed in this way. In addition there are plans for 3 pilot studies looking into the accountability of projects.

Karl Hughes described the conclusions of the Effectiveness Review as "mixed". It is a testament to Oxfam's commitment to transparency that they have published even the unfavourable results. The general conclusion is that most Oxfam interventions make things better, but only on some measures. With further study it is reasonable to hope that Oxfam could dramatically improve their interventions, focusing on projects (like a 

 in Pakistan) which have very strong evidence of effectiveness. Furthermore, they might stop altogether projects that have low or even negative impact, freeing up resources for those projects that work.

The Effectiveness Review was an initial attempt, and it leaves plenty of room for improvement. For instance Oxfam did RCTs only in 12 of the 26 projects they studied. This suggests that there is room to expand the use of the "gold standard of evidence" much further.

The effectiveness review also failed to address the key question that Giving What We Can seeks to answer: How much good can we do at how great a cost? This is a very important point that we hope Oxfam's next Effectiveness Review will consider in detail. However, getting evidence of a positive effect is a very important first step.

There is a lot of good in this report. At the very least, it demonstrates that on average Oxfam has a positive effect. But more than this it tells Oxfam where to focus its efforts. Oxfam now needs to look at the very effective Pakistan disaster risk-management programme and try to replicate its success elsewhere. They also need to look carefully at one of the livelihood projects studied and try to understand why the control group ended up better off than the intervention group.

More generally, this report demonstrates that organisations like Oxfam can take evidence seriously without compromising their work on the ground. It is estimated that each study cost only £10,000 excluding staff time. This is very little, compared with the quarter of a billion that the organisation spends each year. Hopefully this first attempt will inspire Oxfam and other large charities to investigate their effectiveness more systematically. If these studies help Oxfam use its money 1% more effectively, then they will have paid for themselves many times over.

This Review also provides good examples of how to get useful evidence despite difficulties on the ground. In an RCT it is important for the control and intervention groups to be as similar as possible. But this is very hard in practice. For instance in the case of the Pakistan flood risk-management project there were statistically significant differences in the average age, population and distance to market between the two groups of villages. However, Oxfam demonstrated that this need not make the RCTs worthless. By using linear regression and other statistical techniques the report attempted to correct for these discrepancies.

This review was very preliminary, but it offers hope for the future. Key improvements would be a focus on quantified cost/benefit analysis and an expansion of the use of RCTs to a wider range of projects. But the fact that Oxfam has run and published such an extensive study, at a reasonable cost, is a milestone for effective charity. Hopefully other charities will follow their example.