Potential Risks from Advanced Artificial Intelligence: The Philanthropic Opportunity

We’re planning to make potential risks from artificial intelligence a major priority this year. We feel this cause presents an outstanding philanthropic opportunity — with extremely high importance, high neglectedness, and reasonable tractability (our three criteria for causes) — for someone in our position. We believe that the faster we can get fully up to speed on key issues and explore the opportunities we currently see, the faster we can lay the groundwork for informed, effective giving both this year and in the future.

With all of this in mind, we’re placing a larger “bet” on this cause, this year, than we are placing even on other focus areas — not necessarily in terms of funding (we aren’t sure we’ll identify very large funding opportunities this year, and are more focused on laying the groundwork for future years), but in terms of senior staff time, which at this point is a scarcer resource for us. Consistent with our philosophy of hits-based giving, we are doing this not because we have confidence in how the future will play out and how we can impact it, but because we see a risk worth taking. In about a year, we’ll formally review our progress and reconsider how senior staff time is allocated.

This post will first discuss why I consider this cause to be an outstanding philanthropic opportunity. (My views are fairly representative, but not perfectly representative, of those of other staff working on this cause.) It will then give a broad outline of our planned activities for the coming year, some of the key principles we hope to follow in this work, and some of the risks and reservations we have about prioritizing this cause as highly as we are.

In brief:

  • It seems to me that artificial intelligence is currently on a very short list of the most dynamic, unpredictable, and potentially world-changing areas of science. I believe there’s a nontrivial probability that transformative AI will be developed within the next 20 years, with enormous global consequences.
  • By and large, I expect the consequences of this progress — whether or not transformative AI is developed soon — to be positive. However, I also perceive risks. Transformative AI could be a very powerful technology, with potentially globally catastrophic consequences if it is misused or if there is a major accident involving it. Because of this, I see this cause as having extremely high importance (one of our key criteria), even while accounting for substantial uncertainty about the likelihood of developing transformative AI in the coming decades and about the size of the risks. I discuss the nature of potential risks below; note that I think they do not apply to today’s AI systems.
  • I consider this cause to be highly neglected in important respects. There is a substantial and growing field around artificial intelligence and machine learning research, but most of it is not focused on reducing potential risks. We’ve put substantial work into trying to ensure that we have a thorough landscape of the researchers, funders, and key institutions whose work is relevant to potential risks from advanced AI. We believe that the amount of work being done is well short of what it productively could be (despite recent media attention); that philanthropy could be helpful; and that the activities we’re considering wouldn’t be redundant with those of other funders.
  • I believe that there is useful work to be done today in order to mitigate future potential risks. In particular, (a) I think there are important technical problems that can be worked on today, that could prove relevant to reducing accident risks; (b) I preliminarily feel that there is also considerable scope for analysis of potential strategic and policy considerations.
  • More broadly, the Open Philanthropy Project may be able to help support an increase in the number of people – particularly people with strong relevant technical backgrounds – thinking through how to reduce potential risks, which could be important in the future even if the work done in the short term does not prove essential. I believe that one of the things philanthropy is best-positioned to do is provide steady, long-term support as fields and institutions grow.
  • I consider this a challenging cause. I think it would be easy to do harm while trying to do good. For example, trying to raise the profile of potential risks could contribute (and, I believe, has contributed to some degree) to non-nuanced or inaccurate portrayals of risk in the media, which in turn could raise the risks of premature and/or counterproductive regulation. I consider the Open Philanthropy Project relatively well-positioned to work in this cause while being attentive to pitfalls, and to deeply integrate people with strong technical expertise into our work.
  • I see much room for debate in the decision to prioritize this cause as highly as we are. However, I think it is important that a philanthropist in our position be willing to take major risks, and prioritizing this cause is a risk that I see as very worth taking.

My views on this cause have evolved considerably over time. I will discuss the evolution of my thinking in detail in a future post, but this post focuses on the case for prioritizing this cause today.

Importance

It seems to me that AI and machine learning research is currently on a very short list of the most dynamic, unpredictable, and potentially world-changing areas of science.1  In particular, I believe that this research may lead eventually to the development of transformative AI, which we have roughly and conceptually defined as AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution. I believe there is a nontrivial likelihood (at least 10% with moderate robustness, and at least 1% with high robustness) that transformative AI will be developed within the next 20 years. For more detail on the concept of transformative AI (including a more detailed definition), and why I believe it may be developed in the next 20 years, see our previous post.

I believe that today’s AI systems are accomplishing a significant amount of good, and by and large, I expect the consequences of further progress on AI — whether or not transformative AI is developed soon — to be positive. Improvements in AI have enormous potential to improve the speed and accuracy of medical diagnosis; reduce traffic accidents by making autonomous vehicles more viable; help people communicate with better search and translation; facilitate personalized education; speed up science that can improve health and save lives; accelerate development of sustainable energy sources; and contribute on a huge number of other fronts to improving global welfare and productivity. As I’ve written before, I believe that economic and technological development have historically been highly beneficial, often despite the fact that particular developments were subject to substantial pessimism before they played out. I also expect that if and when transformative AI is very close to development, many people will be intensely aware of both the potential benefits and risks, and will work to maximize the benefits and minimize the risks.

With that said, I think the risks are real and important:

  • Misuse risks. One of the main ways in which AI could be transformative is by enabling/accelerating the development of one or more enormously powerful technologies. In the wrong hands, this could make for an enormously powerful tool of authoritarians, terrorists, or other power-seeking individuals or institutions. I think the potential damage in such a scenario is nearly limitless (if transformative AI causes enough acceleration of a powerful enough technology), and could include long-lasting or even permanent effects on the world as a whole. We refer to this class of risk as “misuse risks.” I do not think we should let misuse scenarios dominate our thinking about the potential consequences of AI, any more than for any other powerful technology, but I do think it is worth asking whether there is anything we can do today to lay the groundwork for avoiding misuse risks in the future.
  • Accident risks. I also believe that there is a substantial class of potential “accident risks” that could rise (like misuse risks) to the level of global catastrophic risks. In the course of many conversations with people in the field, we’ve seen substantial (though far from universal) concern that such risks could arise and no clear arguments for being confident that they will be easy to address. These risks are difficult to summarize; we’ve described them in more detail previously, and I will give only a basic outline here. As goal-directed AI systems (such as reinforcement learning systems) become more capable, they will likely pursue the goals (e.g. as implied by a loss function) assigned to them in increasingly effective, unexpected, and hard-to-understand ways. Among these unexpected behaviors, there could be harmful behaviors, arising from (a) mismatches between the goals that programmers conceptually intend and the goals programmers technically, formally specify; (b) failures of AI systems to detect and respond to major context changes (I understand context change to be an area that many currently-highly-capable AI systems perform poorly at); (c) other technical problems. (See below for a slightly more detailed description of one possible failure mode.) It may be difficult to catch undesirable behaviors when an AI system is operating, in part because undesirable behaviors may be hard to distinguish from clever and desirable behaviors. It may, furthermore, be difficult and time-consuming to implement measures for confidently preventing undesirable behaviors, since they might emerge only in particular complex real-world situations (which raise the odds of major context changes and the risks of unexpected strategies for technically achieving specified goals) rather than in testing. If institutions end up “racing” to deploy powerful AI systems, this could create a significant risk of not taking sufficient precautions.The result could be a highly intelligent, autonomous, unchecked system or set of systems optimizing for a problematic goal, which could put powerful technologies to problematic purposes and could cause significant harm. I think the idea of a globally catastrophic accident from AI only makes sense for certain kinds of AI – not for all things I would count as transformative AI. My rough impression at this time is that this sort of risk does not have a high overall likelihood (when taking into account that I expect people to take measures to prevent it), though it may have a high enough likelihood to be a very important consideration given the potential stakes. In conversations on this topic, I’ve perceived very large differences of opinion on the size of this risk, and could imagine changing my view on the matter significantly in the next year or so.
  • Other risks. Some risks could stem from changes that come about due to widespread use of AI systems, rather than from a particular accident or misuse. In particular, AI advances could dramatically transform the economy by leading to the automation of many tasks – including driving and various forms of manufacturing – currently done professionally by many people. The effects of such a transformation seem hard to predict and could be highly positive, but there are risks that it could greatly exacerbate inequality and harm well-being by worsening employment options for many people. We are tentatively less likely to focus on this type of risk than the above two types, since we expect this type of risk to be (a) relatively likely to develop gradually, with opportunities to respond as it develops; (b) less extreme in terms of potential damage, and in particular less likely to be a global catastrophic risk as we’ve defined it, than misuse or accidents; (c) somewhat less neglected than the other risks. But this could easily change depending on what we learn and what opportunities we come across.

The above risks could be amplified if AI capabilities improved relatively rapidly and unexpectedly, making it harder for society to anticipate, prepare for, and adapt to risks. This dynamic could (though won’t necessarily) be an issue if it turns out that a relatively small number of conceptual breakthroughs turn out to have very general applications.

If the above reasoning is right (and I believe much of it is highly debatable, particularly when it comes to my previous post’s arguments as well as the importance of accident risks), I believe it implies that this cause is not just important but something of an outlier in terms of importance, given that we are operating in an expected-value framework and are interested in low-probability, high-potential-impact scenarios.2 The underlying stakes would be qualitatively higher than those of any issues we’ve explored or taken on under the U.S. policy category, to a degree that I think more than compensates for e.g. a “10% chance that this is relevant in the next 20 years” discount. When considering other possible transformative developments, I can’t think of anything else that seems equally likely to be comparably transformative on a similar time frame, while also presenting such a significant potential difference between best- and worst-case imaginable outcomes.

One reason that I’ve focused on a 20-year time frame is that I think this kind of window should, in a sense, be considered “urgent” from a philanthropist’s perspective. I see philanthropy as being well-suited to low-probability, long-term investments. I believe there are many past cases in which it took a very long time for philanthropy to pay off,3 especially when its main value-added was supporting the gradual growth of organizations, fields and research that would eventually make a difference. If I thought there were negligible probability of transformative AI in the next 20 years, I would still consider this cause important enough to be a focus area for us, but we would not be prioritizing it as highly as we plan to this year.

The above has focused on potential risks of transformative AI. There are also many potential AI developments short of transformative AI that could be very important. For example:

  • Autonomous vehicles could become widespread relatively soon.
  • Continued advances in computer vision, audio recognition, etc. could dramatically alter what sorts of surveillance are possible, with a wide variety of potential implications; advances in robotics could have major implications for the future of warfare or policing. These could be important whether or not they ended up being “transformative” in our sense.
  • Automation could have major economic implications, again even if the underlying AI systems are not “transformative” in our sense.

We are interested in these potential developments, and see the possibility of helping to address them as a potential benefit of allocating resources to this cause. With that said, my previously expressed views, if correct, would imply that most of the “importance” (as we’ve defined it) in this cause comes from the enormously high-stakes possibility of transformative AI.

Neglectedness

Both artificial intelligence generally and potential risks have received increased attention in recent years.4 We’ve put substantial work into trying to ensure that we have a thorough landscape of the researchers, funders, and key institutions in this space. We will later be putting out a landscape document, which will be largely consistent with the landscape we published last year. In brief:

  • There is a substantial and growing field, with a significant academic presence and significant corporate funding as well, around artificial intelligence and machine learning research.
  • There are a few organizations focused on reducing potential risks, either by pursuing particular technical research agendas or by highlighting strategic considerations. (An example of the latter is Nick Bostrom’s work, housed at the Future of Humanity Institute, on Superintelligence.) Most of these organizations are connected to the effective altruism community. Based on conversations we’ve had over the last few months, I believe some of these organizations have substantial room for more funding. There tends to be fairly little intersection between the people working at these organizations and people with substantial experience in mainstream research on AI and machine learning.
  • Ideally, I’d like to see leading researchers in AI and machine learning play leading roles in thinking through potential risks, including the associated technical challenges. Under the status quo, I feel that these fields – culturally and institutionally – do not provide much incentive to engage with these issues. While there is some interest in potential risks – in particular, some private labs have expressed informal interest in the matter, and many strong academics applied for the Future of Life Institute request for proposals that we co-funded last year – I believe there is room for much more. In particular, I believe that the amount of dedicated technical work focused on reducing potential risks is relatively small compared to the extent of open technical questions.
  • I’d also like to see a larger set of institutions working on key questions around strategic and policy considerations for reducing risks. I am particularly interested in frameworks for minimizing future misuse risks of transformative AI. I would like to see institutions with strong policy expertise considering different potential scenarios with respect to transformative AI; considering how governments, corporations, and individual researchers should react in those scenarios; and working with AI and machine learning researchers to identify potential signs that particular scenarios are becoming more likely. I believe there may be nearer-term questions (such as how to minimize misuse of advanced surveillance and drones) that can serve as jumping-off points for this sort of thinking.
  • Elon Musk, the majority funder of the Future of Life Institute’s 3-year grant program on robust and beneficial AI, is currently focusing his time and effort (along with significant funding) on OpenAI and its efforts to mitigate potential risks. (OpenAI is an AI research company that operates as a nonprofit.) We’re not aware of other similarly large private funders focused on potential risks from advanced artificial intelligence. There are government funders interested in the area, but they appear to operate under heavy constraints. There are individual donors interested in this space, but it appears to us that they are focused on different aspects of the problem and/or are operating a smaller scale.

Bottom line – I consider this cause to be highly neglected, particularly by philanthropists, and I see major gaps in the relevant fields that a philanthropist could potentially help to address.

Tractability

It’s been the case for a long time that I see this cause as important and neglected, and that my biggest reservation has been tractability. I see transformative AI as very much a future technology – I’ve argued that there is a nontrivial probability that it will be developed in the next 20 years, but it is also quite plausibly more than 100 years away, and even 20 years is a relatively long time. Working to reduce risks from a technology that is so far in the future, and about which so much is still unknown, could easily be futile.

With that said, this cause is not as unique in this respect as it might appear at first. I believe that one of the things philanthropy is best-positioned to do is provide steady, long-term support as fields and institutions grow. This activity is necessarily slow. It requires being willing to support groups based largely on their leadership and mission, rather than immediate plans for impact, in order to lay the groundwork for an uncertain future. I’ve written about this basic approach in the context of policy work, and I believe there is ample precedent for it in the history of philanthropy. It is the approach we favor for several of our other focus areas, such as immigration policy and macroeconomic stabilization policy.

And I have come to believe that there is potentially useful work to be done today that could lay the groundwork for mitigating future potential risks. In particular:

I think there are important technical challenges that could prove relevant to reducing accident risks.

Added June 24: for more on technical challenges, see Concrete Problems in AI Safety.

I’ve previously put significant weight on an argument along the lines of, “By the time transformative AI is developed, the important approaches to AI will be so different from today’s that any technical work done today will have a very low likelihood of being relevant.” My views have shifted significantly for two reasons. First, as discussed previously, I now think there is a nontrivial chance that transformative AI will be developed in the next 20 years, and that the above-quoted argument carries substantially less weight when focusing on that high-stakes potential scenario. Second, having had more conversations about open technical problems that could be relevant to reducing risks, I’ve come to believe that there is a substantial amount of work worth doing today, regardless of how long it will be until the development of transformative AI.

Potentially relevant challenges that we’ve come across so far include value learning (designing AI systems to learn the values of other agents through e.g. inverse reinforcement learning); problems having to do with making reinforcement learning systems and other AI agents less likely to behave in undesirable ways (designing reinforcement learning systems that will not try to gain direct control of their rewards, that will avoid behavior with unreasonably far-reaching impacts, and that will be robust against differences between formally specified rewards and human designers’ intentions in specifying those rewards); reliability and usability of machine learning techniques (including transparency, understandability, and robustness against or at least detection of large changes in input distribution); formal specification and verification of deep learning, reinforcement learning, and other AI systems; better theoretical understanding of desirable properties for powerful AI systems; and a variety of challenges related to an approach laid out in a series of blog posts by Paul Christiano.

Going into the details of these challenges is beyond the scope of this post, but to give a sense for non-technical readers of what a relevant challenge might look like, I will elaborate briefly on one challenge. A reinforcement learning system is designed to learn to behave in a way that maximizes a quantitative “reward” signal that it receives periodically from its environment – for example, DeepMind’s Atari player is a reinforcement learning system that learns to choose controller inputs (its behavior) in order to maximize the game score (which the system receives as “reward”), and this produces very good play on many Atari games. However, if a future reinforcement learning system’s inputs and behaviors are not constrained to a video game, and if the system is good enough at learning, a new solution could become available: the system could maximize rewards by directly modifying its reward “sensor” to always report the maximum possible reward, and by avoiding being shut down or modified back for as long as possible. This behavior is a formally correct solution to the reinforcement learning problem, but it is probably not the desired behavior. And this behavior might not emerge until a system became quite sophisticated and had access to a lot of real-world data (enough to find and execute on this strategy), so a system could appear “safe” based on testing and turn out to be problematic when deployed in a higher-stakes setting. The challenge here is to design a variant of reinforcement learning that would not result in this kind of behavior; intuitively, the challenge would be to design the system to pursue some actual goal in the environment that is only indirectly observable, instead of pursuing problematic proxy measures of that goal (such as a “hackable” reward signal).

It appears to me that work on challenges like the above is possible in the near term, and could be useful in several ways. Solutions to these problems could turn out to directly reduce accident risks from transformative AI systems developed in the future, or could be stepping stones toward techniques that could reduce these risks; work on these problems could clarify desirable properties of present-day systems that apply equally well to systems developed in the longer-term; or work on these problems today could help to build up the community of people who will eventually work on risks posed by longer-term development, which would be difficult to do in the absence of concrete technical challenges.

I preliminarily feel that there is also useful work to be done today in order to reduce future misuse risks and provide useful analysis of strategic and policy considerations.

As mentioned above, I would like to see more institutions working on considering different potential scenarios with respect to transformative AI; considering how governments, corporations, and individual researchers should react in those scenarios; and working with machine learning researchers to identify potential signs that particular scenarios are becoming more likely.

I think it’s worth being careful about funding this sort of work, since it’s possible for it to backfire. My current impression is that government regulation of AI today would probably be unhelpful or even counterproductive (for instance by slowing development of AI systems, which I think currently pose few risks and do significant good, and/or by driving research underground or abroad). If we funded people to think and talk about misuse risks, I’d worry that they’d have incentives to attract as much attention as possible to the issues they worked on, and thus to raise the risk of such premature/counterproductive regulation.

With that said, I believe that potential risks have now received enough attention – some of which has been unfortunately exaggerated in my view – that premature regulation and/or intervention by government agencies is already a live risk. I’d be interested in the possibility of supporting institutions that could provide thoughtful, credible, public analysis of whether and when government regulation/intervention would be advisable, even if it meant simply making the case against such things for the foreseeable future. I think such analysis would likely improve the quality of discussion and decision-making, relative to what will happen without it.

I also think that technical work related to accident risks – along the lines discussed above – could be indirectly useful for reducing misuse risks as well. Currently, it appears to me that different people in the field have very different intuitions about how serious and challenging accident risks are. If it turns out that there are highly promising paths to reducing accident risks – to the point where the risks look a lot less serious – this development could result in a beneficial refocusing of attention on misuse risks. (If, by contrast, it turns out that accident risks are large and present substantial technical challenges, this makes work on such risks extremely valuable.)

Other notes on tractability.

  • I’ve long worried that it’s simply too difficult to make meaningful statements (even probabilistic ones) about the future course of technology and its implications. However, I’ve gradually changed my view on this topic, partly due to reading I’ve done on personal time. It will be challenging to assemble and present the key data points, but I hope to do so at some point this year.
  • Much of our overarching goal for this cause, in the near term, is to support an increase in the number of people – particularly people with strong relevant technical backgrounds – thinking through how to reduce potential risks. Even if the specific technical, strategic and other work we support does not prove useful, helping to support a growing field in this way could be. With that said, I think we will accomplish this goal best if the people we support are doing good and plausibly useful work.

Bottom line. I think there are real questions around the extent to which there is work worth doing today to reduce potential risks from advanced artificial intelligence. That said, I see a reasonable amount of potential if there were more people and institutions focused on the relevant issues; given the importance and neglectedness of this cause, I think that’s sufficient to prioritize it highly.

Some Open-Phil-specific considerations

Networks

I consider this a challenging cause. I think it would be easy to do harm while trying to do good. For example:

  • Trying to raise the profile of potential risks could contribute (and, I believe, has contributed to some degree) to non-nuanced or inaccurate portrayals of risk in the media, which in turn could raise the risks of premature and/or counterproductive regulation. In addition, raising such risks (or being perceived as doing so) could – in turn – cause many AI and machine learning researchers who oppose such regulation to become hostile to the idea of discussing potential risks.
  • Encouraging particular lines of research without sufficient input and buy-in from leading AI and machine learning researchers could be not only unproductive but counterproductive. It could lead to people generally taking risk-focused research less seriously. And since leading researchers tend to be extremely busy, getting thorough input from them can be challenging in itself.

I think it is important for someone working in this space to be highly attentive to these risks. In my view, one of the best ways to achieve this is to be as well-connected as possible to the people who have thought most deeply about the key issues, including both the leading researchers in AI and machine learning and the people/organizations most focused on reducing long-term risks.

I believe the Open Philanthropy Project is unusually well-positioned from this perspective:

Time vs. money

One consideration that has made me hesitant about prioritizing this cause is the fact that I see relatively little in the way of truly “shovel-ready” giving opportunities. I list our likely priorities in the next section; I think they are likely to be very time-consuming for staff, and I am unsure of how long it will take before we see as many concrete giving opportunities as we do in some of our other focus areas.

By default, I prefer to prioritize causes with significant existing “shovel-ready” opportunities and minimal necessary time commitment, because I consider the Open Philanthropy Project to be short on capacity relative to funding at this stage in our development.

However, I think the case for this cause is compelling enough to outweigh this consideration, and I think a major investment of senior staff time this year could leave us much better positioned to find outstanding giving opportunities in the future.

Our plans

For the last couple of months, we have focused on:

  • Talking to as many people as possible in the relevant communities, particularly leading researchers in AI and machine learning, in order to get feedback on our thinking, deepen our understanding of the relevant issues, and ensure that we have open channels of communication with them. Some high-level notes from these conversations are below.
  • Developing our communications strategy for this topic, including this series of blog posts.
  • Investigating the few potential “shovel-ready grants” (by which I mean grants we can investigate and recommend with relatively low time investments) we’re aware of. We will be publishing more about these later.
  • Working with several technical advisors to begin to get a sense of what the most important concrete, known technical challenges are. Our hope is to get to the point of being able to offer substantial funding to support work on the most important challenges. We’re beginning with close contacts and planning to broaden the conversation about the most important technical challenges from there.
  • Working with close technical advisors to flesh out the key considerations around likely timelines to transformative AI. We expect to continue this work, hopefully with an increasingly broad set of researchers engaging in the discussions.
  • Having initial conversations about what sorts of misuse risks we should be most concerned about, and what sorts of strategic and policy considerations seem most important, in order to lay the groundwork for finding potential grantees in this category.
  • Seeking past cases in which philanthropists helped support the growth of technical fields, to see what we can learn.

Ultimately, we expect to seek giving opportunities in the following categories:

  • “Shovel-ready” grants to existing organizations and researchers focused on reducing potential risks from advanced artificial intelligence.
  • Supporting substantial work on the most important technical challenges related to reducing accident risks. This could take the form of funding academic centers, requests for proposals, convenings and workshops, and/or individual researchers.
  • Supporting thoughtful, nuanced, independent analysis seeking to help inform discussions of key strategic and policy considerations for reducing potential risks, including misuse risks.
  • “Pipeline building”: supporting programs, such as fellowships, that can increase the total number of people who are deeply knowledgeable about technical research on artificial intelligence and machine learning, while also being deeply versed in issues relevant to potential risks.
  • Other giving opportunities that we come across, including those that pertain to AI-relevant issues other than those we’ve focused on in this post (some such issues are listed above).

Getting to this point will likely require a great deal more work and discussion – internally and with the relevant communities more broadly. It could be a long time before we are recommending large amounts of giving in this area, and I think that allocating significant senior staff time to the cause will speed our work considerably.

Some overriding principles for our work

As we work in this space, we think it’s especially important to follow a few core principles:

Don’t lose sight of the potential benefits of AI, even as we focus on mitigating risks

Our work is focused on potential risks, because this is the aspect of AI research that seems most neglected at the moment. However, as stated above, I see many ways in which AI has enormous potential to improve the world, and I expect the consequences of advances in AI to be positive on balance. It is important to act and communicate accordingly.

Deeply integrate people with strong technical expertise in our work

The request for proposals we co-funded last year employed an expert review panel for selecting grantees. We wouldn’t have participated if it had involved selecting grantees ourselves with nontechnical staff. We believe that AI and machine learning researchers are the people best positioned to make many assessments that will be important to us, such as which technical problems seem tractable and high-potential and which researchers have impressive accomplishments.

Seek a lot of input, and reflect a good deal, before committing to major grants and other activities

As stated above, I consider this a challenging cause, where well-intentioned actions could easily do harm. We are seeking to be thoroughly networked and to seek substantial advice on our activities from a range of people, both AI and machine learning researchers and people focused on reduction of potential risks.

Support work that could be useful in a variety of ways and in a variety of scenarios, rather than trying to make precise predictions

I don’t think it’s possible to have certainty, today, about when we should expect transformative AI, what form we should expect it to take, and/or what the consequences will be. We have a preference for supporting work that seems robustly likely to be useful. In particular, one of our main goals is to support an increase in the number of people – particularly people with strong relevant technical backgrounds – dedicated to thinking through how to reduce potential risks.

Distinguish between lower-stakes, higher-stakes, and highest-stakes potential risks

There are many imaginable risks of advanced artificial intelligence. Our focus is likely to be on those that seem to have the very highest stakes, to the point of being potential global catastrophic risks. In our view currently, that means misuse risks and accident risks involving transformative AI. We also consider neglectedness (we prefer to work on risks receiving less attention from others) and tractability (we prefer to work on risks where it seems there is useful work to be done today that can help mitigate them).

Notes on AI and machine learning researchers’ views on the topics discussed here

Over the last couple of months, we have been reaching out to AI and machine learning researchers that we don’t already have strong relationships with in order to discuss our plans and background views and get their feedback. We have put particular effort into seeking out skeptics and potential critics. As of today, we have requested 35 conversations along these lines and had 25. About three-fourths of these conversations have been with tenure-track academics or senior researchers at private labs, and the remainder have been with students or junior researchers at top AI and machine learning departments and private labs.

We’ve heard a diverse set of perspectives. Conversations were in confidence and often time-constrained, so we wouldn’t feel comfortable attributing specific views to specific people. Speaking generally, however, it seems to us that:

  • We encountered fewer strong skeptics of this cause than we expected to, given our previous informal impression that there are many researchers who are dismissive of potential risks from advanced artificial intelligence. That said, we spoke to a couple of highly skeptical researchers, and a few high-profile researchers who we think might be highly skeptical declined to speak with us.
  • Most of the researchers we talked to did not seem to have spent significant time or energy engaging with questions around potential risks from advanced artificial intelligence. To the extent they had views, most of the people we talked to seemed generally supportive of the views and goals we’ve laid out in this post (though this does not at all mean that they would endorse everything we’ve said).
  • Overall, these conversations caused us to update slightly positively on the promise of this cause and our plans. We hope to have many more conversations with AI and machine learning researchers in the coming months to deepen our understanding of the different perspectives in the field.

Risks and reservations

I see much room for debate in the decision to prioritize this cause as highly as we are. I have discussed most of the risks and reservations I see in this post and the ones preceding it. Here I list the major ones in one place. In this section, my goal is to provide a consolidated list of risks and reservations, but not necessarily to give my comprehensive take on each.

  • As discussed previously, I assign a nontrivial probability (at least 10% with moderate robustness, at least 1% with high robustness) to the development of transformative AI within the next 20 years. I feel I have thought deeply about this question, with access to strong technical advisors, and that we’ve collected what information we can, though I haven’t been able to share all important inputs into my thinking publicly. I recognize that our information is limited, and my take is highly debatable.
  • I see a risk that our thinking is distorted by being in an “echo chamber,” and that our views on the importance of this cause are overly reinforced by our closest technical advisors and by the effective altruism community. I’ve written previously about why I don’t consider this a fatal concern, but it does remain a concern.
  • I do not want to exacerbate what I see as an unfortunate pattern, to date, of un-nuanced and inaccurate media portrayals of potential risks from advanced artificial intelligence. I think this could lead to premature and/or counterproductive regulation, among other problems. We hope to communicate about our take on this cause with enough nuance to increase interest in reducing risks, without causing people to view AI as more threatening than positive.
  • I think the case that this cause is neglected is fairly strong, but leaves plenty of room for doubt. In particular, the cause has received attention from some high-profile people, and multiple well-funded AI labs and many AI researchers have expressed interest in doing what they can to reduce potential risks. It’s possible that they will end up pursuing essentially all relevant angles, and that the activities listed above will prove superfluous.
  • I’m mindful of the possibility that it might be futile to make meaningful predictions, form meaningful plans, and do meaningful work to reduce fairly far-off and poorly-understood potential risks.
  • I recognize that it’s debatable how important accident risks are. It’s possible that preventing truly catastrophic accidents will prove to be relatively easy, and that early work will look in hindsight like a poor use of resources.

With all of the above noted, I think it is important that a philanthropist in our position be willing to take major risks, and prioritizing this cause is one that I see as very worth taking.

Hits-based Giving

One of our core values is our tolerance for philanthropic “risk.” Our overarching goal is to do as much good as we can, and as part of that, we’re open to supporting work that has a high risk of failing to accomplish its goals. We’re even open to supporting work that is more than 90% likely to fail, as long as the overall expected value is high enough.

And we suspect that, in fact, much of the best philanthropy is likely to fail. We suspect that high-risk, high-reward philanthropy could be described as a “hits business,” where a small number of enormous successes account for a large share of the total impact — and compensate for a large number of failed projects.

If this is true, I believe it calls for approaching our giving with some counterintuitive principles — principles that are very different from those underlying our work on GiveWell. In particular, if we pursue a “hits-based” approach, we will sometimes bet on ideas that contradict conventional wisdom, contradict some expert opinion, and have little in the way of clear evidential support. In supporting such work, we’d run the risk of appearing to some as having formed overconfident views based on insufficient investigation and reflection.

In fact, there is reason to think that some of the best philanthropy is systematically likely to appear to have these properties. With that said, we think that being truly overconfident and underinformed would be extremely detrimental to our work; being well-informed and thoughtful about the ways in which we could be wrong is at the heart of what we do, and we strongly believe that some “high-risk” philanthropic projects are much more promising than others.

This post will:

  • Outline why we think a “hits-based” approach is appropriate.
  • List some principles that we think are sound for much decision-making, but — perhaps counterintuitively — not appropriate for hits-based giving.
  • List principles that we think are helpful for making sure we focus on the best possible high-risk opportunities.

There is a natural analogy here to certain kinds of for-profit investing, and there is some overlap between our thinking and the ideas Paul Graham laid out in a 2012 essay, Black Swan Farming.

1. The basic case for hits-based giving

Conceptually, we’re focused on maximizing the expected value of how much good we accomplish. It’s often not possible to arrive at a precise or even quantified estimate of expected value, but the concept is helpful for illustrating what we’re trying to do. Hypothetically, and simplifying quite a bit, we would see the following opportunities as equally promising: (1) a $1 million grant that would certainly prevent exactly 500 premature deaths; (2) a $1 million grant that would have a 90% chance of accomplishing nothing and a 10% chance of preventing 5000 premature deaths. Both would have an expected value of preventing 500 premature deaths. As this example illustrates, an “expected value” focus means we do not have a fundamental preference for low-risk philanthropy or high-risk, potentially transformative philanthropy. We can opt for either, depending on the details. As a side note, most other funders we‘ve met have strong opinions on whether it’s better to take big risks or fund what’s reliable and proven; we may be unusually agnostic on this question.

That said, I see a few basic reasons to expect that an “expected value” approach will often favor high-risk and potentially transformative giving.

1. History of philanthropy. We previously gave a high-level overview of some major claimed successes for philanthropy. Since then, we’ve investigated this topic further via our History of Philanthropy project, and we expect to publish an updated summary of what we’ve learned by the end of 2016. One of our takeaways is that there are at least a few cases in which a philanthropist took a major risk — funding something that there was no clear reason to expect to succeed — and ended up having enormous impact, enough to potentially make up for many failed projects.

Here are some particularly vivid examples (note that these focus on magnitude of impact, rather than on whether the impact was positive):

  • The Rockefeller Foundation invested in research on improving agricultural productivity in the developing world, which is now commonly believed to have been the catalyst for a “Green Revolution” that Wikipedia states is “credited with saving over a billion people from starvation.” (The Wikipedia article discusses the role of the Rockefeller Foundation, as does this post on the HistPhil blog, which is supported by the Open Philanthropy Project.)
  • In The Birth of the Pill, Jonathan Eig credits philanthropist and feminist Katharine McCormick — advised by Margaret Sanger — with being the sole funder of crucial early-stage research leading to the development of the combined oral contraceptive pill, now one of the most common and convenient birth control methods.
  • In The Rise of the Conservative Legal Movement, Prof. Steve Teles argues that conservatives put a great deal of funding into long-term, high-risk goals with no way of predicting their success. He also argues that their ultimate impact was to profoundly change the way the legal profession operates and the general intellectual stature of political conservatism.

If accurate, these stories would imply that philanthropy — and specifically, philanthropy supporting early-stage research and high-risk projects — played a major role in some of the more significant developments of the last century.[1]There is some possibility of survivorship bias, and a question of how many failed projects there were for each of these successes. However, I note that the examples above aren’t drawn from a huge space of possibilities. (All of the philanthropists covered above would probably — prior to their … Continue reading A philanthropic “portfolio” containing one of these projects, plus a large number of similar failed projects, would probably have a very strong overall performance, in terms of impact per dollar.

2. Comparative advantage. When trying to figure out how to give as well as possible, one heuristic to consider is, “what are philanthropists structurally better-suited (and worse-suited) to do compared with other institutions?” Even major philanthropists tend to have relatively less funding available than governments and for-profit investors, but philanthropists are far less constrained by the need to make a profit or justify their work to a wide audience. They can support work that is very “early,” such as new and unproven ideas or work that is likely to take many decades to have an impact. They can support a number of projects that fail in order to find the ones that succeed. They can support work that requires great depth of knowledge to recognize as important and is hard to justify to a wide audience. All of these things seem to suggest that when philanthropists are funding low-probability, high-upside projects, they’re doing what they do best, relative to other institutions.

3. Analogy to for-profit investing. Many forms of for-profit investing, such as venture capital investing, are “hits businesses.” For some description of this, see the article I mentioned previously. Philanthropy seems similar in some relevant ways to for-profit investing: specifically, it comes down to figuring out how to allocate a set amount of funds between projects that can have a wide variety of outcomes. And many of the differences between for-profit investing and philanthropy (as discussed above) seem to imply that hits-based giving is even more likely to be appropriate than hits-based investing.

2. “Anti-principles” for hits-based giving

This section discusses principles that we think are sound for much decision-making, but not appropriate for hits-based giving. For clarity, these are phrased as “We don’t _____” where _____ is the principle we see as a poor fit with this approach.

A common theme in the items below is that for a principle to be a good fit, it needs to be compatible with the best imaginable giving opportunities — the ones that might resemble cases listed in the previous section, such as the Green Revolution. Any principle that would systematically discourage the biggest hits imaginable is probably not appropriate for hits-based giving, even if it is a good principle in other contexts.

We don’t: require a strong evidence base before funding something. Quality evidence is hard to come by, and usually requires a sustained and well-resourced effort. Requiring quality evidence would therefore be at odds with our interest in neglectedness. It would mean that we were generally backing ideas that others had already explored and funded thoroughly — which would seem to decrease the likelihood of “hit”-sized impact from our participation. And some activities, such as funding work aiming to influence policy or scientific research, are inherently hard to “test” in predictively valid ways. It seems to me that most past cases of philanthropic “hits” were not evidence-backed in the sense of having strong evidence directly predicting success, though evidence probably did enter into the work in less direct ways.

We don’t: seek a high probability of success. In my view, strong evidence is usually needed in order to justifiably assign a high probability to having a reasonably large positive impact. As with venture capital, we need to be willing to back many failures per success — and the successes need to be big enough to justify this.

We don’t: defer to expert opinion or conventional wisdom, though we do seek to be informed about them. Similar to the above point, following expert opinion and conventional wisdom is likely to cut against our goal of seeking neglected causes. If we funded early groundwork for changing expert opinion and/or conventional wisdom on an important topic, this would be a strong candidate for a “hit.” We do think it would be a bad sign if no experts (using the term broadly to mean “people who have a great deal of experience engaging with a given issue”) agreed with our take on a topic, but when there is disagreement between experts, we need to be willing to side with particular ones. In my view, it’s often possible to do this productively by learning enough about the key issues to determine which arguments best fit our values and basic epistemology.

We don’t: avoid controversial positions or adversarial situations. All else equal, we would rather not end up in such situations, but making great effort to avoid them seems incompatible with a hits-based approach. We’re sympathetic to arguments of the form, “You should be less confident in your position when intelligent and well-meaning people take the opposite side” and “It’s unfortunate when two groups of people spend resources opposing each other, resulting in no net change, when they instead could have directed all of their resources to something they agree on, such as directly helping those in need.” We think these arguments give some reason to prefer GiveWell’s top charities. But we feel they need to be set aside when aiming for “hits.”

We feel many “hits” will involve getting a multiplier on our impact by changing social norms or changing key decision-makers’ opinions. And our interest in neglectedness will often point us to issues where social norms, or well-organized groups, are strongly against us. None of the “hits” listed above were without controversy. Note that the combined oral contraceptive is an example of something that was highly controversial at the time (leading, in my view, to the necessary research being neglected by government and other funders) and is now accepted much more broadly; this, to me, is a key part of why it has been such a momentous development.

We don’t: expect to be able to fully justify ourselves in writing. Explaining our opinions in writing is fundamental to the Open Philanthropy Project’s DNA, but we need to be careful to stop this from distorting our decision-making. I fear that when considering a grant, our staff are likely to think ahead to how they’ll justify the grant in our public writeup and shy away if it seems like too tall an order — in particular, when the case seems too complex and reliant on diffuse, hard-to-summarize information. This is a bias we don’t want to have. If we focused on issues that were easy to explain to outsiders with little background knowledge, we’d be focusing on issues that likely have broad appeal, and we’d have more trouble focusing on neglected areas.

A good example is our work on macroeconomic stabilization policy: the issues here are very complex, and we’ve formed our views through years of discussion and engagement with relevant experts and the large body of public argumentation. The difficulty of understanding and summarizing the issue is related, in my view, to why it is such an attractive cause from our perspective: macroeconomic stabilization policy is enormously important but quite esoteric, which I believe explains why certain approaches to it (in particular, approaches that focus on the political environment as opposed to economic research) remain neglected.

Process-wise, we’ve been trying to separate our decision-making process from our public writeup process. Typically, staffers recommend grants via internal writeups. Late in our process, after decision-makers have approved the basic ideas behind the grant, other staff take over and “translate” the internal writeups into writeups that are suitable to post publicly. One reason I’ve been eager to set up our process this way is that I believe it allows people to focus on making the best grants possible, without worrying at the same time about how the grants will be explained.

A core value of ours is to be open about our work. But “open” is distinct from “documenting everything exhaustively” or “arguing everything convincingly.” More on this below.

We don’t: put extremely high weight on avoiding conflicts of interest, intellectual “bubbles” or “echo chambers.” There will be times when we see a given issue very differently from most people in the world, and when the people we find most helpful on the issue will be (not coincidentally) those who see the issue similarly. This can lead to a risk of putting ourselves in an intellectual “bubble” or “echo chamber,” an intellectually insulated set of people who reinforce each others’ views, without bringing needed alternative perspectives and counterarguments.

In some cases, this risk may be compounded by social connections. When hiring specialists in specific causes, we’ve explicitly sought people with deep experience and strong connections in a field. Sometimes, that means our program officers are friends with many of the people who are best suited to be our advisors and grantees.

Other staff, including myself, specialize in choosing between causes rather than in focusing on a specific cause. The mission of “choosing between causes to do the most good possible” is itself an intellectual space with a community around it. Specifically, many of our staff — including myself — are part of the effective altruism community, and have many social ties in that community.

As a result, it sometimes happens that it’s difficult to disentangle the case for a grant from the relationships around it.[2]As of August 2017, we no longer write publicly about personal relationships with partner organizations. This blog post was updated to reflect this change in practice. When these situations occur, there’s a greatly elevated risk that we aren’t being objective, and aren’t weighing the available evidence and arguments reasonably. If our goal were to find the giving opportunities most strongly supported by evidence, this would be a major problem. But the drawbacks for a “hits-based” approach are less clear, and the drawbacks of too strongly avoiding these situations would, in my view, be unacceptable.

To use myself as an example:

  • My strong interest in effective altruism and impact-focused giving has led me to become friends — and live in the same house — with similarly interested people.
  • I spend a lot of time with the people I have found to most strongly share my values and basic epistemology, and to be most interesting and valuable as intellectual peers.
  • If I had a policy of asking my friends to recuse themselves from advising me or seeking support from the Open Philanthropy Project, this would mean disallowing input from some of the people whose opinions I value most.
  • Under a “hits-based” approach, we can expect the very few best projects to account for much (or most) of our impact. So disallowing ideas from some of the people who most closely share our values could dramatically lower the expected value of our work.

This issue is even more pronounced for some of our other staff members, since the staffers who are responsible for investigating funding opportunities in a given area tend to be the ones with the deepest social connections in the relevant communities.

To be clear, I do not believe we should ignore the risks of intellectual “bubbles” or conflicts of interest. To mitigate these risks, we seek to (a) always disclose relevant connections to decision-makers; (b) always make a strong active effort to seek out alternative viewpoints before making decisions, including giving strong consideration to the best counter-arguments we can identify; (c) aim for key staff members to understand the most important issues themselves, rather than relying on the judgment of friends and advisors, to the extent that this is practical; (d) always ask ourselves how our relationships might be distorting our perception of a situation; (e) make sure to seek input from staff who do not have relevant conflicts of interest or social relationships.

But after doing all that, there still will be situations where want to recommend a grant that is strongly supported by many of our friends, while attracting little interest from those outside our intellectual and social circles. I think if we avoided recommending such grants, we would be passing over some of our best chances at impact — an unacceptable cost for a “hits-based” approach.

We don’t: avoid the superficial appearance — accompanied by some real risk — of being overconfident and underinformed.

When I picture the ideal philanthropic “hit,” it takes the form of supporting some extremely important idea, where we see potential while most of the world does not. We would then provide support beyond what any other major funder could in order to pursue the idea and eventually find success and change minds.

In such situations, I’d expect the idea initially to be met with skepticism, perhaps even strong opposition, from most people who encounter it. I’d expect that it would not have strong, clear evidence behind it (or to the extent it did, this evidence would be extremely hard to explain and summarize), and betting on it therefore would be a low-probability play. Taking all of this into account, I’d expect outsiders looking at our work to often perceive us as making a poor decision, grounded primarily in speculation, thin evidence and self-reinforcing intellectual bubbles. I’d therefore expect us to appear to many as overconfident and underinformed. And in fact, by the nature of supporting an unpopular idea, we would be at risk of this being true, no matter how hard we tried (and we should try hard) to seek out and consider alternative perspectives.

I think that a “hits-based” approach means we need to be ready to go forward in such situations and accept the risks that come with them. But, as discussed below, I think there are better and worse ways to do this, and important differences between engaging in this sort of risk-taking and simply pursuing self-serving fantasies.

3. Working principles for doing hits-based giving well

The previous section argues against many principles that are important in other contexts, and that GiveWell fans might have expected us to be following. It is reasonable to ask — if one is ready to make recommendations that aren’t grounded in evidence, expert consensus, or conventional wisdom — is there any principled way to distinguish between good and bad giving? Or should we just be funding what we’re intuitively excited about?

I think it’s hard to say what sort of behavior is most likely to lead to “hits,” which by their nature are rare and probably hard to predict. I don’t know enough about the philanthropists who have been behind past “hits” to be able to say much with confidence. But I can outline some principles we’re working with to try to do “hits-based” giving as well as possible.

Assess importance, neglectedness and tractability. These are the key criteria of the Open Philanthropy Project. I think each of them, all else equal, makes “hits” more likely, and each in isolation can often be assessed fairly straightforwardly. Much of the rest of this section pertains to how to assess these criteria in difficult situations (for example, when there is no expert consensus or clear evidence).

Consider the best and worst plausible cases. Ideally, we’d assign probabilities to each imaginable outcome and focus on the overall expected value. In practice, one approximation is to consider how much impact a project would have if it fully achieved its long-term goals (best plausible case), and how much damage it could do if it were misguided (worst plausible case). The latter gives us some indication of how cautiously we should approach a project, and how much work we should put into exploring possible counterarguments before going forward. The former can serve as a proxy for importance, and we’ve largely taken this approach for assessing importance so far. For example, see the Google spreadsheets linked here, and particularly our estimates of the value of major policy change on different issues.

Goals can often be far more achievable than they appear early on (some examples here), so I believe it’s often worth aiming for a worthy but near-impossible-seeming goal. If successes are rare, it matters a great deal whether we choose to aim for reasonably worthy goals or maximally impactful ones. Despite the uncertainty inherent in this sort of giving, I believe that the question, “How much good could come of the best case?” will have very different answers for different giving opportunities.

Aim for deep understanding of the key issues, literatures, organizations, and people around a cause, either by putting in a great deal of work or by forming a high-trust relationship with someone else who can. If we support projects that seem exciting and high-impact based on superficial understanding, we’re at high risk of being redundant with other funders. If we support projects that seem superficially exciting and high-impact, but aren’t being supported by others, then we risk being systematically biased toward projects that others have chosen not to support for good reasons. By contrast, we generally aim to support projects based on the excitement of trusted people who are at a world-class level of being well-informed, well-connected, and thoughtful in relevant ways.

Achieving this is challenging. It means finding people who are (or can be) maximally well-informed about issues we’ll never have the time to engage with fully, and finding ways to form high-trust relationships with them. As with many other philanthropists, our basic framework for doing this is to choose focus areas and hire staff around those focus areas. In some cases, rather than hiring someone to specialize in a particular cause, we try to ensure that we have a generalist who puts a great deal of time and thought into an area. Either way, our staff aim to become well-networked and form their own high-trust relationships with the best-informed people in the field.

I believe that the payoff of all of this work is the ability to identify ideas that are exciting for reasons that require unusual amounts of thought and knowledge to truly appreciate. That, to me, is a potential recipe for being positioned to support good ideas before they are widely recognized as good, and thus to achieve “hits.”

Minimize the number of people setting strategy and making decisions. When a decision is made as a compromise between a large number of people with very different perspectives, it may have a high probability of being a defensible and reasonable decision, but it seems quite unlikely to be an extraordinarily high-upside decision. I would guess that the latter is more associated with having a distinctive perspective on an issue based on deep thought and context that would be hard to fully communicate to others. Another way to put this is that I’d be more optimistic about a world of individuals pursuing ideas that they’re excited about, with the better ideas gaining traction as more work is done and value is demonstrated, than a world of individuals reaching consensus beforehand on which ideas to pursue.

Formally, grant recommendations currently require signoff from Cari Tuna and myself before they go forward. Informally, our long-term goal is to defer to the staff who know the most about a given case, such that strategy, priorities and grants for a given cause are largely determined by the single person who is most informed about the cause. This means, for example, that we aspire for our criminal justice reform work to be determined by Chloe Cockburn, and our farm animal welfare work to be determined by Lewis Bollard. As stated above, we expect that staff will seek a lot of input from other people, particularly from field experts, but it is ultimately up to them how to consider that input.

Getting to that goal means building and maintaining trust with staff, which in turn means asking them a lot of questions, expecting them to explain a significant amount of their thinking, and hashing out key disagreements. But we never require them to explain all of their thinking; instead, we try to drill down on the arguments that seem most noteworthy or questionable to us. Over time, we aim to lower our level of engagement and scrutiny as we build trust.

I hope to write more about this basic approach in the future.

When possible, support strong leadership with no strings (or minimal strings) attached, rather than supporting unremarkable people/organizations to carry out plans that appeal to us. The case for this principle is an extension of the case for the previous principle, and fits into the same basic approach that I hope to write more about in the future. It’s largely about shifting decision-making power to the people who have the deepest context and understanding.

Understand the other funders in a space, and hesitate to fund things that seem like a fit for them. This is an aspect of “Aim for deep understanding …” that seems worth calling out explicitly. When we fund something that is a conceptual fit for another funder, there’s a good chance that we are either (a) moving only a little more quickly than the other funder, and thus having relatively little impact; or (b) funding something that another funder declined to fund for good reasons. Having a good understanding of the other funders in a space, and ideally having good relationships with them, seems quite important.

Be wary of giving opportunities that seem unlikely (from heuristics) to be neglected. This is largely an extension of the previous principle. When an idea seems to match quite well with conventional wisdom or expert consensus, or serves a particular well-resourced interest, this raises questions about why it hasn’t already attracted support from other funders, and whether it will stay under-funded for long.

Bottom line. The ideal giving opportunity, from my perspective, looks something like: “A trusted staff member with deep knowledge of cause X is very excited to support — with few or no strings attached — the work of person Y, who has an unusual perspective and approach that few others appreciate. The staff member could easily imagine this approach having a massive impact, even if it doesn’t seem likely to. When I first hear the idea, it sounds surprising, and perhaps strange, counterintuitive or unattractive, but when I question the staff member about possible failure modes, concerns, and apparent gaps in the case for the idea, it seems that they are already well-informed and thoughtful about the questions I ask.” This basic setup seems to me to maximize odds of supporting important work that others won’t, and having a chance down the line of changing minds and getting a “hit.”

4. Reconciling a hits-based approach with being open about our work

A core value of ours is to be open about our work. Some reasons for this:

  • We’d like others to be able to take advantage of what we’ve learned, in order to better inform themselves.
  • We’d like others to be able to understand, question and critique our thinking.
  • We’d like there to be a more sophisticated public dialogue about how to give well.

There is some tension between these goals and the fact that, as discussed above, we expect to do many things that are hard to justify in a convincing way to outsiders. We expect that our writeups will frequently not be exhaustive or highly persuasive, and will often leave readers unsure of whether we’ve made a good decision.

However, we think there is room to achieve both goals — being open and having a “hits-based” approach — to a significant degree. For a given decision, we aim to share our thinking to the point where readers can understand:

  • The major pros and cons we perceive.
  • The premises that are key to our views.
  • The process we’ve followed.
  • What sorts of things the reader might do in order to come to the point of confidently agreeing or disagreeing with our thinking, even if they aren’t sure how to feel based on a writeup alone.

A couple of examples:

We believe that this sort of openness can accomplish a lot in terms of the goals above, even though it often won’t be exhaustive or convincing on its own.

In general, this discussion might help clarify why the Open Philanthropy Project is aimed primarily at major philanthropists — people who have the time to engage deeply with the question of where to give — rather than at individual donors. Individual donors do, of course, have the option to trust us and support us even when our views seem unusual and hard to justify. But for those who don’t already trust us, our writeups (unlike, in my view, GiveWell’s writeups) will not always provide sufficient reason to take us at our word.

5. “Hits-based mentality” vs. “arrogance”

As discussed above, I believe “hits-based giving” will often entail the superficial appearance — and a real risk of — having overconfident views based on insufficient investigation and reflection. I use “arrogance” as shorthand for the latter qualities.

However, I think there are important, and observable, differences between the two. I think a “hits-based mentality” can be a reasonable justification for some behaviors commonly associated with arrogance — in particular, putting significant resources into an idea that is controversial and unsupported by strong evidence or expert consensus — but not for other behaviors.

Some specific differences that seem important to me:

Communicating uncertainty. I associate arrogance with being certain that one is right, and communicating accordingly. I find it arrogant when people imply that their favorite causes or projects are clearly the best ones, and especially when they imply that work being done by other people, on other causes, is unimportant. A hits-based mentality, by contrast, is consistent both with being excited about an idea and being uncertain about it. We aim to clearly communicate our doubts and uncertainties about our work, and to acknowledge there could be much we’re getting wrong, even as we put resources into our ideas.

Trying hard to be well-informed. I associate arrogance with jumping to conclusions based on limited information. I believe a well-executed “hits-based mentality” involves putting significant work into achieving a solid understanding of the case both for and against one’s ideas. We aspire to think seriously about questions and objections to our work, even though we won’t be able to answer every one convincingly for all audiences.

Respecting those we interact with and avoiding deception, coercion, and other behavior that violates common-sense ethics. In my view, arrogance is at its most damaging when it involves “ends justify the means” thinking. I believe a great deal of harm has been done by people who were so convinced of their contrarian ideas that they were willing to violate common-sense ethics for them (in the worst cases, even using violence).

As stated above, I’d rather live in a world of individuals pursuing ideas that they’re excited about, with the better ideas gaining traction as more work is done and value is demonstrated, than a world of individuals reaching consensus on which ideas to pursue. That’s some justification for a hits-based approach. But with that said, I’d also rather live in a world where individuals pursue their own ideas while adhering to a baseline of good behavior and everyday ethics than a world of individuals lying to each other, coercing each other, and actively interfering with each other to the point where coordination, communication and exchange break down.

On this front, I think our commitment to being honest in our communications is important. It reflects that we don’t think we have all the answers, and we aren’t interested in being manipulative in pursuit of our views; instead, we want others to freely decide, on the merits, whether and how they want to help us in our pursuit of our mission. We aspire to simultaneously pursue bold ideas and remember how easy it would be for us to be wrong.

Footnotes

Footnotes
1 There is some possibility of survivorship bias, and a question of how many failed projects there were for each of these successes. However, I note that the examples above aren’t drawn from a huge space of possibilities. (All of the philanthropists covered above would probably — prior to their gifts — have made fairly short lists of the most prominent philanthropists interested in their issues.)
2 As of August 2017, we no longer write publicly about personal relationships with partner organizations. This blog post was updated to reflect this change in practice.

Our Grantmaking So Far: Approach and Process

We’ve written previously about our approach to choosing focus areas for the Open Philanthropy Project, and we’ve described the advantages that working within causes (as opposed to being open to making grants in any area at any time) has for grantmaking. To date, however, we haven’t said much about our grantmaking process itself.

As part of my role at Open Philanthropy, I manage the logistics and overall process of how we make grants. In this post, I’ll describe the approach we use to grantmaking within our focus areas, and outline our current process for deciding whether or not to make a particular grant.

For our first couple of years, grantmaking has not been the top priority for the Open Philanthropy Project; so far we have focused most highly on selecting cause areas and building internal capacity (see our most recent update and this blog post for more on how we’re thinking about balancing these with grantmaking). As such, the approach and process outlined in this post are both fairly preliminary — we expect them to change and mature somewhat as we gain more experience as a grantmaking organization.

This post describes:

March 2017 update: Since this blog post was published, our process has largely stayed the same, but we wanted to note a few changes:

  • 50/40/10 rule: In keeping with the philosophy described below, we believe that the best way for us to make outstanding grants is to build trust with program officers who are experts in each area we work in, and rely heavily on their judgment in their own area. The “50/40/10 rule” is a formalization of this idea. It works as follows:
    • At least 50% of each program officer’s grantmaking should be such that Holden and Cari understand and are on board with the case for each grant.
    • At least 90% of the program officer’s grantmaking should be such that Holden and Cari could easily imagine being on board with the grant if they knew more, but may not be persuaded that the grant is a good idea. (When taking the previous bullet point into account, this leaves room for up to 40% of the portfolio to fall in this bucket.)
    • Up to 10% of the program officer’s grantmaking can be done without meeting either of the above two criteria, though there are some basic checks in place to avoid grantmaking that creates risks for Open Philanthropy. We call this “discretionary” grantmaking. Grants in this category generally follow a different, substantially abbreviated approval process. Some examples of discretionary grants are here and here.
  • Internal writeup template: Since this post was published, we have begun using a more formalized template for grant investigators to write about each grant, rather than just a list of questions.
  • Our thinking about openness: As described in a previous blog post, our thinking about openness and information-sharing continues to develop and change. The main substantive change that relates to this post is that we no longer write in detail about every grant we make. Instead, we add basic information about almost all grants to our grants database, and are more likely to write in more detail about grants that are larger, or that we expect to be more controversial or more interesting to our audience.

Our approach to making grants

Our approach to making grants within causes is fairly different from the approach we use to choose which causes to work in. When selecting causes, we believe that thinking about importance, neglectedness, and tractability is a good way for us to identify areas where we can have the greatest impact. We have applied this framework by considering many different areas at a shallow level, investigating the areas that looked promising in more depth, and then selecting focus areas.

In theory, we could use a process like this all the way down to evaluate subfields, organizations, or grant opportunities within a cause, but we feel this would be unnecessarily unwieldy and time-consuming. Indeed, the aim of cause selection is to choose focus areas that a staff member can understand and keep up with in a more holistic, informal-judgment-based way.

We aim to have at least one person on staff who is deeply familiar and highly engaged with each of our focus areas. For some focus areas, such as those with few existing organizations and little existing infrastructure, we think it’s possible to do this by having a generalist staff member get to know the cause. In other fields, we aim to hire specialist program officers with pre-existing knowledge of and networks in the cause (for example, Chloe Cockburn on criminal justice reform and Lewis Bollard on farm animal welfare). Our program officers are immersed in their focus areas: they know the key issues, arguments, and relevant pitfalls, and are well-connected to major organizations and other funders in the area. This lets us make grants in a manner that is both more informed and more flexible than might otherwise be possible. Because we aim to base decisions mostly on the judgment of a person who is highly informed, we are able to make grants where there is not an easily demonstrable evidence base.

Once a program officer is established in their cause, there are a few ways grant opportunities may come to their attention. In many cases, we find out about grants passively, when others within a cause alert us of organizations or projects that match our interests within the space and are seeking funding. Our experience so far with these kind of grants, and our impression that this is a common way for funders to be connected with funding opportunities, make us especially interested in communicating clearly about our interests and priorities within our focus areas.

In other cases, program officers may reach out more actively to identify potential grants. This happens more often when we have a specific project or unusual angle on a cause that we would like to fund.

Once the program officer is aware of a grant opportunity, it is up to them to decide whether or not to pursue it. To a large extent, this is based on their own judgment of what is likely to produce positive results in their field. With that said, our feeling so far is that there are a couple of ‘molds’ grants can fit into that we find particularly compelling:

  • A person or organization working in the space who stands out to us as especially competent and/or aligned with our values, with a reasonable-seeming idea but insufficient funding. When the person or organization is strong enough, we don’t want to worry too much about whether their idea looks strong to us; rather, the idea is mostly evidence that they don’t have all the funding they can productively use.
  • Proposals that build capacity or ‘fill holes’ in the space. Our focus areas are generally fields that we believe should be better-resourced and more fully developed than they presently are; as such, we’re excited about funding projects or organizations that are conspicuously lacking, especially if they provide infrastructure for the field that could enable further growth. We generally try to have a model of what sorts of organizations could exist (for example, see our general model for policy areas), and compare it to what sorts of organizations do exist. Examples of grants motivated by this kind of thinking are our grants to the Future of Life Institute (which supported the first dedicated round of academic grants in a new field) or the Blue Ribbon Study Panel on Biodefense (which supported a panel of experts to consider and report on ways to strengthen U.S. biodefense).

We are also more likely to find a grant promising if either of the below is true, though we generally weigh these factors much less heavily:

  • The grant provides us with an unusually good opportunity to learn about the cause or organization, or to test a hypothesis (some of our early thoughts on ‘giving to learn’ in this post).
  • The grant lets us begin to build and test a working relationship with an organization that we would consider working with more intensively, if the grant went well.

A final important question is whether the grantee is likely to receive funding from other sources, if not from us. In some cases, we may actively reach out to other funders who we believe might be interested in the project, or we may choose to fund only part of the requested amount, in the hopes that other funders will join in. (Taking this approach also lets us test our hypothesis that the grant would not be funded without our support.)

Note that none of the criteria above require that the work we’re supporting have a strong track record or a demonstrable evidence base; we expect to have to exercise informed judgment to determine which grants meet these criteria, rather than relying on formal requirements.

Our decision process for individual grants

The overall process we use to decide whether or not to make a grant is outlined here.

In brief, the person leading investigation into the grant (‘grant investigator’, usually but not always a program officer) goes back and forth between the potential grantee and the Open Philanthropy team in several stages. At each stage, the grant investigator digs further into the details of the proposed grant and discusses remaining uncertainties or concerns with senior Open Philanthropy staff.

Early on, the focus is on getting the basics of the proposed grant. Without spending much time (either their own or the grantee’s), the grant investigator tries to answer questions such as:

  • Why should we make this grant? What’s special about it, how does it relate to our overall strategy and to key needs in the space?
  • How much funding is sought? What could be accomplished at different levels of funding? What will the funding mostly pay for?
  • What is the grantee’s timeline? When do they need to hear back, and when would it be helpful for them to hear back? What is their target for (a) a decision (b) putting out a public announcement involving our support (c) getting the funds?
  • What are other potential revenue sources for this project?
  • What would we expect/hope for from this grant?
  • What are potential risks or downsides of this grant?
  • How does the grantee feel about sharing information publicly?

If the grant investigator is still interested once they have an initial idea of the answers to these questions, they write up what they know (or present it in a meeting) and seek signoff to express what we call “strong interest” in making the grant. “Strong interest” means that we are comfortable with both the grant investigator and the potential grantee putting additional time into discussing the grant, preparing materials, etc. We try to avoid situations where a grantee puts a lot of time into making the case for a grant that we aren’t interested in; that’s why we have a formal requirement of checking in with the team before we have much information about the grant.

Once the grant investigator has signoff to express strong interest, they begin to settle on the details of the grant with the grantee. They might revisit any of the above questions which have not yet been resolved, address suggestions or concerns of ours, or discuss which of several proposed variations on the grant we’re most interested in funding.

In order to recommend that a grant go ahead, the grant investigator needs signoff from both Cari Tuna and Holden Karnofsky (President and Executive Director of the Open Philanthropy Project, respectively). We sometimes also involve another staff member who is responsible for intensive questioning and evaluation of the case, but their formal signoff is not required. The extent to which Holden and Cari involve themselves in a grant decision depends on the focus area, grant size, and the level of trust they have built with the grant investigator. As we wrote when we hired our first cause-specific program officer, our aim over time is to build a high level of trust toward grant investigators. Eventually, we hope that Holden and Cari will only feel the need to spot check a claim or two, or raise questions about a grant to ensure they’ve been considered, with most decisions based primarily on the grant investigator’s judgment.

Overall, we believe that keeping the number of decisionmakers as small as possible allows us to make grants at a high level of sophistication that would be difficult to reach if the case for each grant needed to be understood by a larger number of people. The more trust we build with a given grant investigator, the more capable we become of quickly making decisions that require a high level of knowledge, context and sophistication. And the more we can do this, the more we can make grants that are based on long-term, risk-tolerant thinking and don’t rely on easily verifiable evidence, while still being highly informed.

We try as much as possible to avoid requesting that grantees spend large amounts of time preparing documents for us, especially in the early, exploratory stages of discussion. In some cases, we may request that the grantee prepare a short proposal or similar, or they may send us materials they have already prepared. In general, we spend more time and investigative effort on larger grants, and are more willing to make larger requests of the grantee in these cases.

Similarities to and differences from other foundations

As far as we know, the process laid out above is structurally fairly similar to how many other foundations work, with expert program officers leading most decisions.

However, we believe that there are a few aspects of our approach that are somewhat more unusual:

  • We put a great deal of time and thought into selecting our focus areas, as well as into selecting program officers to lead our work in those areas. (The former is more unusual than the latter.) We believe that careful choices at these stages enable us to make grants much more effectively than we otherwise could.
  • The primary decision-makers work closely and daily with grant investigators, and are checked in with more than once throughout the investigation process. This contrasts with some other foundations, where a Board or benefactor reviews proposed grants at periodic meetings, sometimes leading to long timelines for decision-making and difficulty predicting with confidence which grants will be approved.
  • We prioritize information-sharing to a degree that we haven’t seen at any other foundation. This includes publishing notes of our conversations with experts and grantees as much as possible, writing publicly about what grants we are making (including fairly detailed discussion of why we are making each grant and what risks or potential downsides we see), and making posts like this one. We’ve written previously about some of the challenges that we face in being as transparent as we’d like to be. We continue to think that the net benefits of taking this approach are high.
  • As part of this, instead of requesting proposals or applications from grantees, we do the reverse: we write (and publish) our own summaries of the grants we make. We feel this is a more natural division of labor. When grantees are responsible for preparing writeups, they often need to spend large amounts of time preparing documents and tailoring them to different funders with different interests. With our approach, grant decisions are made based primarily on conversations or shorter documents, while the funder (Open Philanthropy Project) is responsible for creating a written description of the grant laying out the reasons it is attractive to us.

Tradeoffs

The process laid out above can be consistent with very small or very large amounts of investigation, and there are tradeoffs around how much to do. The more investigation we do, the more of our scarce staff capacity we use, and delays in decision-making can also cause problems for grantees trying to plan their activities. In particular, the more we decide to trust the grant investigator, the less investigation we generally need to do.

We have rough working guidelines that suggest putting more investigation into grants when they are larger and higher-stakes. A public version of these guidelines can be found here.

We’re excited to be turning more attention to grantmaking in 2016 and beyond, and look forward to learning more about how to be an effective funder (and sharing what we learn).

A note on ‘recommending’ grants
During our early work on the Open Philanthropy Project (previously called GiveWell Labs), we did some preliminary grantmaking. These grants (example) were usually made by the foundation Good Ventures, at the recommendation of GiveWell.

As of mid-2015, most Open Philanthropy grantmaking comes from a dedicated donor-advised fund (DAF). The legal structure of this fund is such that Open Philanthropy itself does not exert final control over whether a given grant is made or not. Instead, we recommend the grant to the organization that houses the DAF, which then decides whether or not to accept the recommendation.

We use phrases like “Open Philanthropy made a grant of…” when grants are made from the Open Philanthropy DAF at our recommendation.

The Process of Hiring our First Cause-Specific Program Officer

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

Earlier this year, we announced Chloe Cockburn as our incoming Program Officer for criminal justice reform. Chloe started her new role at the end of August.

This hire was the top priority we set in our March update on U.S. policy. It represents the first time we’ve hired someone for a senior, cause-specific role. Chloe will be the primary person responsible for recommending $5+ million a year of grants in this space. As such, hiring Chloe is one of the highest-stakes decisions we’ve made yet for the Open Philanthropy Project, certainly higher-stakes than any particular grant to date. As such, we are writing up a summary of our thinking (including reservations), and the process we ran for this job search.

We also see this blog post as a major part of the case for future grants we make in criminal justice reform. Part of the goal of this process was to hire a person with context, experience, and relationships that go well beyond what it would be realistic to put in a writeup. We expect that future criminal justice reform grants will be subject to a good deal of critical discussion, and accompanied by writeups; at the same time, for readers who want to fully understand the thinking behind our grants, it is important to note that our bigger-picture bet on Chloe’s judgment will be a major input into each grant recommendation in this area.

Note that Chloe reviewed this post.

Continue reading “The Process of Hiring our First Cause-Specific Program Officer”

Key Questions about Philanthropy, Part 3: Making and Evaluating Grants

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

This post is third in a series on fundamental (and under-discussed) questions about philanthropy that we’ve grappled with in starting a grantmaking organization (see previous link for the series intro, and this link for the second installment). This post covers the following questions:

  • When making a grant, should we focus most on evaluating the strategy/intervention, the leadership, or something else? We think both are very important; for a smaller grant we hope to be excited about one or the other, and for a larger grant we hope to thoroughly assess both. A couple of disanalogies between philanthropy and for-profit investing point to a relatively larger role for evaluating strategies/interventions, relative to people. More
  • For a given budget, is it best to make fewer and larger grants or more numerous and smaller grants? We currently lean toward the former. Most of the grants we’ve made so far are either (a) a major grant that we’ve put major time into or (b) a smaller grant that we’ve put less time into, in the hopes of seeding a project that could raise more money down the line. More
  • What sort of paperwork should accompany a grant? Funders often require grantees to complete lengthy writeups about their plans, strengths, weaknesses, and alignment with funder goals. So far, we’ve taken a different approach: we create a writeup ourselves and work informally with the grantee to get the information we need. We do have a standard grant agreement that covers topics such as transparency (setting out our intention to write publicly about the grant) and, when appropriate, research practices (e.g. preregistration and data sharing). More
  • What should the relationship be between different funders? How strongly should we seek collaboration, versus seeking to fund what others won’t? It seems to us that many major funders greatly value collaboration, and often pursue multi-funder partnerships. We don’t fully understand the reasons for this and would like to understand them better. Our instincts tend to run the other way. All else equal, we prefer to fund things that are relatively neglected by other funders. We see a lot of value in informal contact with other funders – in checking in, discussing potential grants, and pitching giving opportunities – but a more formal collaboration with another staffed funder would likely introduce a significant amount of time cost and coordination challenges, and we haven’t yet come across a situation in which that seemed like the best approach. More
  • How should we evaluate the results of our grants? Of all the questions in this series, this is the one we’ve seen the most written about. Our approach is very much case-by-case: for some grants, we find it appropriate to do metrics-driven evaluation with quantifiable targets, while for others we tend to have a long time horizon and high tolerance for uncertainty along the way. More

Continue reading “Key Questions about Philanthropy, Part 3: Making and Evaluating Grants”

Key Questions about Philanthropy, Part 2: Choosing Focus Areas and Hiring Program Staff

Note: The Open Philanthropy Project was formerly known as GiveWell Labs. Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

This post is second in a series on fundamental questions about philanthropy that we’ve grappled with in starting a grantmaking organization (see link for the series intro). In this post, we discuss the following questions:

  • Should a funder set explicit focus areas, and if so, how should they choose the focus areas? We believe it generally makes sense to declare focus areas (i.e., causes that one plans to focus on). While it’s common for philanthropists to choose focus areas based on preexisting personal passions, we are taking a different approach. This choice is arguably the most important a funder makes, and we are trying to be strategic about it. More
  • How many focus areas should one work in? We’re highly undecided about this question. We see most funders working in relatively few focus areas, with a high level of depth and expertise. But we also see some appeal in the idea of pursuing more breadth at the cost of depth. More
  • What does a program staffer do on a day-to-day basis? We use the term “program staff” to refer to the staff who have primary responsibility for finding and evaluating giving opportunities (though not necessarily making the final grant decision). One clear part of their role is checking in on existing grants. But it’s less obvious what the most effective activities are for finding new giving opportunities. Our sense is that the most valuable activities include articulating and refining one’s grantmaking priorities, as well as networking. It strikes us that these activities may often lead to higher returns than running an open-ended grant application process. More
  • What sort of person makes a good program staffer? We are still forming our views on this question. Some qualities we think are generally important include strong alignment and communication with the people who will ultimately be responsible for grantmaking decisions (e.g., any executives and funders working with the program staffers); a strong sense of one’s strengths and weaknesses; and strong interpersonal skills. More

Continue reading “Key Questions about Philanthropy, Part 2: Choosing Focus Areas and Hiring Program Staff”

Key Questions about Philanthropy, Part 1: What is the Role of a Funder?

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post. This post was updated on July 6 with language edits but substantially unchanged content.

As a new funder, we’ve found it surprisingly difficult to “learn the ropes” of philanthropy. We’ve found relatively little reading material – public or private – on some of the key questions we’re grappling with in starting a grantmaking organization, such as “What sorts of people should staff a foundation?” and “What makes a good grant?” To be sure, there is some written advice on philanthropy, but it leaves many of these foundational questions unaddressed. As we’ve worked on the Open Philanthropy Project, we’ve accumulated a list of questions and opinions piecemeal. This blog post is the first in a series that aims to share what we’ve gathered so far. We’ll outline some of the most important questions we’ve grappled with, and we’ll give our working answer for each one, partly to help clarify what the question means, and partly to record our thoughts, which we hope will make it easier to get feedback and track our evolution over time. We’d love to see others – particularly experienced philanthropists – write more about how they’ve thought through these questions, and other key questions we’ve neglected to raise. We hope that some day new philanthropists will be able to easily get a sense for the range of opinions among experienced funders, so that they can make informed decisions about what kind of philanthropist they want to be, rather than starting largely from scratch. This post focuses on the question: “what is the role of a funder, relative to other organizations?” In brief:

  • At first glance, it seems like a funder’s main comparative advantage is providing funding, and one might guess that a funder would do well to stick to this role as closely as possible. In other words, a funder might seek to play a “passive” role, by considering others’ ideas and choosing which ones to fund, without trying to actively influence what partner organizations work on or how they work on it.
  • In practice, this doesn’t seem to be how the vast majority of major funders operate. It’s common for funders to develop their own strategies, provide funding restricted for specific purposes, develop ideas for new organizations and pitch them to potential founders, and more. Below, we lay out a spectrum from “highly passive” funders (focused on supporting others’ ideas) to “highly active” funders (focused on executing their own strategies, with strong oversight of grantees). More
  • In the final section of this post, we lay out our rough take on when we think it’s appropriate for us, as a funder, to do more than write a check. In addition to some roles that may be familiar from for-profit investing – such as providing connections, helping with fundraising and providing basic oversight – we believe it is also worth noting the role of funders play via cause selection, and the role a funder can play in filling gaps in a field by creating organizations. More

Continue reading “Key Questions about Philanthropy, Part 1: What is the Role of a Funder?”

Funder-Initiated Startups

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

We’ve come across many cases where a funder took a leading role in creating a now-major nonprofit. This has been surprising to us: it intuitively seems that the people best suited to initiate new organizations are the people who can work full-time on conceiving an organization, fundraising for it, and doing the legwork to create it. Most successful companies seem to have been created by entrepreneurs rather than investors, and the idea that a philanthropist can “create” a successful organization (largely through concept development, recruiting and funding, without full-time operational involvement) seems strange. Yet we’ve seen many strong examples:

This is not anything approaching a comprehensive list. It’s a set of organizations we’ve come across in our work, many of which we perceive as prominent and important. I would struggle to think of many analogous cases of for-profit companies for which the original concept, recruitment, etc. came from investors rather than full-time founding employees.

Assuming this difference is real, what might explain it? While I’m not sure, I’ll list a few speculative possibilities:

  • A nonprofit startup must raise funds from a relatively thin and fragmented market. Investors ultimately all want the same thing (returns); philanthropists want very different things, and a nonprofit won’t be able to get off the ground if it can’t find a match. One symptom of “philanthropists want different things” is that nonprofit proposals are generally highly tailored to the values of funders. Thus, people with ideas may choose not to write up and shop proposals until they’ve identified a highly interested funder.
  • A nonprofit startup also doesn’t have an analogous option to bootstrapping to prove its value and raise its negotiating power. It can hope eventually to reach the point where its donor base is highly diversified, but early on nonprofits will very often live or die by major funders’ preferences.
  • Starting a new company is generally associated with high (financial) risk and high potential reward. But without a solid source of funding, starting a nonprofit means taking high financial risk without high potential reward. Furthermore, some nonprofits (like some for-profits) are best suited to be started by people relatively late in their careers; the difference is that late-career people in the for-profit sector seem more likely to have built up significant savings that they can use as a cushion. This is another reason that funder interest can be the key factor in what nonprofits get started.
  • The dynamics of competition may be different. If someone sees a for-profit with a good concept and poor execution, s/he might start a competitor. Someone who sees a nonprofit with a good concept and poor execution (and a solid funding situation) might be more likely to try to improve the nonprofit, e.g. by working for it. If true, this might make funder-initiated organizations – which, it seems, would be hard to find the right leadership match for – more viable on the nonprofit side than the for-profit side.

Our tentative view is that funders should think of “creating an organization” as a viable possibility, though as something of a last resort, since it is likely to be a much more intensive project than supporting an existing organization.

Breakthrough Fundamental Science

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

We’ve been looking for gaps in the world of scientific research funding: areas that the existing system doesn’t put enough investment into, leaving potential opportunities to do unusually large amounts of good with philanthropic funding. We previously wrote about the alleged “valley of death” that makes it challenging to translate academic insights about biology into new medical technologies. This post is about a different issue, one that has come up in the vast majority of conversations I’ve had with scientists: it is believed to be extremely difficult to do what this post will call “breakthrough fundamental science” in the existing life sciences ecosystem.

Breakthrough fundamental science is the term I’m using for what I believe many of the people I’ve spoken to have meant when they’ve used terms such as “basic research,” “high-risk/high-reward research” and “revolutionary/path-breaking research.” My subject matter knowledge is extremely limited, so I can’t be confident that I’ve correctly classified the comments I’ve heard as having a consistent theme or that I’m correctly defining the theme, but I’m attempting to do so anyway because the theme has seemed consistent and important. In brief, “breakthrough fundamental science” (in the context of life sciences) refers to research that achieves important, broadly applicable insights about biological processes, such that the insights bring on many new promising directions for research, yet it is difficult to anticipate all the specific ways in which they will be applied and thus difficult to be assured of “results” in the sense of new clinical applications. This type of work stands in contrast to research that is primarily aimed at producing a particular new drug, diagnostic or other medical technology.

This definition doesn’t lend itself to fully objective classifications, but a couple of illustrative examples would be: (a) understanding the genetic code and the structure of DNA; (b) (more recently) work on the CRISPR/CAS system and developing it to the point where it can be used to “edit” an organism’s DNA. Each of these has opened up many possible directions for research, while not having immediately clear relevance for a particular disease, condition or clinical application.

This post will:

  • Give examples of the wide variety of people who have noted the difficulty of securing support for attempts at breakthrough fundamental science in the current system.
  • Discuss what the roots of this “gap” might be.

Comments about breakthrough fundamental science

Many of the conversations we’ve had, as I’ve interpreted them, have stressed the difficulty of securing support for attempts at breakthrough fundamental science in the current system. It has been a common theme in discussions with relatively junior scientists we interviewed as potential advisors (off the record), including those who serve as our advisors now. It has also been emphasized by a number of very senior scientists with substantial credentials and authority. Some examples of the latter follow. Quotes are not verbatim; they are taken from our public conversation notes, which paraphrase the points made by the speaker.

Susan Desmond-Hellmann, current CEO of the Gates Foundation (then Chancellor of UCSF, and formerly president of product development at Genentech):

The NIH faces a large number of applicants for a relatively small number of grants. Its current methods for selecting recipients have difficulty ensuring fairness and reliable support for good scientists. In addition, these methods are likely biased toward incremental and established research over higher-risk, higher-reward research. It is particularly difficult for young researchers to secure adequate funding.

Neal Lane, currently Provost at Rice University, who has headed both the National Science Foundation and the White House’s Office of Science and Technology Policy:

The National Science Foundation (NSF), the National Institutes of Health (NIH), as well as the Department of Energy’s Office of Science, NASA and other agencies support basic research. But, increasingly, these agencies have been challenged to ensure that the research they support has potential practical benefits for the country. As a result, support for bold, sometimes called “high risk,” research has suffered. There has been a growing pressure to identify outcomes, and that discourages potentially path-breaking investigations.

Bruce Alberts, currently of UCSF, formerly Editor-in-Chief of Science and President of the National Academy of Sciences:

The current funding system for scientific research is biased toward supporting short-term, translational research (research that looks for practical applications of basic science) … I am painfully aware of the huge gaps in our understanding of fundamental life processes. Many great opportunities to advance this understanding through basic research in biology are not receiving funding from the National Institutes of Health (NIH), the largest funder of biomedical research. Changing incentives to more effectively recognize the critical importance of such understanding would have a strong effect on researchers’ choices and help produce more outstanding basic research.

Robert Tjian and Cheryl Moore, President and Chief Operating Officer of Howard Hughes Medical Institute (which I believe is the largest private science funder in the U.S.):

One of the major issues in biomedical research is that biology is not understood well enough to get to the root of problems … There’s a lot of pressure to push science in applied or clinical directions before it’s ready, which can result in money being poorly spent.

A paper in PNAS co-authored by Bruce Alberts (listed above), Harold Varmus (former Director of the National Cancer Institute and former Director of the National Institutes of Health) and others:

The system now favors those who can guarantee results rather than those with potentially path-breaking ideas that, by definition, cannot promise success. Young investigators are discouraged from departing too far from their postdoctoral work, when they should instead be posing new questions and inventing new approaches. Seasoned investigators are inclined to stick to their tried-and-true formulas for success rather than explore new fields … Many surprising discoveries, powerful research tools, and important medical benefits have arisen from efforts to decipher complex biological phenomena in model organisms. In a climate that discourages such work by emphasizing short-term goals, scientific progress will inevitably be slowed, and revolutionary findings will be deferred (3).

A few notes based on my recollections, though largely not captured in public records:

  • My recollection is that many were particularly energized about the difficulty of funding research aiming to improve tools and techniques, which I discussed in a previous post (see classification (A) in that post).
  • Nobody claimed that there is a small amount of research projects attempting breakthrough fundamental science, only that there ought to be far more due to the high importance.
  • In addition, it’s worth noting that breakthrough fundamental science is often greatly rewarded in the long run; for example, many relevant Nobel Prizes seem to be for work that broadly fits in this category. (That is to say, many of the Prizes seem to have gone to work with broad applications for understanding biological processes in general, but no obvious application to a particular disease, condition or applied medical technology.) But having a chance at a Nobel Prize decades down the line isn’t necessarily helpful for a scientist seeking to do breakthrough fundamental research; the work needs to be funded today in order to be practicable.
  • The concept of “risk” is somewhat ambiguous in some of the quotes above. It could refer to the risk that a project will fail on its own terms (e.g. failing to answer its own question or effectively test its own hypothesis). It could also refer to the uncertainty involved in the applications of particular research. My sense is that most attempts at breakthrough fundamental science are risky in both senses, but particularly the second. Regarding the first – it seems likely that attempts to make major breakthroughs will rarely be able to stick with familiar approaches and be assured of useful results. Regarding the second – when one’s goal is to achieve major insights useful for understanding biological processes in general, it may often be difficult to say in advance just what sorts of clinical applications these insights will have. This could be a problem for funders focused on the most direct, high-confidence paths to new drugs, diagnostics and other medical technologies.

What is the underlying dynamic?

As noted above, there are a good number of people voicing support for the idea of supporting more attempts at breakthrough fundamental science. However, the problem arguably derives from factors that are fairly deep and difficult to change.

The PNAS paper mentioned in the previous section lists multiple “systemic flaws” in the current system. The one it focuses most on is increasing competitiveness between scientists, brought about by an imbalance between supply and demand for academic positions:

There is now a severe imbalance between the dollars available for research and the still-growing scientific community in the United States. This imbalance has created a hypercompetitive atmosphere in which scientific productivity is reduced and promising careers are threatened … Now that the percentage of NIH grant applications that can be funded has fallen from around 30% into the low teens, biomedical scientists are spending far too much of their time writing and revising grant applications and far too little thinking about science and conducting experiments. The low success rates have induced conservative, short-term thinking in applicants, reviewers, and funders.

As this chart from the NIH shows, success rates for research project grants have fallen from ~30% to just under 20% since 1998, and the change has been driven by a growing number of applicants for a fairly constant number of annual awards. One might imagine that more applicants and more competitiveness would be a good thing, if the process consistently funded the most promising projects. However, my impression is that the NIH grant review process isn’t necessarily optimized for identifying the most promising projects and applicants, as opposed to simply eliminating the least promising ones. Thus, it may be poorly suited to such a high level of competitiveness. For example, grant applications are given scores on a 1-9 scale by all reviewers, and then ultimately funded (or not) based on the average; this arguably privileges incremental science (likely to appear clearly worthwhile to large numbers of people) over higher-risk science (which might appear extremely promising to some and not at all promising to others).

The PNAS paper lists multiple problems brought about by high competitiveness, in addition to the risk aversion discussed above:

  • It argues that competing for publication in top journals has caused scientists to “rush into print, cut corners, exaggerate their findings, and overstate the significance of their work”, contributing to issues with reproducibility that we’ve written about before.
  • It points to the increasing domination of the field by later-career scientists, and states that early-career scientists now face poor prospects and long time frames for getting substantial support for their research. I believe this sort of dynamic risks driving out the most promising scientists (who may have other career options) while retaining less promising ones; it also risks mis-allocating support, by funding scientists whose most productive years are behind rather than ahead of them.
  • It discusses the “crippling demands on a scientist’s time” brought on by the increasing difficulty of grant applications (it also cites an increasing regulatory burden as being relevant here). It argues that in addition to reducing time for scientific reflection, the increasing administrative burdens on senior scientists reduce the time they have available for peer review, which worsens the quality of the peer review process.
  • It explicitly argues that there is excessive interest in translational science, and that this is another “manifestation of [a] shift to short-term thinking,” which in turn may be another outgrowth of increased competitiveness.

In my view, all of the above represent different aspects of distortion caused by the disconnect between what science is most valuable and what science is most straightforward to evaluate. Breakthrough fundamental science is characterized by being highly innovative (making it difficult to form a consistent framework for judging it), and by having far-in-the-future and difficult-to-predict ultimate impacts. It may be possible for top scientists to evaluate it using their judgment and intuitions, but any system that seeks consistent, well-defined, practically important outcome metrics will likely struggle to do so. Instead, such a system risks rewarding those who can game it, as well as those who can show more quick and consistent (even if ultimately less important) results.

It’s worth noting that the criticism of “rewarding the measurable, rather than the important” has often been leveled at GiveWell’s work on top charities. I have long felt that focusing on the measurable is quite appropriate when (a) serving individual donors seeking somewhere they can give without having to invest a lot of time in learning; (b) working on issues related to global health and development, where higher-risk/higher-reward approaches have a history of coming up empty. However, the world of scientific research is very different. In this environment, it seems to me that insisting on accountability to meaningful short-term metrics could easily do more harm than good.

Should we focus on funding breakthrough fundamental science?

The idea that breakthrough fundamental science is under-supported makes a good deal of sense to me, and I perceive a great deal of consensus on this point among scientists. However, evaluating – and implementing – the goal of “funding breakthrough fundamental science” is fraught with challenges. Defining just what constitutes potential “breakthrough fundamental science” seems to be extremely difficult and to require a good deal of scientific expertise and judgment. It would be a major challenge to estimate how much is being spent, vs. how much “should be” spent, on potential breakthrough fundamental science – far more so than with neglected goals, and more so even than with translational research.

In addition, it certainly isn’t the case that this type of work is highly neglected. After all, it appears that breakthrough fundamental science is well-represented among Nobel Prize winners, and as the quotes above show, it is a major concern of some very large funders. It’s highly possible that there are still far too few attempts at breakthrough fundamental science, but it’s far from clear how to determine this.

At this time, our biggest focus is on trying to improve our general capacity to investigate scientific research, which we’re working on as described previously. We’re also trying to get more context on the history of major breakthroughs in biomedical sciences, and the role of different kinds of science in these breakthroughs. We will hopefully be better equipped for more investigation of breakthrough fundamental science after we’ve made more progress on those fronts.

One more note: while the “breakthrough fundamental science” idea is often presented as a contrast to focusing on “translational research”, the two are not mutually exclusive. It could easily be the case that the existing system under-supports both, while focusing most of its resources on a particular kind of research that fits in neither category. My current picture is that, when looking at the stages of research I laid out earlier, the existing system is quite focused on stage (C) – identifying the cause of particular diseases and conditions of interest – while potentially underinvesting in multiple other stages (some of which might be classified as “breakthrough fundamental science” and some of which might be classified as “translational research”).

The Path to Biomedical Progress

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

We’ve continued to look into scientific research funding for the purposes of the Open Philanthropy Project. This hasn’t been a high priority for the last year, and our investigation remains preliminary, but I plan to write several posts about what we’ve found so far. Our early focus has been on biomedical research specifically.

Most useful new technologies are the product of many different lines of research, which progress in different ways and on different time frames. I think that when most people think about scientific research, they tend to instinctively picture only a subset of it. For example, people hoping for better cancer treatment tend instinctively to think about “studying cancer” as opposed to “studying general behavior of cells” or “studying microscopy techniques,” even though all three can be essential for making progress on cancer treatment. Picturing only a particular kind of research can affect the way people choose what science to support.

I’m planning to write a fair amount about what I see as promising approaches to biomedical sciences philanthropy. Much of what I’m interested in will be hard to explain without some basic background and vocabulary around different types of research, and I’ve been unable to find an existing guide that provides this background. (Indeed, many of what I consider “overlooked opportunities to do good” may be overlooked because of donors’ tendencies to focus on the easiest-to-understand types of science.)

This post will:

  • Lay out a basic guide to the roles of different types of biomedical research: improving tools and techniques, studying healthy biological processes, studying diseases and conditions of interest, generating possible treatments, preliminarily evaluating possible treatments, and clinical trials.
  • Use the example of the cancer drug Herceptin to compare the roles of these different sorts of research more concretely.
  • Go through what I see as some common misconceptions that stem from overfocusing on a particular kind of research, rather than on the complementary roles of many kinds of research.

Basic guide to the roles of different types of biomedical research

Below are some distinctions I’ve found it helpful to draw between different kinds of research. This picture is highly simplified: many types of research don’t fit neatly into one category, and the relationships between the different categories can be complex: any type of research can influence any other kind. In the diagram to the right (click to expand), I’ve highlighted the directions of influence I believe are generally most salient.

(A) Improving tools and techniques. Biomedical researchers rely on a variety of tools and techniques that were largely developed for the general purpose of measuring and understanding biological processes, rather than with any particular treatment or disease/condition in mind. Well-known examples include microscopes and DNA sequencing, both of which have been essential for developing more specific knowledge about particular diseases and conditions. More recent examples include CRISPR-related gene editing techniques, RNA interference, and using embryonic stem cells to genetically modify mice. All three of these provide ways of experimenting with changes in the genetic code and seeing what results. The former two may have direct applications for treatment approaches in addition to their value in research; the latter two were both relatively recently honored with Nobel Prizes. Improvements in tools and techniques can be a key factor in improving most kinds of research on this list. Sometimes improvements in tools and techniques (e.g., faster/cheaper DNA sequencing; more precise microscopes) can be as important as the development of new ones.

(B) Studying healthy biological processes. Basic knowledge about how cells function, how the immune system works, the nature of DNA, etc. has been essential to much progress in biomedical research. Many of the recent Nobel Prizes in Physiology or Medicine were for work in this category, some of which led directly to the development of new tools and techniques (as in the case of CRISPR-based gene editing, which is drawn from insights about bacterial immune systems).

(C) Studying diseases and conditions of interest. Much research focuses on understanding exactly what causes a particular disease and condition, as specifically and mechanistically as possible. Determining that a disease is caused by bacteria, a virus, or by a particular overactive gene or protein can have major implications for how to treat it; for example, the cancer drug Gleevec was developed by looking for a drug that would bind to a particular protein, which researchers had identified as key to a particular cancer. Note that (C) and (B) can often be tightly intertwined, as studying differences between healthy and diseased organisms can tell us a great deal both about the disease of interest and about the general ways in which healthy organisms function. However, (B) may have more trouble attracting support from non-scientists, since the applications can be less predictable and clear.

(D) Generating possible treatments. No matter how much we know about the causes of a particular disease/condition, this doesn’t guarantee that we’ll be able to find an effective treatment. Sometimes (as with Herceptin – more below) treatments will suggest themselves based on prior knowledge; other times the process comes down largely to trial and error. For example, malaria researchers know a fair amount about the parasite that causes malaria, but have only identified a limited number of chemicals that can kill it; because of the ongoing threat of drug resistance developing, they continue to go through many thousands of chemicals per year in a trial-and-error process, checking whether each shows potential for killing the relevant parasite. (Source)

(E) Preliminarily evaluating possible treatments (sometimes called “preclinical” work). Possible treatments are often first tested “in vitro” – in a simplified environment, where researchers can isolate how they work. (For example, seeing whether a chemical can kill isolated parasites in a dish.) But ultimately, a treatment’s value depends on how it interacts with the complex biology of the human body, and whether its benefits outweigh its side effects. Since clinical trials (next paragraph) are extremely expensive and time-consuming, it can be valuable to first test and refine possible treatments in other ways. This can include animal testing, as well as other methods for predicting a treatment’s performance.

(F) Clinical trials. Before a treatment comes to market, it usually goes through clinical trials: studies (often highly rigorous experiments) in which the treatment is given to humans and the results are assessed. Clinical trials typically involve four different phases: early phases focused on safety and preliminary information, and later phases with larger trials focused on definitively understanding the drug’s effects. Many people instinctively picture clinical trials when they think about biomedical research, and clinical trials account for a great deal of research spending (one estimate, which I haven’t vetted, is that clinical trials cost tens of billions of dollars a year, over half of industry R&D spending). However, the number of clinical trials going on generally is – or should be – a function of the promising leads that are generated by other types of research, and the most important leverage points for improving treatment are often within these other types of research.

(A) – (C) are generally associated with academia, while (D) – (F) are generally associated with industry. There are a variety of more detailed guides to (D) – (F), often referred to as the “drug discovery process” (example).

Example: Herceptin

Herceptin is a drug used for certain breast cancers, first approved in 1998. Its development relied on relatively recent insights and techniques, and it is notable for its relative lack of toxicity and side effects compared to other cancer drugs. I perceive it as one of the major recent success stories of biomedical research (in terms of improving treatment, as opposed to gaining knowledge) – it was one of the best-selling drugs of 2013 – and it’s an unusually easy drug to trace the development of because there is a book about it, Her-2: The Making of Herceptin (which I recommend).

Here I list, in chronological order, some of the developments which seem to have been crucial for developing Herceptin. My knowledge of this topic is quite limited, and I don’t mean this as an exhaustive list. I also wish to emphasize that many of the items on this list were the result of general inquiries into biology and cancer – they weren’t necessarily aimed at developing something like Herceptin, but they ended up being crucial to it. Throughout this summary, I note which of the above types of research were particularly relevant, using the same letters in parentheses that I used above.

  • In the 1950s, there was a great deal of research focused on understanding the genetic code (B). For purposes of this post, it’s sufficient to know that a gene serves the function of a set of instructions for building a protein, a kind of molecule that can come in many different forms serving a variety of biological functions. The research that led to understanding the genetic code was itself helped along by multiple new tools and techniques (A) such as Linus Pauling’s techniques for modeling possible three-dimensional structures (more).
  • In the 1970s, studies on chicken viruses that were associated with cancer led to establishing the idea of an oncogene: a particular gene (often resulting from a mutation) that, when it occurs, causes cancer. (C)
  • In 1983, several scientists established a link between oncogenes and a particular sort of protein called epidermal growth factor receptors (EGFRs), which give cells instructions to grow and proliferate. In particular, they determined that a particular EGFR was identical to the protein associated with a known chicken oncogene. This work was a mix of (B) and (C), as it grew partly out of a general interest in the role played by EGFRs. It also required being able to establish which gene coded for a particular protein, using techniques that were likely established in the 1970s or later (A).
  • In 1986, an academic scientist collaborated with Genentech to analyze the genes present in a series of cancerous tumors, and cross-reference them with a list of possible cancer-associated EGFRs (C). One match involved a particular gene called HER2/neu; tumors with this gene (in a mutated form) showed excessive production of the associated protein, which suggested that (a) the mutated HER2/neu gene was overproducing HER2/neu proteins, causing excessive cell proliferation and thus cancer; (b) this particular sort of cancer might be mitigated if one could destroy or disable HER2/neu proteins. This work likely benefited from advances in being able to “read” a genetic code more cheaply and quickly.
  • The next step was to find a drug that could destroy or disable the HER2/neu proteins (D). This was done using a relatively recent technique (A), developed in the 1970s, that relied on a strong understanding of the immune system (B) and of another sort of cancer that altered the immune system in a particular way (C). Specifically, researchers were able to mass-produce antibodies designed to recognize and attach to the EGFR in question, thus signaling the immune system to destroy them.
  • At that time, monoclonal antibodies (mass-produced antibodies as described above) were seen as highly risky drug candidates, since they were produced from other animals and likely to be rejected by human immune systems. However, in the midst of the research described above, a new technique (A) was created for getting the body to accept these antibodies, greatly improving the prospects for getting a drug.
  • Researchers then took advantage of a relatively recent technique (A) for inserting human tumors into modified mice, which allowed them to test the drug and produce compelling preliminary evidence (E) that the drug might be highly effective.
  • At this point – 1988 – there was a potential drug and some supportive evidence behind it, but its ultimate effect on cancer in humans was unknown. It would be another ten years before the drug went through all relevant clinical trials (F) and received FDA approval, under the name Herceptin. Her-2: The Making of Herceptin gives a great deal of detail on the challenges of this period.

As detailed above, many essential insights necessary for Herceptin’s development came out very long before the idea of Herceptin had been established. My impression is that most major biomedical breakthroughs of the last few decades have a similar degree of reliance on a large number of previous insights, many of them fundamentally concerning tools and techniques (A) or the functioning of healthy organisms (B) rather than just disease-specific discoveries.

General misperceptions that can arise from over-focusing on certain types of research

I believe that science supporters often have misperceptions about the promising paths to progress, stemming from picturing only certain types of research. Below, I informally list some of these misperceptions, as informal non-attributed quotes.

  • “Publicly funded research is unnecessary; the best research is done in the for-profit sector.” My impression is that most industry research falls into categories (D)-(F). (A)-(C), by contrast, tend to be a poor fit for industry research, because they are so far removed from treatments both in terms of time and risk. Because it is so hard to say what the eventual use is of a new tool/technique or insight into healthy organisms, it is likely more efficient for researchers to put insights into the public domain rather than trying to monetize them directly.
  • “Drug companies don’t do valuable research – they just monetize what academia provides them for free.” This is the flipside of the above misconception, and I think it overfocuses on (A)-(C) without recognizing the challenges and costs of (D)-(F). Given the very high expenses of research in categories (D)-(F), and the current norms and funding mechanisms of academia, (D)-(F) are not a good fit for academia.
  • “The best giving opportunities will be for diseases that aren’t profitable for drug companies to work on.” This might be true for research in categories (D)-(F), but one should also consider research in categories (A)-(C); this research is generally based on a different set of incentives from those of drug companies, and so I’d expect the best giving opportunities to follow a different pattern.
  • “Much more is spent on disease X than disease Y; therefore disease Y is underfunded.” I think this kind of statement often overweights the importance of (F), the most expensive but not necessarily most crucial category of research. If more is spent on disease X than on disease Y, this may be simply because there are more promising clinical trial candidates for disease X than disease Y. Generally, I am wary of “total spending” figures that include clinical trials; I don’t think such figures necessarily tell us much about society’s priorities.
  • “Academia is too focused on knowledge for its own sake; we need to get it to think more about practical solutions and treatments.” I believe this attitude undervalues (A)-(B) and understates how important general insights and tools can be.
  • “We should focus on funding research with a clear hypothesis, preliminary support for the hypothesis, and a clear plan for further testing the hypothesis.” I’ve heard multiple complaints that much of the NIH takes this attitude in allocating funding. Research in category (A) is often not hypothesis-driven at all, yet can be very useful. More on this in a future post.
  • “The key obstacles to biomedical progress are related to reproducibility and reliability of studies.” I think that reproducibility is important, and potentially relevant to most types of research, but it is most core to clinical trials (F). Studies on humans are generally expensive and long-running, and so they may affect policy and practice for decades without ever being replicated. By contrast, for many other kinds of research, there is some cheap effective “replication” – or re-testing of the basic claims – via researchers trying to build on insights in their own lab work, so a non-reproducible study might in many cases mean a relatively modest waste of resources. I’ve heard varying opinions on how much waste is created by reproducibility-related issues in early-stage research, and think it is possible that this issue is a major one, but it is far from clear that it is the key issue.