## New Shallow Investigations: Telecommunications and Civil Conflict Reduction

We recently published two shallow investigations on potential focus areas to the Effective Altruism Forum. Shallow investigations, which are part of our cause selection process, are mainly intended as quick writeups for internal audiences and aren’t optimized for public consumption. However, we’re sharing these two publicly in case others find them useful.

The default outcome for shallow investigations is that we do not move forward to a deeper investigation or grantmaking, though we investigate further when results are particularly promising.

If you have thoughts or questions on either of these investigations, please use this feedback form or leave a comment on the EA Forum.

## Telecommunications in Low and Middle-Income Countries (LMICs)

By Research Fellow Lauren Gilbert (EA Forum link)

• Lauren finds that expanding cellular phone and internet access appears to cost-effectively increase incomes. Randomized trials and quasi-experimental studies in LMICs showed that gaining internet access led to substantial increases in income, with high social returns on investment.
• We find these reported effects surprisingly large, and are continuing to dig into them more.
• Lauren estimates that 3-9% of the world’s population do not have access to cellular service, and ~40% of the world’s population either have no access to mobile internet or do not use it. Lauren finds that the biggest barrier to usage is the cost of devices and coverage. These coverage gaps and costs are shrinking over time.
• A large majority of spending on telecommunications is private/commercial, with a smaller amount of philanthropic spending. While the private investments are large, they aren’t as focused as a philanthropist might be on improving access for poor and rural communities.
• Philanthropists could potentially help improve access by subsidizing investments in cell phone towers to improve coverage, and in internet cables to reduce the cost of internet. Lauren’s rough back-of-the-envelope calculation suggests that these investments may be cost-effective. A funder could also potentially lobby for policy changes to reduce costs — for example, reducing tariffs on imported electronics or changing the rules around how spectrum can be licensed.

## Civil Conflict Reduction

Also by Lauren Gilbert (EA Forum link)

• Civil conflict is a very important problem. Lauren estimates that civil wars directly and indirectly cause the loss of around 1/2 as many disability-adjusted life years as malaria and neglected tropical diseases combined. Civil wars also substantially impede economic growth, mostly in countries that are already very poor.
• While civil conflict is important and arguably neglected, it isn’t clear how tractable it is. However, some interventions have shown promise.
• Lauren finds some evidence that UN peacekeeping missions are effective, and argues philanthropists could lobby for more funding.
• Some micro-level interventions, such as mediation or cognitive behavioral therapy, also have suggestive empirical evidence behind them. Philanthropists could fund more research into these interventions.

## In a nutshell

• What is the problem? Hundreds of millions of people are infected with sexually transmitted diseases (STDs), and more than a hundred thousand die due to STDs other than HIV each year. The World Health Organization (WHO) and Global Burden of Disease (GBD) Study estimate that STDs other than HIV and HPV are responsible for approximately 9-13 million disability-adjusted life years (DALYs) per year. Based on their impact and the availability of treatments and preventative measures, we chose three diseases to investigate in more detail: herpes simplex virus (HSV), syphilis, and human papillomavirus (HPV). The GBD estimated the burden of HSV at approximately 300k DALYs per year, but this estimate does not include the burden of some additional conditions caused by HSV. We briefly looked at the burden of these conditions and concluded that the true burden may be somewhat or substantially larger. HSV cannot be cured and there is no vaccine, although antivirals can be used to treat it. Syphilis is responsible for most of the direct burden of STDs (7-11 million DALYs per year, according to the WHO and GBD), and there is no vaccine, but it can be cured with antibiotics. HPV is a necessary cause of cervical cancer, which is estimated to cause approximately 7-9 million DALYs per year, and can contribute to the development of other cancers as well (the burden of HPV is not included in WHO and GBD assessments of the burden of STDs, but cervical cancer burden is included separately in these sources). There are vaccines that appear to confer immunity against some strains of HPV, and precancerous lesions can often be removed once detected, but HPV cannot be cured.
• Who else is working on it? It’s our impression that there is relatively little nonprofit involvement in and private funding for STD research and development. The NIH contributed approximately $250 million in funding to support STD research in 2015 (funding for HIV and HPV research and development categorized separately from the$250 million figure). Grantome searches suggest NIH and NSF funding of approximately $100 million,$150 million, and $20 million in funding for HSV, HPV, and syphilis research in 2013, respectively. ## 1. What is the problem? Sexually transmitted diseases (STDs) are widespread. Hundreds of millions of people are infected with at least one STD,1 and the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) estimated that STDs other than HIV and HPV caused approximately 142,000 deaths in 2013, while HIV caused approximately 1,341,000 deaths in 2013.2 We may conduct a separate investigation into HIV/AIDS at a later date, but did not include it in this investigation. We believe it is widely accepted that mother-to-child STD transmission can result in stillbirths, infant mortality, and chronic health conditions. Stigma, fear, and avoidance of sexual intimacy associated with STDs may lower quality of life. Common STDs, including genital herpes, HPV, and syphilis, are associated with an increased risk of HIV transmission, but it appears that high-quality evidence of a causal relationship is scarce. (See below for our take on this.) Vaccines are available to prevent HPV (although note that currently-available HPV vaccines do not protect against all strains of HPV3) and hepatitis B.4 Other STDs, including syphilis, chlamydia, and trichomoniasis, are treatable and generally curable with antibiotics,5 although some STDs, especially gonorrhea, are developing resistance to antibiotics.6 We formulated an initial list of STDs from several other lists of STDs that we found.7 We declined to include infections such as Pelvic Inflammatory Disease (PID) that result from infection with other STDs. Of these, we initially decided to assess the impact of and funding for diseases that are predominantly sexually-transmitted other than HIV, and that are currently difficult or impossible to treat or cure because we thought it was more likely that additional scientific research would be beneficial for cases where reliable cures or treatments have not been found. We later decided to investigate predominantly sexually-transmitted diseases that had a reported impact of greater than five million disability-adjusted life years (DALYs) per year as well, regardless of whether treatments or cures for those are available.8 Based on these processes, we decided to examine herpes simplex virus (HSV), syphilis, and human papillomavirus (HPV) in more detail. #### 1.1 Size of the problem #### 1.1.1 Overall burden To get a sense of the impact of different STDs on public health, we looked at the most recent estimates of the global annual burden of major STDs (excluding HIV and HPV) in DALYs from the Global Burden of Disease Study (GBD) and the World Health Organization’s (WHO) Global Health Observatory (GHO). The results are below: DISEASE GLOBAL BURDEN OF DISEASE (2013) WORLD HEALTH ORGANIZATION (2012) Syphilis 11,324,500 7,038,630 Chlamydial infections/Chlamydia 692,400 1,429,973 Gonococcal infections/Gonorrhea 313,900 545,145 Genital herpes 311,600 NA Trichomoniasis 113,900 172,850 Other STDs 101,000 734,0759 Total burden of STDs (excluding HIV) in DALYs 12,857,200 9,920,672 However, it’s our impression that these numbers are very uncertain (indeed, they are not in close agreement with one another). The estimates did not include the impacts of neonatal forms of some STDs, and may have missed other impacts as well.10 Additionally, we note that there are major negative impacts of STDs that we do not quantitatively assess in this report, such as the psychological and sociological effects of the presence of STDs in a community (e.g. fear of transmission and stigmatization). In addition, there may be impacts we do not know of and that our brief investigation did not uncover. #### 1.1.2 STDs and HIV Transmission There is evidence of a correlation between HIV and infection with other STDs, including HSV, HPV, and gonorrhea.11 Some researchers also believe there are plausible mechanisms by which the presence of other STDs could increase the likelihood of HIV transmission, namely by 1) increasing the incidence of lesioned regions of flesh around the genitals, and 2) recruiting cells carrying CD4 receptors (which HIV uses to gain entry into host T-cells) to the area.12 However, we believe the ability to definitively determine a causal relationship is complicated by confounding variables (for example, behavioral and health-status factors that contribute to the risk of contracting one STD may increase the risk of contracting another), and limitations on the types of experiments that can be performed without harming participants. We are not confident that attempts to fully control for confounding variables are able to do so. We know of two small RCTs which concluded that HSV antivirals reduced HIV levels, either in the seminal fluids or in plasma and around the cervix.13 However, according to @Mayer and Venkatesh [email protected], in a meta-analysis of randomized controlled trials (RCTs) on interventions to distribute antivirals for HSV and antibiotics for bacterial STDs, six out of seven RCTs they examined did not find a statistically-significant reduction in HIV transmission, nor have studies in subsequent reviews (note that there is overlap between the studies included in the different reviews).14 Given that the majority of studies we encountered did not find an effect of STD treatment and prevention on HIV transmission, we chose not to include a quantitative estimate of the potential impact of STDs on HIV. However, we note that if STDs are in fact responsible for a substantial proportion of HIV transmission, that might make their overall impact substantially larger than our estimates below.15 Some studies have detected an association between HIV and HPV.16 Additionally, mechanisms by which HPV infection could increase HIV transmission have been proposed.17 However, it is our impression that as of 2015, no RCTs have been conducted on the impact of HPV vaccination on HIV transmission.18 We do not know if vaccines or improved treatments for STDs would reduce HIV transmission, or whether increased access to currently-available STD treatments would help prevent HIV. #### 1.1.3 Genital herpes The herpes simplex virus (HSV) occurs in two forms, HSV-1 and HSV-2.19 Hundreds of millions of people are infected with one or both types of HSV, although many show no symptoms.20 Both types can be transmitted by sex.21 There is no commercially-available HSV vaccine and HSV infections cannot be cured, but antiviral agents can reduce outbreaks.22 Major sources of disease burden from HSV include: Genital ulcers: The Global Burden of Disease reports that genital HSV resulted in 311,600 years lived with disability (YLD) in 2013 due to genital ulcer disease.23 We have not vetted this estimate, but did attempt to independently derive it by assigning what we felt was a plausible disability weight to genital ulcers and multiplying by its prevalence, and found that it was within the range of what we expected. Neonatal deaths from HSV: We found it difficult to come to an understanding the impact of neonatal HSV. This is because: • We have major uncertainty about current global neonatal herpes incidence, although the data we found suggest that the overall incidence was likely more than 1/100,000 and less than 1/100 in the late 1990s and early 2000s.24 • We are highly uncertain about the neonatal herpes mortality rate, and the sources we’ve found have not given us a strong sense of the likely true number because the estimates vary somewhat. We haven’t investigated their methodology, and we are unsure of how the mortality rate varies by country.25 We don’t know if the difference in estimates of the mortality rates stem primarily from differences in incidence at the different study sites and times, differences in proportion of cases treated, differences in the quality of the treatment, other factors, or some combination of these. We think it require substantial additional work to understand neonatal HSV mortality rates somewhat better, but that even if we put in more time, we might not gain clarity on this. • We did not look into the number of infants that sustain lifelong sources of disability from neonatal HSV. Given the number of births/year,26 it seems likely to us that thousands or tens of thousands of infants are infected with HSV every year, and it seems possible to us that more than 10% of those infants die. If that were true, the impact of neonatal HSV might represent a substantial fraction or the majority of the total burden of HSV.27 Severe vision impairment from HSV: One source suggests that 40,000 people per year may become profoundly visually impaired in one or both eyes due to HSV keratitis (an inflammation of the cornea).28 We did not vet this estimate, and are not confident that these numbers reflect the true burden of severe vision impairment from HSV. We don’t know how many DALYs HSV keratitis is likely responsible for. It’s our impression that most cases of HSV keratitis occur late in life but may be more severe in children. 29 In many cases, HSV vision impairment may be the result of HSV-1 that was not directly sexually transmitted. Many researchers believe that HSV-1 can be and often is transmitted by non-sexual kissing (for example by family members) or by sharing items that touch the mouth such as eating utensils and toothbrushes. Other impacts: We did not investigate several other impacts of HSV, including: • Sequelae in non-fatal cases of neonatal HSV • Cases of mildly-moderately impaired vision from HSV • Oral HSV ulcers • A proposed connection between HSV-1 infection and the development of Alzheimer’s Disease30 • Encephalitis and meningitis from herpes31 • Herpes whitlow and gladiatorum32 Overall burden: The GBD estimated that genital herpes resulted in approximately 300k DALYs in 2013. Based on our research into conditions caused by HSV but not included in the GBD estimate (namely, the unquantified but possibly-substantial impact of HSV keratitis and neonatal HSV, the possible and (if real) plausibly substantial impact of HSV on HIV/AIDs transmission and Alzheimer’s Disease, and the likely small impact of the other conditions above) we would guess that the true impact of HSV is substantially larger, although we don’t know how much larger. #### 1.1.4 Human papillomavirus HPV is a common infection; the WHO estimates that approximately 12% of women with normal cytological findings (which we believe to mean no cellular signs indicating cervical cancer or precancerous changes in the cervix, indicating that this number is likely an underestimate of the proportion of women with HPV) and 21% of men worldwide are infected at a given time.33 There are many strains of HPV, most of which are asymptomatic but some of which increase cancer risk around the infected area (especially the cervix) or cause genital warts.34 There is no treatment we know of for HPV, but there are vaccines that confer protection against some strains of the virus.35 All the vaccines protect against strains 16 and 18,36 which, according to the National Cancer Institute, are together responsible for 70% of cervical cancers.37 There are also procedures that allow healthcare workers to identify and, if necessary, remove precancerous lesions on the cervix.38 According to the WHO, HPV infections of healthy individuals often spontaneously resolve themselves within two years39 and it usually takes approximately ten years for HPV infection to progress to an invasive cancer.40 Major sources of disease burden from HPV include: Cervical cancer: In 2012 it was reported that there were 528,000 cases of cervical cancer and 266,000 deaths from cervical cancer,41 all the result of some strain of HPV.42 The DALY burden of cervical cancer was estimated at 6.9M in 2013 by the GBD,43 and 9.2M by the WHO in 2012.44 Cancers of the vulva, vagina, penis, anus, mouth, and oropharynx attributable to HPV: There is evidence linking HPV to cancers of the vulva, vagina, penis, anus, mouth, and oropharynx.45 One source indicated that in 2006, 87.8% of cancers from HPV were cervical cancers.46 so we believe that the estimate of the burden of cervical cancer captures the majority of the known direct harm from HPV. Other impacts: There are other impacts of HPV which are not included in our calculation of the burden of HPV. These include: • Respiratory papillomatosis47 • A possible association between lung cancer and HPV. It’s our impression that a causal link has not been established, and we are uncertain about the quality of the evidence for the connection.48 • Genital warts • Common skin warts and rare skin conditions in immunocompromised individuals.49 However, it is our understanding that these result from strains of HPV that are not predominantly sexually transmitted. Some research suggests that HPV vaccination may increase the risk of becoming infected with the strains that the vaccine does not protect against.50 We have not fully investigated this claim, and do not know how much it detracts from the public health benefit of HPV vaccines, if at all. Overall burden: In summary- • The GBD estimates the burden of cervical cancer at 6.9M DALYs • The WHO estimates the burden of cervical cancer at 9.2M DALYs • HPV can cause other cancers as well, but these cancers probably contribute a small proportion of the total burden. • There is evidence of an associated between HPV infection and HIV transmission, but we are unsure about the strength and cause of this association. • We have not included other health consequences of HPV, which we believe to be small in comparison. #### 1.1.5 Syphilis Syphilis is a bacterial infection caused by the bacterium Treponema pallidum.51 When left untreated, it can lead to sores, rashes, eye problems, neurological and heart problems, and death.52 It can be treated and the infection cured with antibiotics.53 Overall burden: In 2013 the GBD estimated that there were 136,848 deaths due to syphilis, of which 120,537 were in children five years of age or under,54 while the WHO GHO estimated that there were 78,910 deaths from syphilis in 2012, of which 67,489 were in children five years of age or under.55 The DALY burden from syphilis (including neonatal syphilis) was estimated by the GBD at 11.3M DALYs and the WHO GHO at 7.0M DALYs.56 Potential effects of syphilis on HIV transmission were not included in these assessments,57 but we do not know of other direct impacts of syphilis that may have been excluded (despite a brief search). ## 2. Who else is working on this? We do not have a comprehensive understanding of which organizations fund research on STDs, as opposed to raising awareness, offering services, or advocating on behalf of individuals that suffer from STDs.58 However, it’s our understanding that philanthropic and nonprofit involvement in STD research is minimal. We are not aware of non-profit organizations specifically supporting HSV or syphilis research. #### 2.1 How much funding is in this area? Total The total NIH funding for sexually transmitted diseases/herpes (not including funding for HIV/AIDS, any hepatitis virus, or HPV/cervical cancer vaccines) was approximately$250M in 2015.59 It was our impression of NIH STD funding that there is substantial funding for HPV vaccine trials, testing the effects of the HPV vaccine, and cervical cancer diagnostics. Even though HIV/AIDS had its own section, there seemed to be some funding allocated for HIV/AIDS research listed in the “sexually transmitted diseases/herpes” section, some funding for social science programs on sexual health and safety, and research on HSV and HSV vaccine candidates. We saw several funded projects on potential chlamydia and gonorrhea vaccine candidates.60 In addition, we identified approximately $1.4M in funding for STDs from private foundations in 2012.61 We searched Grantome.com, a database of scientific research grants, for information about grants that included the words “sexually transmitted,” and found that in 2013 there was$188M in funding reported that met these search criteria.62

We looked into the assets and funding from other sources, including smaller sources and those predominantly dedicated to STD advocacy, awareness-building, and service provision rather than research and development, but these numbers did not significantly affect our view of the crowdedness of this space.63

The funding for research and development for each of our STDs of interest is below.

HSV

• Grantome, 2013, approximate: $104.3M64 • NIH Project Reporter, 2015, approximate:$5.8-137.9M65

HPV

• Grantome, 2013, approximate: $147.6M66 • NIH, Estimates for Funding of Varion Research, Condition, and Disease Categories, 2015:$31M67
• NIH Project Reporter, 2015, approximate: $35.2-204.3M68 Syphilis • Grantome, 2013, approximate:$17.3M69
• NIH Project Reporter, 2015, approximate: : $2.2-22.9M70 ## 3. Our process We initially decided to investigate this area because we thought STD research might be impactful and neglected due to associated stigma. We focused on quickly determining this, without investigating potential interventions in this space. The specific steps we took to investigate importance and neglectedness are as follows. For investigating importance, we: • Looked at global DALY burden from each STD in the GBD 2013 and WHO GHO 2012 data. • Attempted to independently derive DALY estimates, where possible, from lives lost. • Briefly researched the diseases we focused on, their sequelae, and treatments. We did this by reading fact sheets and Wikipedia articles about the diseases in question, scanning for highly-cited recent articles in Google Scholar about them, and investigating further points that seemed important. • Looked for relevant Cochrane Library articles on HPV, HSV, syphilis, and STDs in general. For investigating neglectedness, we: ## 4. Questions for further investigation Our investigation so far has focused almost exclusively on the burden of these diseases and how much attention they receive from funders of scientific research. If we were to do further research here, our primary focus would be on the tractability of potential research directions. Some questions we might aim to address include: • What are the greatest barriers to STD diagnosis? Would improved diagnostics for STD infections lead to higher rates of treatment and cure? • What are the greatest barriers to the development of new vaccines for STDs, including syphilis? • Would it be practical to deliver a syphilis vaccine to the required population? • How do HPV vaccines affect the prevalence and transmission of HPV strains against which they do not confer immunity? How is this likely to impact the future burden of HPV? • To what extent do STDs increase HIV transmission? To what extent do STD treatments and vaccines reduce HIV transmission, if at all? • What other research and development projects could potentially decrease the burden of STDs? ## 5. Sources DOCUMENT SOURCE 990 Finder American Sexually Transmitted Diseases Association Form 990 2014 Source 990 Finder National Coalition of STD Directors Form 990 2013 Source 990 Finder The Foundation for Research into Sexually Transmitted Diseases Form 990 2013 Source 990 Finder: HPV and Anal Cancer Foundation Form 990 2013 Source Baeten et al. 2008 Source CDC Office of Financial Resources 2015 Annual Report Source CDC: HPV Vaccine Information for Clinicians Source CIA World Factbook Source Clinuvel: Herpes Simplex Virus Source Cochrane Library Source Corey and Wald 2009 Source Farazmand Woolley and Kinghorn 2011 Source Farooq and Shukla 2012 Source FDA News Release: Gardasil 9 Source Foundation giving based on Foundation Center data Source GBD 2013 DALYs from all causes Source GBD 2013 deaths from syphilis Source GBD 2013 International Classification of Diseases codes mapped to the Global Burden of Disease cause list Source Genital Herpes – CDC Fact Sheet (Detailed) Source GiveWell: DALY Source Global Burden of Disease 2013: Mortality and Causes of Death Source Global Burden of Disease Study 2013 (GBD 2013) Data Downloads – Full Results Source Globocan Cervical Cancer Fact Sheet Source Grantome.com Source Grantome.com “herpes simplex” Source Grantome.com “hpv” Source Grantome.com “sexually transmitted” Source Grantome.com “syphilis” Source Guo et al. 2015 Source Head et al. 2015 Source Houlihan et al. 2012 Source Human papillomavirus vaccines: WHO position paper, October 2014 Source ICD 10 Data: Anogenital herpes Source Johnston, Gottlieb, and Wald 2016 Source Looker et al. 2015 Source Looker Garnett and Schmid 2008 Source Mayer and Venkatesh 2011 Source Morris et al. 2008 Source Mutua, M’Imunya, and Wiysonge 2012 Source National Cancer Institute: HPV Vaccine Fact Sheet Source Ng et al. 2011 Source NIH 2015 STD/Herpes project listing Source NIH Estimates of Funding for Various Research, Condition, and Disease Categories (RCDC) Source NIH Project Reporter Source NIH Reporter- Herpes simplex funding, project titles only Source NIH Reporter- Herpes simplex funding, project titles, terms, and abstracts Source NIH Reporter- HPV funding, project titles only Source NIH Reporter- HPV funding, project titles, terms, and abstracts Source NIH Reporter- Syphilis funding, project titles only Source NIH Reporter- Syphilis funding, project titles, terms, and abstracts Source Parkin 2006 Source Patel et al. 2013 Source Prabhu Jayalekshmi and Pillai 2012 Source Rositch et al. 2014 Source Shrestha and Englund 2010 Source STD.gov List of All STDs and Their Symptoms Source Syphilis – CDC Fact Sheet (Detailed) Source The HPV and Anal Cancer Foundation: Role and Impact Source The STD Project’s List of All STDs Source Wald and Corey 2007 Source Wald and Link 2002 Source Wang and Ritterband: Herpes Simplex Keratitis Epidemiology Source Ward and Rönn 2010 Source WHO GHO DALYs by Cause Source WHO GHO Data Source WHO GHO Deaths by Cause Source WHO: Global Burden of Disease Source WHO STIs Fact Sheet Source Wikipedia: Causes of sexually transmitted infections Source Wikipedia: Herpes simplex Source Wikipedia: Herpes simplex virus Source Wikipedia: Human papillomavirus Source Zuckerman et al. 2009 Source ## Cannabis Policy This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### What is the problem? Our impression is that despite the spread of efforts to legalize recreational cannabis use, there is currently relatively little discussion of which legislation models are best for public health. We believe ineffective regulation may carry substantial public health costs. #### What are possible interventions? A philanthropist could fund research into different legalization models and their effects, or could support increased discussion between advocates of cannabis legalization and public health experts with experience in policy related to intoxicants. #### Who else is working on it? A number of foundations do some work on cannabis policy, often within the broader area of drug policy. We are not aware of any major funders who are focusing on designing cannabis policy with a view to protecting public health. ## 1. What is the problem? Since 2012, four U.S. states have legalized recreational use and commercial production of cannabis.1 Broadly speaking, we consider it likely that the number of states where recreational cannabis consumption is legal will continue to increase. This view is based on conversations with drug policy experts,2 increased support for legalization in national polls,3 and our general impressions of the current political climate in the U.S. As discussed below, we believe that different ways of legalizing cannabis use are likely to have widely varying public health implications. In this investigation, we focused on these potential public health effects; we did not thoroughly consider other potential effects of legalization, including (for example) effects on arrests or incarceration. We chose to focus on this aspect because we see the question of how to legalize in a public-health-friendly way as receiving significantly less attention than the question of whether or not to legalize. We see the former question as potentially very important if legalization efforts continue to succeed (which we find likely). #### 1.1 Public health effects of commercial cannabis production Our understanding is that the public health impacts of cannabis legalization are likely to depend on the specifics of how legalization is carried out. However, our impression is that relatively little is known about how different kinds of regulatory regimes are likely to affect public health. We believe that it could be especially important for early-moving states to implement well-designed policy, as we consider it likely that later legalization models will be based on earlier ones (i.e. we expect the form that legalization takes overall to be relatively path dependent). Another consideration in favor of careful policy design early on is that it may be difficult to change existing policies once states have legalized cannabis, especially if, for example, a commercial marijuana industry were to emerge and become established.4 An example of how policy design might affect public health is the extent to which cannabis is allowed to be produced commercially. Legalization models that enable cannabis production to be fully commercialized may cause cannabis prices to drop considerably; we have heard estimates that prices under commercialization might be one tenth or less of current prices.5 Such a price change seems likely to lead to increased consumption of cannabis. In general terms, we expect increased consumption of intoxicants (like cannabis) to have humanitarian costs both for individuals and for society. In addition, our impression is that cannabis specifically can cause harm in a number of known ways. In this investigation, we focused on cannabis dependence as the main relevant harm. Cannabis dependence is associated with a variety of negative effects, including low energy and motivation; relationship and family problems; financial difficulties; and sleep and memory problems.6 #### 1.2 Size of the problem As part of our investigation, we estimated how much harm could potentially be averted by well-designed legislation. See footnote for more details of the calculation.7 In summary: • We assumed that the main harms of sub-optimal legalization would be due to increased cannabis dependence, then used our best guesses of how harmful dependence is, and how much it might increase under full commercialization, to estimate what the difference between well- and badly-designed legislation might be. • About 18 million people in the U.S. use cannabis at least once per month. • About 7% of these, or 1.25 million people, meet criteria for what we would consider morally-relevant cannabis dependence (although other sources report 12% have “met criteria for cannabis dependence in the past year”). • We use an estimated disability weight for cannabis dependence of 0.05 (note that this is lower than the Global Burden of Disease study’s disability weight for cannabis dependence, which is 0.329). • The estimates we’ve seen suggest that if legalization were accompanied by full commercialization (and therefore much lower prices), cannabis consumption would roughly double. • We assumed that a doubling in consumption would correspond to a doubling in dependent users. Combining these figures with a cost of$50,000 per disability adjusted life year (DALY), the cost of full commercialization comes out at roughly $3 billion, although we have very limited confidence in this estimate. This is relatively low compared to our rough estimates of the importance of other cause areas within U.S. policy. ## 2. What are possible interventions? We have focused our attention on ways to help cannabis legalization go as well as possible in jurisdictions where it does take place (rather than focusing on affecting the likelihood of legalization itself, which we see as drawing more attention from funders). Different legalization models could vary along many dimensions, including the following:8 • What type of organizations are allowed to provide cannabis? • What regulations apply to these organizations? • What types of products can be sold? • How much will the products cost? A funder in this space might consider supporting: • Research into different legalization models and their effects, e.g. analyzing previous instances of legalization, forecasting the effects of new proposals, or developing new legislative approaches. • Increased communication between cannabis legalization advocates and public health experts, e.g. holding meetings to discuss different approaches to legalization. Work of this kind could affect cannabis policy in the short term (e.g. by affecting the language of a specific ballot initiative) or the longer term (e.g. by promoting effective models of regulation which can be adopted more widely in the future, or more generally shifting the discourse around cannabis policy). ## 3. Who else is working on this? We do not have a comprehensive picture of which organizations actively support the careful design of cannabis-related policy, as opposed to supporting drug policy reform or cannabis legalization efforts in general. Our impression is that not many groups focus on this aspect of cannabis reform. The Open Society Foundations (OSF) funds some work on cannabis policy as part of its broader program of domestic drug policy reform but our understanding is that this is not a focus area for OSF.9 A number of other funders, including the Drug Policy Alliance, the Riverstyx Foundation, and the Libra Foundation, fund advocacy around improving drug policy more broadly, but do not focus on cannabis policy specifically as far as we know.10 ## 4. Questions for further investigation If we were to do further research here, questions we might aim to address include: • How responsive will legalization advocates be to proposed measures to protect public health? • What policy designs can best capture the benefits of cannabis legalization while preventing large declines in prices and attendant increases in dependence? • How does legalizing cannabis affect alcohol consumption patterns and their associated social costs? ## 5. Our process We have been interested in this area for several years. We have had many conversations with experts on the topics of criminal justice reform and drug policy reform, some of which have also included discussion of cannabis legalization. Public notes are available from several of these conversations: This area is also a personal interest of Cari Tuna and Dustin Moskovitz, co-founders of the foundation Good Ventures, who have made several personal gifts to support cannabis legalization efforts. Cari and Dustin are members of the Open Philanthropy Project Board of Managers. The Open Philanthropy Project has made grants to support the following: We are not currently planning further grants in this area in the near future, but may revisit the issue once we have a clearer sense of whether these grants have been successful. ## 6. Sources DOCUMENT SOURCE Budney et al. 2007 Source Burns et al. 2013 Source Caulkins 2013 Source Caulkins 2014a Source Caulkins 2014b Source Caulkins et al. 2012 Source Caulkins et al. 2015 Source GiveWell’s non-verbatim summary of a conversation with Open Society Foundations, November 19, 2012 Source Global Burden of Disease Study 2010 Source Mackin, Martin and McGavin 2007 Source NIDA – Cocaine, Scope of Use Source NIDA – DrugFacts, Treatment Statistics Source NIDA – Marijuana, Is Marijuana Addictive Source Office of National Drug Control Policy, State Laws Related to Marijuana Source Our non-verbatim summary of a conversation with Andy Ko, November 20, 2013 Source Our non-verbatim summary of a conversation with Open Society Foundations, November 19, 2012 Source Our non-verbatim summary of conversations with Gara LaMarche on May 22, May 31, and June 14, 2013 Source Our non-verbatim summary of conversations with Mark Schmitt on May 22 and June 14, 2013 Source Pew Research Center 2015 Source SAMHSA 2014 Source ## Federal Tax Reform Note: this is a shallow overview of a topic that we have not previously examined. For shallow overviews, we typically work for a fixed amount of time, rather than continuing until we answer all possible questions to the best of our abilities. Accordingly, this is not researched and vetted to the same level as our standard recommendations. If you have additional information on this cause that you feel we should consider, please feel free to get in touch. We use our shallow overviews to help determine how to prioritize further research. ## In a nutshell #### What is the problem? The federal tax system in the U.S. is inefficient, overly complicated, unlikely to be able to cover rising federal expenditures in the long run, and may constrain economic growth. #### What are possible interventions? Fundamental income tax reforms, including shifting the tax base to consumption or broadening the income tax base by eliminating many tax expenditures, may increase rates of economic growth and help address long-run fiscal issues. Smaller adjustments to federal tax policy may also have substantial benefits. These reforms face several political obstacles, and we do not have a strong sense of how additional funding would be able to create policy change. #### Who else is working on it? Federal tax reform efforts attract significant attention from many think tanks and foundations, including the Tax Policy Center and the Peter G. Peterson Foundation, amongst others. ## 1. What is the problem? Federal taxes in the U.S. are widely believed to be inefficient and overly complicated.1 Even taxpayers with simple tax situations spend considerable time and money filing income tax returns.2 Other individuals and businesses navigate a complex system of around 200 potentially applicable tax expenditures (e.g. deductions, exemptions, and credits).3 The system of tax expenditures makes federal taxes less progressive, narrows the overall base of taxable income, and reduces federal revenue, which may not be sustainable in the long run.4 Many specific tax expenditures reduce federal revenue without achieving any policy goals.5 More fundamentally, many scholars have argued that taxing income (rather than consumption) discourages savings and investment, which may negatively impact economic growth.6 ## 2. What policy changes could be helpful? We consider shifting the tax base from income to consumption or broadening the base of taxable income by eliminating many tax expenditures to be fundamental reforms of the federal tax system.7 We have seen arguments for the benefits of fundamental federal tax reforms: • Consumption taxes, including value-added taxes (VATs), the “X tax,” and the personal expenditure tax (PET), are simpler and create greater incentives for savings and investment than income taxes, though there seems to be disagreement about the magnitude of the gains from switching from an income to a consumption tax base.8 • VATs require businesses to pay taxes on the difference between their sales and their purchases of inputs (i.e. the business is taxed on the value it adds to products or services). Each business that adds value to a product or service is taxed, which leads to increased retail prices for consumers.9 VATs are well-studied since they have been implemented in 160 countries, and have been discussed as a supplement to or partial replacement of U.S. income taxes.10 Since low-income households spend a greater proportion of their income than wealthy households, VATs are regressive unless they are offset by another program.11 • The X tax is a VAT modified to be more progressive. Businesses pay taxes on value added, but are able to exempt wages from their tax liability. Workers are also taxed at a progressive rate on their wages.12 The X tax is usually discussed as a replacement, rather than just a supplement, for the federal income tax system.13 • The PET is another alternative intended to replace the income tax system. Households are progressively taxed on their total annual expenses, but businesses are not directly taxed.14 • Base-broadening, rate-reducing tax reform would eliminate many tax expenditures while still using income as the tax base. Eliminating tax expenditures increases the overall proportion of income taxed, allowing for (some combination of) lower tax rates and higher federal revenues.15 Lower income tax rates may also incentivize savings and investment.16 Fundamental tax reform that caused greater rates of savings and investment could in turn lead to greater rates of economic growth.17 A dynamic simulation model by Altig et al. 2001 finds that replacing the federal income tax with a progressive consumption tax could increase rates of economic growth and add hundreds of billions of additional dollars each year to the national income while maintaining current progressivity.18 Other fundamental reforms modeled by Altig et al. 2001 would also increase growth rates, but would be quite regressive.19 However, we interpret these growth rate estimates with caution since the models that yield these results rely on relatively strong assumptions about the effects of tax incentives on saving and investment behavior, which may not turn out to be realistic.20 More generally, economists seem to disagree considerably about the magnitude of economic gains that tax reforms might yield.21 We have also seen arguments for the benefits of non-fundamental federal tax reforms: • Simplifying tax filing: For most taxpayers, the IRS already has all the information it needs for tax filing.22 A “Simple Return” program, for which the IRS pre-files taxes for individuals with uncomplicated tax situations, could save individuals significant amounts of time and money. Goolsbee 2006 estimates that IRS pre-filing could collectively save up to 225 million hours of time and$2 billion a year paid in tax preparation fees.23
• Eliminating specific tax expenditures: Instead of focusing on the tax expenditure system as a whole (which also includes many social programs), reform efforts could target particularly inefficient or ineffective tax expenditures.24 The mortgage interest tax deduction, for example, may not actually promote home ownership as intended.25
• Limits on the use of tax expenditures: Reforms might also raise federal revenues by regulating the use of tax expenditures in upper tax brackets, or by placing a cap on the total permissible value of tax expenditures as relative to income.26

Carbon taxes and land value taxes are also mentioned in discussions of optimal tax policy, but we have not investigated them thoroughly for this overview.27

## 3. Prospects for reform

Federal tax reform faces several political obstacles:

• Our impression is that politicians are reluctant to support fundamental tax reform. Although some tax reforms could create widely-distributed long-term benefits, groups that would suffer losses would likely oppose the reforms,28 and the potential for widely-distributed benefits may be illusory.29
• Any fundamental reforms are likely to alienate powerful stakeholders on either the left or the right. The addition of VATs to the income tax system is often opposed both by those who want to limit government spending and those who are worried about the tax’s regressiveness.30 Similarly, an X tax may have difficulty gaining political support both because it would appear that wealthy citizens who primarily make their income from investments rather than wages would only be lightly taxed and because it would actually impose large one-time losses on those existing wealth-holders.31 A PET may also have difficulty gaining public support because no taxes are levied on businesses, and because individuals would be taxed on borrowed money.32
• Anti-tax groups have expressed opposition to reforms simplifying tax filing.33

On the other hand, projected increases in federal spending are expected to eventually require some type of revenue-increasing tax reform.34 In the near future, reforms that place a cap or limit on the use of tax expenditures seem more politically feasible than eliminating tax expenditures.35

All things considered, we’re pessimistic about the prospects of any fundamental federal tax reforms in the near future.36 Some of the smaller reforms discussed above may be considerably more tractable.37

## 4. Who already works on this?

Federal tax reform efforts attract significant attention from a wide variety of players, and we have not attempted to develop a full sense of the landscape.

Much of our assessment above has been informed by work from the Tax Policy Center, a joint venture of the Urban Institute and the Brookings Institution, which is a leading voice in tax policy discussions.38 Several major foundations, including the Bill and Melinda Gates Foundation, the John D. and Catherine T. MacArthur Foundation, the Ford Foundation, and the Rockefeller Foundation, support the Tax Policy Center.39

Our understanding is that many foundations tend to support tax reforms relevant to social programs, but not fundamental tax reform.40 A potential exception would be the Peter G. Peterson Foundation, which disbursed around $8.6 million in grants in the fiscal year 2013-2014, and is devoted to addressing long-term challenges to the federal budget.41 Amongst other things, it has promoted reducing the number of tax expenditures and simplifying the federal tax system as a response to projected long-term budget deficits.42 Many other advocacy and interest groups participate in tax policy discussions. ## 5. What could a new funder support? Funders interested in this area could support a variety of research and advocacy activities aimed at promoting reforms.43 We briefly explored the possibility that technical capacity to translate high-level reforms into detailed technical proposals or legislative text was a gap in the field, but the experts we talked to did not think this was the case.44 We have not otherwise investigated the expected impact of additional funding for different avenues of support. ## 6. Questions for further investigation Our research in this area has been relatively limited, and many important questions remain unanswered by our investigation. Further research on this cause might address: • How likely would further funding for fundamental tax reform research and advocacy be to accelerate reforms? • How likely is it for advocacy efforts in favor of simplifying tax filing or other non-fundamental reforms to overcome political opposition? Are there fairly tractable short-term opportunities to advance such reforms? • To what degree do the efficiency-improving justifications for fundamental tax reform apply to more modest reform efforts (e.g. reforming the mortgage interest deduction)? • What are the distributional consequences of various tax reform proposals? ## 7. Our process We decided to investigate this area due to our strong impression that the complexity of the U.S. tax code creates significant compliance costs, and due to our weak impression that alternative tax systems could positively impact economic growth. Our investigation consisted of conversations with tax policy experts and some limited desk research. Public notes are available from our conversations with: • Alan D. Viard,45 Resident Scholar, American Enterprise Institute • Bill Gale,46 Senior Fellow, Economic Studies Program, Brookings Institution; Co-Director, Urban-Brookings Tax Policy Center • Daniel Shaviro,47 Wayne Perry Professor of Taxation, New York University School of Law • David Kamin,48 Assistant Professor of Law, New York University • Richard England,49 Visiting Fellow, Lincoln Institute of Land Policy; Professor of Economics and Natural Resources, University of New Hampshire ## 8. Sources DOCUMENT SOURCE Altig et al. 2001 Source Barro 2015 Source Brown and Gale 2012 Source Burman 2011 Source Burman 2014 Source Goolsbee 2006 Source Matthews 2012 Source Nguyen et al. 2012 Source Our non-verbatim summary of a conversation with Alan D. Viard on March 25, 2014 Source Our non-verbatim summary of a conversation with Bill Gale on March 28, 2014 Source Our non-verbatim summary of a conversation with Daniel Shaviro on July 17, 2014 Source Our non-verbatim summary of a conversation with David Kamin on August 1, 2014 Source Our non-verbatim summary of a conversation with Richard England on March 27, 2014 Source Peterson Foundation 990 2013-2014 Source Peterson Foundation Revenues and Taxes 2010 Source Tax Policy Center Funders 2015 Source World Bank GNI 2015 Source ## Alcohol Taxation This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### What is the problem? The Centers for Disease Control and Prevention reports that excessive drinking causes tens of thousands of deaths and costs society hundreds of billions of dollars every year in the United States. #### What are possible interventions? We focus on alcohol excise taxes in this investigation. The evidence suggests that alcohol consumption is sensitive to price changes, and so alcohol excise taxes are likely to decrease consumption and related social costs. In general, the value of taxes on alcohol has eroded in the past few decades. However, some state-wide campaigns to raise alcohol taxes have been successful in recent years. A funder could support a variety of research or advocacy efforts at the state or federal level to encourage alcohol tax increases. #### Who else is working on it? A number of organizations support work related to alcohol abuse prevention and treatment, but our understanding is that few focus on alcohol tax policy, and we are not aware of any major active funders in the area. Some research is being conducted by the US government and other institutions. ## 1. What is the problem? Excessive drinking has been linked to significant costs to society, including increased liver disease, reckless driving, lost productivity, unsafe sex and STIs, and violent crime.1 Although the exact social cost is uncertain, the Centers for Disease Control and Prevention (CDC) reports that alcohol causes around 88,000 deaths (10% of all deaths among working-age adults) and costs the economy around$224 billion per year in the US.2

Binge drinking (defined as having four or more drinks in one session for women, and five or more for men) is by far the most common and socially harmful form of excessive alcohol consumption in the US.3 However, only a small fraction of binge-drinkers would be classified as “alcohol dependent.”4 Interventions that target binge-drinkers, not just people who suffer from alcohol dependence, may help substantially reduce social harms from excessive drinking.

## 2. What are possible interventions?

We began investigating alcohol tax policy after hearing from Mark Kleiman and Phil Cook that they found the research on the impact of alcohol taxes promising.5 In particular, Kleiman claimed that increasing alcohol taxes is likely one of the most effective ways to reduce crime.6

We have focused primarily on alcohol tax policy, which we see as having several advantages as an approach to reducing excessive drinking:

• We start with an assumption that raising prices will reduce consumption.
• The taxes would be relatively easy to implement and enforce, and governments are incentivized to enforce the law in order to collect revenue. 7
• There has been extensive research on the effects of alcohol excise taxes on harms from alcohol, which we discuss below.

We are aware of other potential interventions, but have not investigated them. These include:

• Supporting the implementation of social host ordinances. These ordinances render adults criminally and/or financially liable for underage drinking on their property.8
• Further restricting when and where individuals are able to purchase alcohol.9
• Imposing marketing restrictions on alcohol companies, especially for marketing aimed at young people.10
• Increasing the price of alcohol by restricting “happy hours” and other discounts.11
• “Swift and certain” sanctions (mild but almost immediate punishment for alcohol abuse) for people on probation for alcohol-related crimes.12
• Stricter enforcement of current laws and regulations, such as the minimum drinking age and driving under the influence laws.13
• Supporting litigation campaigns against alcohol companies.14

#### 2.1 Alcohol Taxation

The tax rate of alcohol in inflation-adjusted terms has decayed significantly in recent years.15 Federal alcohol taxes have not been raised since 1991, and many states have kept alcohol taxes level for decades.16 The experts we spoke with suggested that increasing alcohol taxes can reduce consumption and bring alcohol’s private costs more in line with its social costs.17 Heavy drinkers would bear most of the cost of the tax, and contribute most to raising revenue.18

The Open Philanthropy Project commissioned David Roodman to conduct a replication review of the existing literature to assess the evidence about the likely humanitarian impacts of raising alcohol taxes. He summarizes the large existing literature and estimates that a 10% increase in the price of alcohol would lead to roughly a 5% reduction in the amount of alcohol consumed.19 The best evidence also seems to indicate that higher alcohol prices cause a substantial decrease in alcohol-caused death (particularly due to cirrhosis).20 We have less information about possible tax-induced reductions in violence and traffic deaths.21 Overall, Roodman’s review estimates that a 10% tax-induced price hike in the US would reduce the number of alcohol-caused deaths by 9-25%, which would amount to 2,000-6,000 fewer deaths per year.22 Using the more expansive CDC definition of alcohol-caused deaths and assuming that alcohol tax increases would have the same proportional effect on them, a 10% price increase would lead to 8,000-22,000 fewer deaths per year in the U.S.23 However, our impression is that a 10% increase in prices is significantly larger than recent state alcohol tax increases, which have raised prices by more like 3%.24

#### 2.2 What has the track record of past advocacy efforts been?

David Jernigan reports that reform advocates attempt to increase alcohol taxes in 20-30 states per year, of which the vast majority fail.25

At least two states (Illinois and Maryland) have recently raised alcohol taxes, and the Maryland increase included a public health (not just a revenue-raising) rationale; it combined a public health message with a coalition of 1,200 groups and won a sales tax increase of three percent, or roughly five cents per drink, which amounted to a $75 million annual increase in state revenues.26 We do not know how feasible it would be to replicate the Maryland tax increase in other states. A consideration in favor of state (rather than federal) campaigns is the possibility that the success of several campaigns could create enough momentum to prompt changes in other states or at the federal level. A similar process occurred after several states first altered their tobacco tax policies in response to an increased understanding of the dangers of smoking.27 ## 3. Who else is working on this? Our understanding is that alcohol policy research and advocacy has experienced a decline in private funding and there is little support for alcohol policy reform from large foundations today.28 A number of organizations, such as Mothers Against Drunk Driving, work on alcohol policy.29 These organizations represent a strong alcohol abuse prevention and treatment community. However, our impression is that most concentrate their efforts on scientific research, alcohol addiction, abuse treatment, and decreasing drunk driving, rather than optimal alcohol tax policy.30 We are aware of only a couple small exceptions: • The Center on Alcohol Marketing and Youth (CAMY) offers research and advocacy training relating to alcohol tax policy increases.31 • The CDC funds a few million dollars’ worth of research and advocacy on alcohol policy each year, including supporting CAMY’s work on alcohol tax policy.32 ## 4. Questions for further investigation Our investigation in this area has been limited, and many important questions remain unanswered by our investigation. Amongst other topics, our further research on this cause area might address: • How would additional funding translate into additional advocacy efforts and to what extent is funding the bottleneck to advocacy success on alcohol tax issues? • How should we think about the “momentum” dynamics at the state and federal level?33 • What are the possible negative social and economic consequences of alcohol tax increases? • What other interventions might cost-effectively reduce the harms from alcohol? • Do people displace alcohol consumption with the consumption of other drugs when alcohol taxes rise? ## 5. Our process We initially decided to investigate this issue because Mark Kleiman, one of our criminal justice reform grantees, told us that excessive alcohol consumption was the most important public policy issue he knew of with no major advocate.34 While Kleiman was primarily focused on the impact of alcohol on crime, the bulk of our research has been on the public health aspects of alcohol consumption. We’ve spoken to roughly half a dozen individuals in our exploration of this issue to date. Public notes are available from our conversations with: We also commissioned David Roodman to conduct a replication review for the Open Philanthropy Project on the impacts of alcohol taxes on consumption and health outcomes. ## 6. Sources DOCUMENT SOURCE Roodman 2015 Source Our non-verbatim summary of a conversation with David Jernigan, August 6, 2014 Source Our non-verbatim summary of a conversation with James Mosher, August 12, 2014 Source Our non-verbatim summary of a conversation with Mark Kleiman, July 2, 2013 Source Our non-verbatim summary of a conversation with Phil Cook, July 29, 2014 Source Our non-verbatim summary of a conversation with the CDC’s Alcohol Program, September 5, 2014 Source Stahre et al. 2014 Source ## Governance of Solar Radiation Management This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### What is the problem? Solar radiation management is a type of geoengineering that aims to cool the earth by reflecting sunlight away from it. As a category, solar radiation management appears to be both riskier and closer to being ready for use than other types of geoengineering. However, there are currently no specific systems in place to govern research into solar radiation management, or its deployment at any scale. Whether or not solar radiation management turns out to be safe and beneficial, we believe improved governance would make it more likely that decisions about research and deployment of the technology are made wisely and in the interests of humanity as a whole. #### What are possible interventions? A philanthropist interested in supporting the governance of solar radiation management could fund research into possible approaches to governance, or encourage discussion of and education about this issue among decision-makers and the general public. #### Who else is working on it? Funding for governance initiatives is limited, and comes mostly from government-funded agencies. We believe there is currently little private philanthropy in this area. ## 1. What is the problem? #### 1.1 What is solar radiation management? During our investigation into geoengineering, we repeatedly heard about the lack of governance around the research and deployment of solar radiation management (SRM). SRM is a type of geoengineering which aims to reflect away a small percentage of the sunlight directed towards the earth. It is expected to have a cooling effect on the planet, but may not counteract other effects of high carbon dioxide concentrations (such as ocean acidification). SRM is one of two main types of geoengineering; the other is carbon dioxide removal (CDR), which aims to remove carbon dioxide from the atmosphere after it has already been emitted. We have focused on the governance of SRM because our impression is that as a technology it is generally cheaper, riskier, and closer to being ready for deployment.1 Several possible approaches to SRM have been proposed; the two that we have seen discussed most frequently are injecting sulfate particles into the stratosphere and using saltwater spray to brighten clouds.2 Both of these techniques have been brought to researchers’ attention by effects which have already been observed in the physical environment: the idea of injecting aerosols into the stratosphere is based on the cooling effect that follows the release of large amounts of sulfur dioxide into the stratosphere during volcanic eruptions; the idea of brightening clouds is based on observations of brightened areas of clouds produced by aerosol particles in the exhaust emissions of commercial cargo ships.3 Beyond these observations, the current scientific understanding of SRM comes mostly from modeling studies using climate simulations. Studies of this kind examining aerosols in the stratosphere suggest that it may be possible to create large cooling effects without the need for many (or any) major technological breakthroughs, and at a relatively low cost.4 We note that scientists and policymakers interested in SRM research emphasize that they do not see SRM as a replacement for climate change mitigation (i.e. global efforts to reduce greenhouse gas emissions).5Rather, it is generally considered worth developing as part of a suite of possible responses to climate change, or as a tool to be used only in the event of particularly dangerous or severe climate change.6 #### 1.2 What is SRM governance? ‘Governance’ in this context refers both to ‘hard’ governance, such as government regulations or international treaties, and ‘soft’ governance, such as codes of conduct or community norms. We believe it could be beneficial to support the development of governance mechanisms relating both to scientific research into SRM, and to possible deployment of SRM. (In the footnote, we describe two cases of potentially risky technologies which were subject to specific governance arrangements while under development, which could provide a precedent for governance arrangements around SRM research.)7 In this shallow investigation, we focus on governance of research, mostly due to our impression that this is likely to be relevant on a shorter time frame than governance of deployment. Possible approaches to the governance of SRM research could include researcher-driven codes of conduct, national regulations, or international treaties. #### 1.3 Why is SRM governance important? Deployment of SRM may be risky and its potential effects are poorly understood; if deployed, it might help reduce the harmful effects of climate change, but would likely have global effects which might be difficult to reverse; it also appears that it would be low-cost and relatively straightforward to deploy.8 Taken together, we find these to be strong reasons to support some form of governance in order to ensure, firstly, that research into SRM is well-directed and safe; and secondly, that if the technology is ever ready to be deployed, decisions about its use are made in the interests of the global community. We find that thinking in terms of possible worst-case scenarios clarifies why one could believe that improved governance of SRM could be important, without necessarily supporting the development or use of the technology itself. Roughly, we see four possible scenarios in which handling SRM poorly could cause bad outcomes: • If SRM is not used when it should have been, or too little SRM is used, or it is used too late (i.e. if extreme climate change occurs, and SRM could have safely reduced its effects but is not deployed); • If SRM is used when it should not have been, or too much is used, or it is used too soon (i.e. if someone capable of deploying SRM decides to do so, but it has large negative side effects, or if an experiment which is carried out causes damage in some way); • If SRM is successfully deployed, but its use is abruptly terminated in a way that causes harm (i.e. if SRM is being used to counteract the warming effects of high greenhouse gas levels, then if SRM suddenly stopped being used, the earth could potentially experience large, rapid rises in temperature);9 • If disagreements about SRM lead to conflict (we find this especially plausible if the benefits and harms of deployment would be unevenly spread, and one state or group were planning to act unilaterally). It is difficult to estimate how bad each of these four scenarios could be. Taking them in order: • Our page on extreme risks from climate change puts the chance of climate change having much worse effects than expected (greater than 6.4ºC warming by 2100) at around 10%, but we are highly uncertain about what this would mean for human welfare. • According to researchers, SRM might have unintended effects on many features of the global climate, including precipitation, atmospheric and oceanic circulation patterns, and ozone levels.10 We have not seen any research attempting to describe these changes in terms of humanitarian effects, but our understanding is that they could potentially be destructive. • We have heard from several people that the possible negative effects of ceasing to deploy SRM after it has been in use for some time represent a major unknown, and could be severe. In particular, the possibility of rapid warming resulting from SRM ‘termination’ is seen as potentially much more dangerous than more gradual warming due to gradual greenhouse gas accumulation. 11 • Likewise, although it is difficult to predict the results of a hypothetical conflict over SRM, it does not seem unreasonable to believe that the humanitarian costs could be very large, especially if one or more major powers were involved. We believe that these possible scenarios illustrate how improved governance of SRM could be important, even if it turns out that the technology is dangerous and should not be used. In each of the scenarios, robust governance arrangements that are specifically designed for the case of SRM would presumably make it more likely that good decisions are made about what research is done, and about how and whether to use SRM. If research is governed well, it will be less likely that dangerous experiments are carried out or that research gets halted prematurely; a well-functioning governance system for deployment would make it more likely that SRM deployment could be prevented if it would be dangerous, and supported if it would be beneficial, without resorting to armed conflict. The strongest consideration we have found against supporting the field (even if only indirectly, via supporting governance initiatives) is the possibility that it could lead to a ‘slippery slope’ towards deployment, sometimes called ‘technological lock-in’. The idea is that as research continues, it will gather momentum as well as funders and supporters whose influence will grow with the field.12 However, we feel that this risk is outweighed by the considerations outlined above. ## 2. What are possible interventions? Governance of SRM research is being actively discussed by climate change and geoengineering scientists, as well as policymakers. There have been repeated calls for the formation of some kind of governance strategy for SRM research; some common themes have surfaced, including points of agreement and questions for further deliberation. More detail in the footnote.13 We see several avenues that a philanthropist interested in supporting the governance of SRM research could pursue. These include: • Funding research into the governance of scientific fields in general, or of SRM research in particular. Research of this kind could include investigating precedent cases (i.e. other scientific fields which have been subject to field-specific governance), or developing proposals for the case of SRM research. • Supporting efforts to move towards the formation of some kind of governance agreement or roadmap. This could take the form of: • Supporting convenings or conferences for the relevant decision-makers; this could be some combination of SRM scientists, scientific policy experts, government officials, environmentalists, and others. • Outreach or education campaigns to increase public understanding and awareness of SRM and the issues surrounding it. • Advocacy in support of a particular approach to governance; this could be directed towards governments, scientists, international bodies, or funders, for example. At this point we have not put significant time into considering how to promote the governance of SRM deployment, although we believe that approaches similar to those listed above could be adapted for the development of governance systems for deployment. We also find it plausible that the systems developed to govern research will have significant influence on how deployment is eventually governed.14 ## 3. Who else is working on this? Below we list some agencies and funders we have identified as contributing funding to projects relating to SRM governance. During our 2013 investigation of geoengineering, we compiled a spreadsheet of solar geoengineering projects and their funding sources. Two of these projects are explicitly focused on governance, and we identified several others which relate at least partially to governance; funders of these projects appear below.15 We also list below the funders of the four geoengineering reports which have informed this investigation.16 Each report includes significant coverage of SRM governance issues. Funders of projects listed in our spreadsheet of geoengineering projects with at least some governance component: • National Science Foundation (NSF) • National Aeronautics and Space Administration (NASA) • National Oceanic and Atmospheric Administration (NOAA) • Central Intelligence Agency (CIA) • UK Natural Environment Research Council • UK Engineering and Physical Sciences Research Council • UK Economic and Social Research Council • UK Arts and Humanities Research Council • German Research Foundation (DFG) • German Federal Ministry of Education and Research (BMBF) • Brandenburg Ministry for Science, Research and the Arts • Academy of Finland’s Research Program on Climate Change (FICCA) • EU Seventh Framework Programme for Research (FP7) • Bill Gates Organizations which have funded reports on SRM: • The Royal Society • Bipartisan Policy Center • Environmental Defense Fund • The World Academy of Sciences • The Carbon War Room • Zennström Philanthropies • Fund for Innovative Climate and Energy Research (FICER) • Bipartisan Policy Center • National Academy of Sciences • NOAA • NASA • US Department of Energy Our impression, which is borne out by this list, is that most current funding for SRM governance comes from government agencies or government-funded bodies, while private philanthropy in this area is very limited.17 We would guess that total funding directed towards SRM governance is currently less than$10 million per year. When we investigated geoengineering in 2013, we identified approximately $11 million per year of funding for geoengineering projects which explicitly included a solar geoengineering (SRM) component.18 However, these projects do not necessarily relate to governance, and we believe only a fraction of that funding was directed towards SRM governance.19 We have not searched extensively for new SRM governance projects that could have received funding since 2013, although we tentatively believe that we would have heard about large new projects in the course of our conversations with experts in the field (more on our process below). In addition, our spreadsheet of projects does not account for research that is supported by general institutional resources (such as unrestricted funding to a university, graduate students’ stipends, or computing resources). We would plan to make further inquiries about new projects before committing significant resources to the field. ## 4. Questions for further investigation Our research in this area has been relatively limited, and many important questions remain unanswered by our investigation. If we were to do further research in this area, we might attempt to answer questions such as: • How has the development of other potentially dangerous technologies been governed? • How can a philanthropist contribute to the development of governance mechanisms for geoengineering? • Does public discussion of SRM either increase or decrease climate change mitigation efforts? • How likely is it that one state would decide to deploy SRM unilaterally? ## 5. Our process We were initially introduced to SRM governance as an issue during our investigations of climate change and geoengineering. During those investigations, and as part of this targeted investigation of SRM governance, we have spoken to experts on climate change, geoengineering and SRM governance. The following conversations contained at least some discussion of SRM governance and contributed to our understanding of the topic: We had several follow-up conversations with Andy Parker of SRMGI. We also read several papers by SRM scientists which were recommended to us during these conversations, as well as four major reports on geoengineering: by the Royal Society20, the Bipartisan Policy Center21, the SRM Governance Initiative (SRMGI)22, and the National Research Council23. Each provides a broad overview of the field of SRM governance at the time of publication. ## 6. Sources DOCUMENT SOURCE Bipartisan Policy Center 2011 Source (archive) FICER website 2015 Source (archive) GiveWell’s non-verbatim summary of a conversation with Jane C.S. Long, June 1, 2012 Source Morgan, Nordhaus and Gottlieb 2013 Source (archive) National Research Council 2015 Source (archive) Parson and Keith 2013 Source (archive) Robock 2008 Source (archive) Royal Society 2009 Source (archive) Schäfer et al. 2013 Source (archive) SRMGI 2011 Source (archive) ## Nuclear Weapons Policy This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### What is the problem? Nuclear risks range in magnitude from an accident at a nuclear power plant to an individual detonation to a regional or global nuclear war. Our investigation has focused on the risks from nuclear war, which, while unlikely, would have a catastrophic global impact. #### What are possible interventions? A philanthropist could fund research or advocacy aimed at reducing nuclear arsenals, preventing nuclear proliferation, securing nuclear materials from terrorists, or attempting to more directly prevent the use of nuclear weapons in a conflict (e.g. by working with civil society actors to reduce the risk of conflict). A funder could also raise awareness about risks from nuclear weapons in general by working with media or educators, or through grassroots advocacy. #### Who else is working on it? Several major U.S. foundations fund approximately$30 million/year of work on nuclear weapons issues, with most of this work supporting U.S.-based policy research and graduate/post-graduate education, some advocacy, and “track II diplomacy” (i.e. meetings between nuclear policy analysts and current and former government officials, often from different states). We do not have an estimate of funding from other non-profits in the space, but the Nuclear Threat Initiative has an annual budget of $17-18 million and is not primarily funded by foundations. The U.S., other governments, and the International Atomic Energy Agency spend much larger amounts of money managing risks from nuclear weapons. We see work on nuclear weapons policy outside of the U.S. and U.S.-based advocacy as the largest potential gaps in the field, with the former gap being larger, but also harder for a U.S.-based philanthropist to fill. ## 1. What is the problem? #### 1.1 Our focus on nuclear war There are numerous conceivable scenarios in which some sort of nuclear incident could occur, ranging from a meltdown at a nuclear power plant to the detonation of a “dirty bomb” (i.e. a bomb that combines radioactive material with conventional explosives) to an outright nuclear war between states.1< Though this is not a question we have thoroughly investigated, the risk of nuclear war between states strikes us as the most potentially destructive scenario because of the magnitude of some states’ nuclear arsenals, the possibility of wider escalation, and the possibility of nuclear winter. Accordingly, although we recognize the devastating potential of other kinds of nuclear incidents, our discussion below focuses on the risk from nuclear war.2 Note that we interpret efforts to address the risk of nuclear war broadly, to include issues like reduction of nuclear arsenals, prevention of nuclear proliferation, securing nuclear materials, and facilitating domestic civil discourse regarding nuclear weapons among countries that may seek to acquire nuclear weapons. The U.S. and Russia hold the vast majority of the world’s nuclear weapons:3 COUNTRY ESTIMATED NUCLEAR WEAPONS INVENTORY Russia 8,000 (4,300 in military custody) United States 7,300 (4,760 in military stockpile, 1,980 deployed) France 300 China 250 Britain 225 Pakistan 100-120 India 90-110 Israel 80 North Korea <10 Total ~16,300 Since the end of the Cold War, U.S. and Russia nuclear weapons inventories have greatly (and fairly continuously) declined, as illustrated by the graph below:4 We have not thoroughly investigated the probability or likely consequences of a nuclear detonation or a broader nuclear war, though we see both scenarios as possibilities. Our understanding is that the risk of global nuclear escalation has decreased substantially since the end of the Cold War.5 We note, however, that some people we spoke with suggested that total nuclear risk had increased since the Cold War.6 #### 1.2 Which conflicts are most worrisome? Though very unlikely in the present climate, a nuclear war between the U.S. and Russia would have the greatest destructive potential. A 1979 report by the U.S. Office of Technology Assessment estimated that in an all-out nuclear war between the U.S. and Russia, 35-77% of the U.S. population and 20-40% of the Russian population would die within the first 30 days of the attack.7 We have not vetted this estimate, but note that at the time, combined U.S./Russia nuclear weapons stockpiles were approximately three times as large as they are today, as seen in the graph above. An additional potential risk of a U.S./Russia nuclear would be the nuclear winter that might follow. Nuclear winter could potentially disrupt global food production and result in an even larger number of deaths, though we have not thoroughly explored the likelihood or likely consequences of this scenario. People we spoke with generally perceived the greatest risk of nuclear conflict in South Asia, where Pakistan has pledged to respond to any Indian attack on its territory with a nuclear bomb.8 Some scholars have argued that a war between India and Pakistan could alter the global climate, potentially threatening up to a billion people with starvation,9 though this estimate strikes us as high. Our understanding is that this claim is primarily based on: • Modeling the amount of smoke that would reach the stratosphere in the event of a nuclear war between India and Pakistan in which 100 nuclear weapons strike cities.10 • Using a climate model to simulate the effect of that smoke going into the stratosphere.11 • Using a crop model to estimate the effect of those climactic changes on the yields of crops in China and the Midwest.12 The last paper cited estimated that a 100-weapon nuclear exchange between India and Pakistan would result in a 10% decline in average caloric intake in China, with potentially similar consequences elsewhere.13 We have not vetted this analysis. However, even accepting its conclusions, our impression is that relatively little work has been done to consider the likely human consequences of the subsequent decline in agricultural production.14 We would guess that the predicted loss of crops would be insufficient to cause a billion people to starve for the following reasons: • China has significant food reserves, which, according to the paper estimating changes in the yields of crops cited above, would not be depleted until two years after the initial nuclear exchange.15 • Our impression is that the crop model was not designed with extreme scenarios like this in mind, and is not accounting for pressure to make significant changes to food production in the midst of a global crisis. • A substantial portion of crops grown are used to feed livestock, which is significantly less efficient than slaughtering existing livestock and directly eating the food we normally use to raise them. In the short run, livestock reserves could be slaughtered to meet demand for food.16 In the longer run, we would guess that a decrease in the food supply would raise food prices, creating incentives to produce more food (e.g. by using more land for food production and shifting away from less efficient animal-based means of production). ## 2. What are possible interventions? #### 2.1 Areas for nuclear policy work Almost any kind of progress on nuclear security ultimately requires some kind of change on the part of government, or the prevention of some change. Accordingly, much of the grant-making in this field focuses on: • Policy analysis by think tanks and university centers focused on nuclear weapons issues. • Advanced education to create better policy analysis in the future, such as support for academic centers that provide graduate and post-doctoral training in the field of nuclear weapons policy.17 • Advocacy and communications, which is a major focus of the Ploughshares Fund (discussed below). • Track II diplomacy—meetings between nuclear policy analysts and current and former government officials, often from different states. George Perkovich, Director of the Nuclear Policy Program at the Carnegie Endowment for International Peace, identified several different goals for this work:18 • Reduction of nuclear arsenals • Prevention of nuclear proliferation • Preventing terrorists from obtaining nuclear weapons • Direct efforts to prevent nuclear escalation Additionally, work can be categorized by the region the topic relates to (e.g. policy related to Iran, Russia, South Asia, North Korea, United States) and where the work is done (e.g. is the scholar/advocate working in the U.S. or India?). #### 2.1.1 Reduction of existing nuclear arsenals Work to reduce nuclear arsenals typically focuses on the five permanent members of the Non-Proliferation Treaty, which possess the largest numbers of nuclear weapons. Several approaches have been proposed for encouraging these countries to reduce their nuclear arsenals: • Laying the intellectual groundwork for further talks between the U.S. and Russia about reducing the number of warheads in each country’s stockpile • Advocating for the U.S. executive branch to reduce the U.S. nuclear stockpile19 • Supporting policy research and advocacy related to ballistic missile defense. Russia and China are concerned that U.S. investment in ballistic missile defense—i.e. systems for shooting down incoming ballistic missiles—could destabilize deterrence relationships. If ballistic missile defense were sufficiently effective, it could increase the number of nuclear weapons needed to ensure successful retaliation. Accordingly, U.S. investment in this area poses a potential obstacle to negotiating further arms reductions in Russia and China.20 Some organizations, such as Global Zero, advocate for the elimination of all nuclear weapons. We have not investigated the question of whether outright elimination is likely to be the best policy path, but we understand this to be an area of active debate within the academic community.21 The Ploughshares Fund is the largest funder of advocacy efforts with an annual budget of approximately eight million dollars. It argues in favor of cutting the U.S. federal budget for nuclear weapons.22 Joe Cirincione (President, Ploughshares Fund) and Philip Yun (Executive Director, Ploughshares Fund) identified the following advocacy opportunities for reducing the size of nuclear stockpiles in the U.S.: • Promoting public support for the ideas that nuclear weapons continue to pose significant risks, have limited deterrence value, and are costly to maintain. A philanthropist could seek to build support for these ideas by working with filmmakers or improving online educational materials and social media campaigns. • Supporting efforts to revive the U.S.-Russia dialogue on nuclear weapons. • Seeking to influence the nuclear policy of the next presidential candidates by supporting relevant policy analysis.23 • Advocating for lawmakers to reduce the U.S. nuclear arsenal. • Supporting policy analysis and advocacy opposing the development of ballistic missile defense in the U.S. in hopes of making U.S.-Russia and U.S.-China mutual arms reductions more likely (discussed above). While arms reductions in the U.S. and Russia may appear to be a natural target for a funder interested in reducing global catastrophic risk, we are uncertain about how much additional philanthropy could assist with arms reduction at this time. As noted above, the U.S./Russia weapons inventories have steadily declined since the end of the Cold War, suggesting that progress on the problem may continue in the absence of additional philanthropy. In addition, people we spoke with and consultants surveying the field for other funders saw limited opportunities for additional philanthropy to push forward U.S./Russia arms reductions.24 #### 2.1.2 Prevention of nuclear proliferation Non-proliferation work is focused on preventing additional countries from obtaining nuclear weapons. Within non-proliferation, the most common concern among people we spoke with was that Iran might obtain a nuclear weapon. For example, Joe Cirincione suggested that a philanthropist could advocate in favor of making a deal with Iran that would prevent Iran from obtaining nuclear weapons,25 which is a major focus for the Ploughshares Fund (see “Philanthropic Funders” below). If Iran obtains nuclear weapons, it could potentially destabilize the Middle East and encourage other countries to obtain nuclear weapons.26 In early 2015, after most of this review was already written, the U.S. and Iran reached an agreement on a framework for monitoring Iran’s nuclear program, though the deal has not yet been approved. We have a limited understanding of how this might affect philanthropic approaches related to non-proliferation in Iran.27 Some funders appear to focus on advocacy to individual countries that may attempt to acquire nuclear weapons, or on funding academic and civil discourse related to nuclear weapons in key regions through non-nuclear states such as Turkey, Brazil and South Korea.28 Others focus more on supporting international institutions such as the International Atomic Energy Agency (IAEA).29 Carl Robichaud—a Program Officer focusing on the International Peace and Security Program at the Carnegie Corporation of New York—identified the following options for a philanthropist to support the IAEA:30 • Fund open source analysis on topics that would be of use to the IAEA • Hold workshops where IAEA staff can learn how to utilize new tools and approaches, such as geospatial analytics and big data analysis (the Carnegie Corporation put on a workshop between IAEA staff and innovators from other sectors in December) • Support the Vienna Center for Disarmament and Non-Proliferation, which serves as research and training center for the IAEA • Sponsor dialogues within the IAEA to reduce politicization #### 2.1.3 Preventing terrorists from obtaining nuclear weapons Another approach would be to support appropriate monitoring of existing nuclear material in an attempt to prevent it from falling into the hands of terrorists. The difficulty and high costs of manufacturing nuclear weapons makes securing nuclear materials important for preventing nuclear terrorism.31 The Nuclear Threat Initiative (NTI) has been addressing this problem by using the Nuclear Security Summits to develop buy-in for global, enforceable nuclear security standards, which currently don’t exist.32 A nuclear terrorist attack is not a nuclear war and therefore not directly in the center of this investigation, but we would guess that nuclear terrorism—particularly in South Asia—could potentially ignite a nuclear war. A philanthropist could try to prevent terrorists from obtaining nuclear weapons through: • Policy development and advocacy related to creating a standard set of international norms regarding the security of nuclear materials.33 • Funding innovative demonstration projects in hopes of causing governments to scale them up. For example, Joan Rohlfing —President and COO of NTI—told us that the Nuclear Threat Initiative worked with Serbia to return non-secure nuclear materials to Russia, and that the effort led to the creation of a U.S. government program that spent billions of dollars on similar projects.34 Our impression is that the security of nuclear materials already receives significant attention. It is the primary focus of the Nuclear Threat Initiative,35 and a priority for national governments.36 #### 2.1.4 Direct efforts to prevent nuclear war Direct efforts to prevent nuclear war may focus on attempting to reduce the likelihood of deployment of nuclear weapons in a given conflict situation, or on attempting to reduce the risk of conflict between nuclear states. In our conversation, Dr. Perkovich gave some examples of both kinds of efforts with respect to India and Pakistan, focusing on various forms of civil society engagement, “track II” diplomacy, and policy research.37 We have not thoroughly explored this area, but people we spoke with have suggested there may be limited potential for additional philanthropy to address these issues, at least in Russia (as discussed above) and in South Asia.38 #### 2.2 Approaches to improving nuclear weapons policy and their track records #### 2.2.1 Policy analysis and advanced education A majority of funding supports policy research and advanced education, especially from the large foundations in this space.39 However, we have a fairly limited sense of the track record of policy analysis for improving nuclear weapons policy. Nevertheless, we understand that funding from the Carnegie Corporation and the MacArthur Foundation is generally believed to have played a major role in the passage of the Nunn-Lugar Act40, which was responsible for “the dismantling or elimination of 7,514 nuclear war-heads, 768 ICBMs, 498 ICBM sites, 155 bombers, 651 submarine-launched ballistic missiles, 32 nuclear submarines, and 960 metric tons of chemical weapons.”41 For context, according to one estimate, the U.S. and Russia held 19,008 and 29,154 nuclear weapons (respectively) when the Nunn-Lugar Act was passed in 1991, so that Nunn-Lugar was responsible for eliminating or dismantling approximately 15% of the total U.S./Russia nuclear arsenal.42 Policy research funded by philanthropists may have played a role in improving nuclear policy in a few other cases besides the Nunn-Lugar Act—such as establishing theories of deterrence and the New START treaty, which further reduced deployed warheads in the U.S. and Russia.43 Due to our limited understanding of nuclear weapons policy analysis, we also have a limited understanding of the potential impact of advanced education on nuclear weapons policy. #### 2.2.2 Advocacy Multiple people suggested to us that work on advocacy and communications is relatively neglected in nuclear weapons policy.44 Our investigation therefore focused more closely on opportunities within advocacy. In addition to the advocacy opportunities already mentioned (especially under the heading “Reduction of existing nuclear arsenals”), building general capacity for advocacy in order to change policy if a window of opportunity arises may be valuable.45 We are uncertain about the role for grassroots advocacy on nuclear weapons issues, in comparison with more technocratically-oriented advocacy and policy analysis. On this topic: • Robert Einhorn—a senior fellow with the Arms Control and Non-Proliferation Initiative and the Center for 21st Century Security and Intelligence at the Brookings Institution—cautioned that grassroots efforts have a mixed record at best in changing nuclear policy, pointing to the fact that the U.S. has not ratified the Comprehensive Nuclear Test Ban Treaty despite its support from a large majority of Americans.46 • Mr. Robichaud pointed to the New START treaty as an instance where public engagement played an important role.47 #### 2.3 Philanthropic opportunities in other countries Very little foundation funding supports nuclear policy work abroad, which means that other countries have much more limited capacity for developing and advocating for policy related to nuclear weapons,48 though U.S. foundations have funded some policy research in Russia and Asia: • The MacArthur Foundation’s Asia Security Initiative supported increased communication and dialogue between policy analysts working on security issues relevant to Asia, including supporting researchers in Asia. This program has since ended.49 • The Carnegie Endowment for International Peace has a satellite center in Moscow, and it has received funding from the Carnegie Corporation.50 A philanthropist looking to further support opportunities in other countries could: • Fund the creation of a center for nuclear policy research in Pakistan in hopes of increasing Pakistan’s willingness to comply with international nuclear law and norms.51 • Fund fellows programs for people from Asia, perhaps including an exchange component during which the fellows spend time at U.S. or U.K. institutions, to train the next generation of nuclear policy analysts.52 • Fund professors/senior researchers from U.S. nuclear policy institutions to visit think tanks in Asia and help train young nuclear policy analysts.53 However, people we spoke with suggested it would be challenging for a foundation to make its first grants on nuclear weapons policy in support of programs abroad54 and stressed the importance of having a local presence for monitoring grants.55 Our impression is that a philanthropist supporting research, education, and/or advocacy abroad would face significant challenges in terms of networking, communication, understanding context, and monitoring/evaluation. We distinguish between work done in other countries and work about the policies of other countries. While the former receives limited attention, the latter does not, and has been discussed in various contexts above. For example, work related to potential conflict in South Asia—where the greatest threat is perceived56—is relatively crowded. Nuclear issues in South Asia receive substantial attention in the form of programs at universities and think tanks57 as well as track II diplomacy, and some funders see little room for additional philanthropy on the topic.58 ## 3. Who else is working on this? #### 3.1 Government According to Gary Samore, Executive Director for Research at the Belfer Center for Science and International Affairs at Harvard:59 Nearly all government security agencies are involved with nuclear policy to some degree, including: • The White House International Security Council • The Department of Defense • The Department of State • The Department of Energy, involved in both nuclear energy and nuclear security (through the National Nuclear Security Administration) • Intelligence agencies We have not investigated the amount of funding these agencies devote to nuclear issues and have a limited understanding of their activities. But some agencies with large budgets focus primarily on keeping people safe from nuclear weapons: AGENCY BUDGET ACTIVITIES National Nuclear Security Administration (NNSA)$12.6B, of which $1.9B is categorized as non-proliferation.60 “NNSA maintains and enhances the safety, security, reliability and performance of the U.S. nuclear weapons stockpile without nuclear testing; works to reduce global danger from weapons of mass destruction; provides the U.S. Navy with safe and effective nuclear propulsion; and responds to nuclear and radiological emergencies in the U.S. and abroad.”61 Domestic Nuclear Detection Office (DNDO)$304M62 The DNDO coordinates U.S. government efforts to detect and prevent nuclear and radiological terrorism against the United States.63

In addition, some intergovernmental organizations devote substantial funding to nuclear security issues. For example, in 2014, the International Atomic Energy Agency had a budget of €344M.64
We currently have a very limited understanding of the activities of these U.S. government agencies and the IAEA. However, our understanding is that the government provides at most limited support for the primary areas addressed by philanthropic funders, such as policy development, advanced education, advocacy, and track II diplomacy.65

#### 3.2 Philanthropic funders

A 2012 report by Redstone Strategy Group, commissioned by the Hewlett Foundation, estimated that philanthropic funding for work on nuclear security between 2010 and 2012 was $31 million/year.66 Our impression is that the total funding in the field has not substantially changed, though some funders have exited the field. A few foundations account for the vast majority of this funding. FUNDER BUDGET (2014 ESTIMATE) FOCUS AREAS MacArthur Foundation ~$10M67 Policy research and advanced education in the U.S., focused on control of fissile materials and preventing the spread of nuclear weapons to terrorists.68
Carnegie Corporation of New York ~$10M69 Policy research, advanced education in the U.S., track II diplomacy. Focused on arms reductions, non-proliferation, and security of nuclear materials.70 Ploughshares Fund ~$8M, ~$5.5M in grants Advocacy, especially U.S. policy towards Iran and the U.S. nuclear budget.71 Hewlett Foundation Previously$4M (2012 estimate),72 exiting the field.73 Security of nuclear materials and reducing nuclear arsenals.74 Hewlett’s largest grants in this field were to the Carnegie Endowment for International Peace and the Ploughshares Fund.75
Sloan Foundation Previously $2.9 million (2012 estimate),76 exited the field.77 Policy research and advanced education.78 Skoll Global Threats Fund$1-2M Advocacy, especially U.S. policy toward Iran. Ploughshares is a major grantee.79
Stanton Foundation $2.3M (2012 estimate)80 Advanced education and policy research in the U.S.81 There are also some nonprofits that work on nuclear security issues that do not receive most of their funding from foundations, including, most notably, the Nuclear Threat Initiative (NTI), whose 2015 budget is about$17-18M. Of this, about 90% is devoted to nuclear weapons and about 30% is granted to other organizations (though NTI’s grants usually resemble contracts for service with partners who carry out specific projects that NTI has designed).82 Within nuclear weapons policy, NTI primarily emphasizes securing nuclear materials in order to prevent terrorism, with non-proliferation as a secondary emphasis.83 We do not have an overall accounting of activity by other nonprofits in this area, but a list of the top grant recipients in peace and security in 2008-2009 is available in a report by the Peace and Security Funders Group.84 We would guess that many of the major nonprofits working on nuclear weapons policy are listed there.
Foundations also provide substantial funding for peace and security that isn’t explicitly classified as work on nuclear weapons policy. According to the Peace and Security Funders Group, in 2008-2009, 91 U.S. foundations gave a total of $257M to promote peace and security, for an average of about$130M per year.85 The largest focus areas for these funders (measured by dollars granted) were:

• Controlling and Eliminating Weaponry (which is described as “mainly focused on nuclear weapons”)
• Prevention and Resolution of Violent Conflict
• Promoting International Security and Stability

#### 3.3 Crowdedness of different philanthropic approaches

This section primarily organizes information presented above in order to summarize relative crowdedness of different areas of work on nuclear weapons policy.
The largest potential gaps in this space appear to be work on nuclear weapons policy outside of the U.S. and U.S.-based advocacy, with the former gap being larger but harder for a U.S.-based philanthropist to fill.
As mentioned above, very little foundation funding supports nuclear policy work abroad, which means that other countries have much more limited capacity for developing and advocating for policy related to nuclear weapons.87 However, foundations have funded some work in this area:

• The MacArthur Foundation’s Asia Security Initiative supported increased communication and dialogue between policy analysts working on security issues relevant to Asia, including supporting researchers in Asia. This program has since ended.88
• The Carnegie Endowment for International Peace has a satellite center in Moscow, and it has received funding from the Carnegie Corporation.89

A majority of funding supports policy research and advanced education, especially from the large foundations in this space.90 The Ploughshares Fund is the largest foundation focused primarily on advocacy, and is spending about 8M per year. Multiple people suggested to us that work on advocacy and communications were relatively neglected in nuclear weapons policy.91 We have a more limited sense of which areas are most crowded in terms of regional focus (e.g. Iran, Russia, South Asia, North Korea, United States) or objective pursued (e.g. preventing new states from getting nuclear weapons, decreasing the number of nuclear weapons held by countries that already have them, or securing nuclear materials to prevent terrorists from gaining access to them). However, our impression is that work related to potential conflict in South Asia—where the greatest threat is perceived92—is relatively crowded. Nuclear issues in South Asia receive substantial attention in the form of programs at universities and think tanks93 as well as track II diplomacy, and some funders see little room for additional philanthropy on the topic.94 We also note that in early 2015, the U.S. and Iran reached an agreement on a framework for monitoring Iran’s nuclear program, though we have a limited understanding of how this might affect philanthropic approaches related to non-proliferation in Iran.95 Our impression is that the security of nuclear materials also receives significant attention. It is the primary focus of the Nuclear Threat Initiative,96 and a priority for national governments.97 ## 4. Questions for further investigation We have not deeply explored this field, and many important questions remain unanswered by our investigation. Amongst other topics, our further research on this cause might address: • What are the options for a funder seeking to support work on nuclear weapons policy abroad, particularly in Russia or South Asia? What comparable work has been done in the past, and what is its track record? • What other advocacy-based approaches could be pursued within nuclear weapons policy? • What other strategies are there for reducing U.S./Russia nuclear inventories? Have we missed any particularly promising strategies that are not being pursued as aggressively as they could be? How would the potential severity of a nuclear winter decline as nuclear inventories shrink? • What are the areas where policy research might lead to improved policy? What does such work typically involve? How has policy research in this field resulted in policy change in the past? Are any areas particularly neglected? • To what extent could work normally classified under “international peace and security” but not “nuclear weapons policy” or “nuclear security” contribute to reducing the risk of a nuclear war? • How likely is the detonation of a nuclear weapon or a broader nuclear escalation? Which sources or conflicts contribute the most to these risks? ## 5. Our process We initially decided to investigate the cause of nuclear safety because: • The potential devastation from the use of nuclear weapons is so great that an investment in nuclear safety could conceptually have high returns. • Unlike some other global catastrophic risks, there is an established philanthropic community working to address nuclear safety issues. We spoke with 12 individuals with knowledge of the field, including: • Joe Cirincione, President, Ploughshares Fund • Robert Einhorn, Senior Fellow, Brookings Institution • Megan Garcia, Program Officer for the Hewlett Foundation’s Nuclear Security Initiative • Erika Gregory, Founding Director, N Square: The Crossroads for Nuclear Security Innovation • Bruce Lowry, Director of Policy and Communications, Skoll Global Threats Fund • George Perkovich, Director of the Nuclear Policy Program at the Carnegie Endowment for International Peace. • Carl Robichaud, Program Officer, International Peace and Security, Carnegie Corporation • Joan Rohlfing, President and COO, Nuclear Threat Initiative • Gary Samore, Executive Director for Research, Belfer Center for Science and International Affairs, Harvard Kennedy School • Philip Yun, Executive Director and COO, Ploughshares Fund In addition to these conversations, we also reviewed documents that were shared with us and had some additional informal conversations. Previous version of this page here. ## 6. Sources DOCUMENT SOURCE Carnegie Corporation Annual Report, 2013 Source Carnegie Corporation Nuclear Security Program Website Source (archive) Department of Homeland Security, Budget-in-Brief: FY 2015 Source (archive) GiveWell’s non-verbatim summary of a conversation with Bruce Lowry, November 5, 2014 Source GiveWell’s non-verbatim summary of a conversation with Carl Robichaud, October 14, 2014 Source GiveWell’s non-verbatim summary of a conversation with Erika Gregory, September 22, 2014 Source GiveWell’s non-verbatim summary of a conversation with Gary Samore, September 8, 2014 Source GiveWell’s non-verbatim summary of a conversation with George Perkovich, June 6, 2013 Source GiveWell’s non-verbatim summary of a conversation with Joan Rohlfing, December 8, 2014 Source GiveWell’s non-verbatim summary of a conversation with Joe Cirincione, November 15, 2014 Source GiveWell’s non-verbatim summary of a conversation with Philip Yun, October 16, 2014 Source GiveWell’s non-verbatim summary of a conversation with Robert Einhorn, November 10, 2014 Source Hewlett Foundation Grants Database, 2015 Source (archive) Hewlett Foundation Nuclear Security Initiative Logic Model Source Hewlett Foundation Nuclear Security Initiative Webpage Source (archive) IAEA Regular Budget for 2014 Source (archive) Kristensen and Norris 2013 Source (archive) Kristensen and Norris 2014 Source (archive) MacArthur Foundation’s description of its nuclear security program, March 2014 Source (archive) MacArthur Foundation International Peace & Security Grant Guidelines, December 2014 Source (archive) National Nuclear Security Administration, About us Source (archive) National Nuclear Security Administration, Budget Source (archive) NTI 2012 Annual Report Source (archive) Nunn-Lugar Report for GiveWell’s History of Philanthropy Project by Benjamin Soskis, July 2013 (DOCX) Source Office of Technology Assessment 1979 Source (archive) Peace and Security Funders Group 2011 Source (archive) Ploughshares Fund blog post Source (archive) Redstone Strategy Group 2012 Source Robock and Toon 2009 Source (archive) Robock et al. 2007 Source (archive) Shulman 2012 Source (archive) Stanton Foundation International and Nuclear Security Webpage Source (archive) Toon et al. 2007 Source (archive) Vox, Meet the political scientist who thinks the spread of nuclear weapons prevents war Source (archive) Vox, The Iran nuclear deal translated into plain English, April 2015 Source (archive) Wikimedia Commons, US and USSR nuclear stockpiles Source (archive) Xia et al. 2013 Source (archive) ## Potential Risks from Advanced Artificial Intelligence We have updated our thinking on this subject since this page was published. For our most current content on this topic, see this blog post. This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### What is the problem? It seems plausible that some time this century, people will develop algorithmic systems capable of efficiently performing many or even all of the cognitive tasks that humans perform. These advances could lead to extreme positive developments, but could also potentially pose risks from intentional misuse or catastrophic accidents. For example, it seems possible that (i) the technology could be weaponized or used as a tool for social control, or (ii) someone might create an extremely powerful artificial intelligence agent with values misaligned with humanity’s interests. It also seems possible that progress along these directions could be surprisingly rapid, leaving society underprepared for the transition. #### What are possible interventions? A philanthropist could fund technical research aimed at ensuring the robustness, predictability, and goal-alignment of advanced artificial intelligence systems; research in law, ethics, economics, and policy related to advanced artificial intelligence; and/or education related to such research. #### Who else is working on this? Elon Musk has donated10 million to the Future of Life Institute for regranting to researchers focused on addressing these and other potential future risks from advanced artificial intelligence. In addition, a few relatively small nonprofit/academic institutes work on potential future risks from advanced artificial intelligence. The Machine Intelligence Research Institute and the Future of Humanity Institute each have an annual budget of about 1 million, and a couple of other new organizations work on these issues as well. ## 1. Background and process We have been engaging in informal discussions around this topic for several years, and have done a significant amount of reading on it. The debates are in some cases complex; this write-up focuses on reporting the primary factors we’ve considered and the strongest sources we know of that bear on our views. For readers highly interested in this topic, we would recommend the following as particular useful for getting up to speed: • Getting a basic sense of what recent progress in AI has looked like, and what it might look like going forward. Unfortunately, we know of no single source for doing this, as our main source has been conversations with AI researchers. However, we have extensive conversation notes forthcoming for one such conversation that can provide some background. • Reading Superintelligence by Nick Bostrom, which gives a detailed discussion of some potential risks, with particular attention to a particular kind of accident risk. (Those seeking a shorter and more accessible introduction might prefer two highly informal posts by blogger Tim Urban,1 who largely attempts to summarize Superintelligence as a layperson. We do not necessarily agree with all of the content of his posts, but believe they offer a good introduction to the subject.) We don’t endorse all of the arguments of this book, but it is the most detailed argument for a particular potential risk associated with artificial intelligence, and we believe it would be instructive to review both the book and some of the response to it (see immediately below). • Reviewing an Edge.org online discussion responding to Superintelligence.2. We feel that the arguments made in this forum are broadly representative of the arguments we’ve seen against the idea that risks from artificial intelligence are important. For our part, our understanding of the matter is informed by the following: • GiveWell staff have been informally discussing this topic with the Machine Intelligence Research Institute (an organization that focuses on it, formerly called the “Singularity Institute”) for some time, partly due to the fact that GiveWell’s audience has generally expressed significant interest in MIRI. For public notes from some relevant conversations, see the following: • In 2011, GiveWell spoke with Jasen Murray and others from the Singularity Institute for Artificial Intelligence on February 2011. • In 2011, GiveWell had conversations with Skype co-founder Jaan Tallinn, who is a major MIRI donor and went on to co-found CSER and FLI. • In 2012, Holden Karnofsky wrote a post discussing the work of the Singularity Institute. • In 2013, Holden Karnofsky, Jacob Steinhardt, and Dario Amodei (two science advisors for the Open Philanthropy Project) met with MIRI to discuss MIRI’s organizational strategy.3 In 2014, Holden had another similar meeting with MIRI.4 • We have had many informal conversations about the future of artificial intelligence with two of the Open Philanthropy Project’s scientific advisors: • Dario Amodei, Research Scientist, Baidu Silicon Valley AI Lab • Jacob Steinhardt, PhD student in Computer Science, Stanford University • In addition, we have published notes from conversations with some senior researchers in computer science, including: When we began our investigation of global catastrophic risks, we believed that this topic was worth looking into due to the high potential stakes and our impression that it was getting little attention from philanthropists. We were already broadly familiar with the arguments that this issue is important, and we initially focused on trying to determine why these arguments hadn’t seemed to get much engagement from mainstream computer scientists. However, we paused our investigations (other than keeping up on major new materials such as Bostrom 2014 and some of the critical response to it) when we learned about an upcoming conference specifically on this topic,5 which we attended. Since then, we have reviewed further relevant materials such as FLI’s open letter and research priorities document.6 ## 2. What is the problem? #### 2.1 Timeline According to many machine learning researchers, there has been substantial progress in machine learning in recent years, and the field could potentially have an enormous impact on the world.7 It appears possible that the coming decades will see substantial progress in artificial intelligence, potentially even to the point where machines come to outperform humans in many or nearly all intellectual domains, though it is difficult or impossible to make confident forecasts in this area. For example, recent surveys of researchers in artificial intelligence found that many researchers assigned a substantial probability to the creation of machine intelligences “that can carry out most human professions at least as well as a typical human” in 10-40 years. Following Muller and Bostrom, who organized the survey, we will refer to such machine intelligences as “high-level machine intelligences” (HLMI).8 More information about timelines for the development of advanced AI capabilities is available here. #### 2.2 Loss of control of advanced agents In addition to significant benefits, creating advanced artificial intelligence could carry significant dangers. One potential danger that has received particular attention—and has been the subject of particularly detailed arguments—is the one discussed by Prof. Nick Bostrom in his 2014 book Superintelligence. Prof. Bostrom has argued that the transition from high-level machine intelligence to AI much more intelligent than humans could potentially happen very quickly,9 and could result in the creation of an extremely powerful agent whose objectives are misaligned with human interests. This scenario, he argues, could potentially lead to the extinction of humanity.10 Prof. Bostrom has offered the two following highly simplified scenarios illustrating potential risks:11 • Riemann hypothesis catastrophe. An AI, given the final goal of evaluating the Riemann hypothesis, pursues this goal by transforming the Solar System into “computronium” (physical resources arranged in a way that is optimized for computation)— including the atoms in the bodies of whomever once cared about the answer. • Paperclip AI. An AI, designed to manage production in a factory, is given the final goal of maximizing the manufacture of paperclips, and proceeds by converting first the Earth and then increasingly large chunks of the observable universe into paperclips. Stuart Russell (a Professor of Computer Science at UC Berkeley and co-author of a leading textbook on artificial intelligence) has expressed similar concerns.12 While it is unlikely that these specific scenarios would occur, they are illustrative of a general potential failure mode: an advanced agent with a seemingly innocuous, limited goal could seek out a vast quantity of physical resources—including resources crucial for humans—in order to fulfill that goal as effectively as possible.13 To be clear, the risk Bostrom and Russell are describing is not that an extremely intelligent agent would misunderstand what humans would want it to do and then do something else. Instead, the risk is that intensely pursuing the precise (but flawed) goal that the agent is programmed to pursue could pose large risks. The above argument is difficult to briefly summarize and highly speculative, but we think it highlights plausible scenarios that seem worth considering and preparing for. Some considerations that make this argument seem relatively plausible to us, and/or point to a more general case for seeing AI as a potential source of major global catastrophic risks: • Over a relatively short geological timescale, humans have come to have enormous impacts on the biosphere, often leaving the welfare of other species dependent on the objectives and decisions of humans. It seems plausible that the intellectual advantages humans have over other animals have been crucial in allowing humans to build up the scientific and technological capabilities that have made this possible. If advanced artificial intelligence agents become significantly more powerful than humans, it seems possible that they could become the dominant force in the biosphere, leaving humans’ welfare dependent on their objectives and decisions. As with the interaction between humans and other species in the natural environment, these problems could be the result of competition for resources rather than malice.14 • In comparison with other evolutionary changes, there was relatively little time between our hominid ancestors and the evolution of humans. There was therefore relatively little time for evolutionary pressure to lead to improvements in human intelligence relative to the intelligence of our hominid ancestors, suggesting that the increases in intelligence may be small on some absolute scale. Yet it seems that these increases in intelligence have meant the difference between mammals with a limited impact on the biosphere and a species that has had massive impact. In turn, this makes it seem plausible that creating intelligent agents that are more intelligent than humans could have dramatic real-world consequences even if the difference in intelligence is small in an absolute sense.15 • Highly capable AI systems may learn from experience and run at a much faster serial processing speed than humans. This could mean that their capabilities change quickly and make them hard to manage with trial-and-error processes. This might pose novel safety challenges in very open-ended domains. Whereas it is possible to establish the safety of a bridge by relying on well-characterized engineering properties in a limited range of circumstances and tasks, it is unclear how to establish the safety of a highly capable AI agent that would operate in a wide variety of circumstances.16 • When tasks are delegated to opaque autonomous systems—as they were in the 2010 Flash Crash—there can be unanticipated negative consequences. Jacob Steinhardt, a PhD student in computer science at Stanford University and a scientific advisor to the Open Philanthropy Project, suggested that as such systems become increasingly complex in the long term, “humans may lose the ability to meaningfully understand or intervene in such systems, which could lead to a loss of sovereignty if autonomous systems are employed in executive-level functions (e.g. government, economy).”17 • It seems plausible that advances in artificial intelligence could eventually enable superhuman capabilities in areas like programming, strategic planning, social influence, cybersecurity, research and development, and other knowledge work. These capabilities could potentially allow an advanced artificial intelligence agent to increase its power, develop new technology, outsmart opposition, exploit existing infrastructure, or exert influence over humans.18 • Concerns regarding the loss of control of advanced artificial intelligence agents were included among many other issues in a research priorities document linked to in the open letter discussed above,19 which was signed by highly-credentialed machine learning researchers, scientists, and technology entrepreneurs.20 Prior to the release of this open letter, potential risks from advanced artificial intelligence received limited attention from the mainstream computer science community, apart from some discussions that we found unconvincing.21 We are uncertain about the extent to which the people who signed this open letter saw themselves as supporting the idea that loss of control of advanced artificial intelligence agents is a problem worth doing research to address. To the extent that they do see themselves as actively supporting more research on this topic, we see that as reason to take the problem more seriously. To the extent that they did not, we feel that signing the letter (without public comments or disclaimers beyond what we’ve seen) indicates a general lack of engagement with this question, which we would take as—in itself—a reason to err on the side of being concerned about and investing in preparation for the risk, as it would imply that some people in a strong position to be carefully examining the issue and communicating their views may be failing to do so. • Our understanding is that it is not clearly possible to create an advanced artificial intelligence agent that avoids all challenges of this sort.22 In particular, our impression is that existing machine learning frameworks have made much more progress on the task of acquiring knowledge than on the task of acquiring appropriate goals/values.23 #### 2.3 Peace, security, and privacy It seems plausible to us that highly advanced artificial intelligence systems could potentially be weaponized or used for social control. For example: • In the shorter term, machine learning could potentially be used by governments to efficiently analyze vast amounts of data collected through surveillance.24 • Cyberattacks in particular—especially if combined with the trend toward the “Internet of Things”—could potentially pose military/terrorist risks in the future.25 • The capabilities described above—such as superhuman capabilities in areas like programming, strategic planning, social influence, cybersecurity, research and development, and other knowledge work—could be powerful tools in the hands of governments or other organizations. For example, an advanced AI system might significantly enhance or even automate the management and strategy of a country’s military operations, with strategic implications different from the possibilities associated with autonomous weapons. If one nation anticipates such advances on the part of another, it could potentially destabilize geopolitics, including nuclear deterrence relationships. Our scientific advisor Dario Amodei suggested to us that this may be one of the most understudied and serious risks of advanced AI, though also potentially among the most challenging to address. Our understanding is that this class of scenarios has not been a major focus for the organizations that have been most active in this space, such as the Machine Intelligence Research Institute (MIRI) and the Future of Humanity Institute (FHI), and there seems to have been less analysis and debate regarding them, but risks of this kind seem potentially as important as the risks related to loss of control.26 #### 2.4 Other potential concerns There are a number of other possible concerns related to advanced artificial intelligence that we have not examined closely, including social issues such as technological disemployment and the legal and moral standing of advanced artificial intelligence agents. We may investigate these and other possible issues more deeply in the future. #### 2.5 Uncertainty about these risks We regard many aspects of these potential risks as highly uncertain. For example: • It seems highly uncertain when high-level machine intelligence might be developed. • Losing control of an advanced agent would seem to require an extremely broad-scope artificial intelligence, considering a wide space of possible actions and reasoning about a wide space of different domains. A “narrower” artificial intelligence might, for example, simply analyze scientific papers and propose further experiments, without having intelligence in other domains such as strategic planning, social influence, cybersecurity, etc. Narrower artificial intelligence might change the world significantly, to the point where the nature of the risks change dramatically from the current picture, before fully general artificial intelligence is ever developed. • Losing control of an advanced agent would also seem to require that advanced artificial intelligence will function as an agent: identifying actions, using a world model to estimate their likely consequences, using a scoring system (such as a utility function) to score actions as a function of their likely consequences, and selecting high- or highest-scoring actions. While it seems plausible that such agents will eventually be created, it also seems plausible that the creation of such agents could come after other artificial intelligence tools—which do not rely on an agent-based architecture—have been created. Elsewhere, Holden Karnofsky (co-founder of GiveWell) has argued that creating advanced non-agents before agents is plausible and could substantially change the strategic situation for those preparing for risks from advanced artificial intelligence.27 • It isn’t a given that superior intelligence, coupled with a problematic goal, would lead to domination of the biosphere. It’s possible (though it seems unlikely to us) that there are limited benefits to having substantially more intelligence than humans, and it’s possible that an artificial intelligence would maximize a problematic utility function primarily via degenerate behavior (e.g., hacking itself and manually setting its reward function to the maximum) rather than behaving in a way that could pose a global catastrophic risk. • It seems highly uncertain to us how quickly advanced artificial intelligence will progress from subhuman to superhuman intelligence. For example, it took decades for chess algorithms to progress from being competitive with the top few tens of thousands of players to being better than any human.28 At the same time, these risks seem plausible to us, and we believe the extreme uncertainty about the situation—when combined with plausibility and extremely large potential stakes—favors preparing for potential risks. We have made fairly extensive attempts to look for people making sophisticated arguments that the risks aren’t worth preparing for (which is distinct from saying that they won’t necessarily materialize), including reaching out to senior computer scientists working in AI-relevant fields (not all notes are public, but we provide the ones that are) and attending a conference specifically on the topic.29 We feel that the Edge.org online discussion responding to Superintelligence30 is broadly representative of the arguments we’ve seen against the idea that risks from artificial intelligence are important, and we find those arguments largely unconvincing. We invite interested readers to review those arguments in light of the reasoning laid out on this page, and draw their own conclusions about whether the former provide strong counter-considerations to the latter. We agree with Stuart Russell’s assessment that many of these critiques do not engage the most compelling arguments (e.g. by discussing scenarios involving conscious AI systems driven by negative emotions instead of scenarios where an advanced AI system causes harm by faithfully pursuing a badly specified objective).31 For a more comprehensive discussion of these and other critiques, see a collection of objections and replies created by Luke Muehlhauser, the former Executive Director of MIRI. Luke is a GiveWell research analyst, but he did not produce this collection as part of his work for us. We agree with much of Luke’s analysis, but we have not closely examined it and do not necessarily agree with all of it.32 ## 3. What are possible interventions? #### 3.1 Potential research agendas we are aware of Many prominent33 researchers in machine learning and other fields recently signed an open letter recommending “expanded research aimed at ensuring that increasingly capable AI systems are robust and beneficial,” and listing many possible areas of research for this purpose.34 The Future of Life Institute recently issued a request for proposals on this topic, listing possible research topics including:35 • Computer Science: • Verification: how to prove that a system satisfies certain desired formal properties. (“Did I build the system right?”) • Validity: how to ensure that a system that meets its formal requirements does not have unwanted behaviors and consequences. (“Did I build the right system?”) • Security: how to prevent intentional manipulation by unauthorized parties. • Control: how to enable meaningful human control over an AI system after it begins to operate. • Law and ethics: • How should the law handle liability for autonomous systems? Must some autonomous systems remain under meaningful human control? • Should some categories of autonomous weapons be banned? • Machine ethics: How should an autonomous vehicle trade off, say, a small probability of injury to a human against the near-certainty of a large material cost? Should such trade-offs be the subject of national standards? • To what extent can/should privacy be safeguarded as AI gets better at interpreting the data obtained from surveillance cameras, phone lines, emails, shopping habits, etc.? • Economics: • Labor market forecasting • Labor market policy • How can a low-employment society flourish? • Education and outreach: • Summer/winter schools on AI and its relation to society, targeted at AI graduate students and postdocs • Non-technical mini-schools/symposia on AI targeted at journalists, policymakers, philanthropists and other opinion leaders. FLI also requested proposals for centers focused an AI policy, which could address questions such as:36 • What is the space of AI policies worth studying? Possible dimensions include implementation level (global, national, organizational, etc.), strictness (mandatory regulations, industry guidelines, etc.) and type (policies/monitoring focused on software, hardware, projects, individuals, etc.) • Which criteria should be used to determine the merits of a policy? Candidates include verifiability of compliance, enforceability, ability to reduce risk, ability to avoid stifling desirable technology development, adoptability, and ability to adapt over time to changing circumstances to prevent intentional manipulation by unauthorized parties. • Which policies are best when evaluated against these criteria of merit? Addressing this question (which is anticipated to involve the lion’s share of the proposed work) would include detailed forecasting of how AI development will unfold under different policy options. This agenda is very broad, and open to multiple possible interpretations. Research agendas have also been proposed by the Machine Intelligence Research Institute (MIRI) and Stanford One Hundred Year Study on Artificial Intelligence (AI100). MIRI’s research tends to involve more mathematics, formal logic, and formal philosophy than much work in machine learning.37 Some specific research areas highlighted by our scientific advisors Dario Amodei and Jacob Steinhardt include: 1. Improving the ability of algorithms to learn values, goal systems, and utility functions, rather than requiring them to be hand-coded. Work on inverse reinforcement learning and weakly supervised learning could potentially contribute to this goal. 2. Improving the calibration of machine learning systems, i.e., their ability to accurately distinguish between predictions that are highly likely to be right vs. predictions that are based on potentially confusing data and could be dramatically wrong. 3. Making decisions/conclusions made by machine learning systems easier for humans to understand. 4. Making the performance of machine learning systems more robust to changes in context. 5. Improving the user interfaces of machine learning systems. Sustained progress in these areas could potentially reduce risks from unintended consequences—including loss of control—of future artificial intelligence systems.38 #### 3.2 Is it possible to make progress in this area today? It seems hard to know in advance whether work on the problems described here will ultimately reduce risks posed by advanced artificial intelligence. At this point, we feel the case comes down to the following: • Currently, work in this field receives very little attention from researchers dedicated to addressing the issues we have described (see “Who else is working on this?”), and very little of this attention has come from researchers with substantial expertise in machine learning. • However, as mentioned above, many researchers in machine learning have recommended expanded research in this field. Moreover, FLI has received over 300 grant applications, requesting a total of nearly100 million for research in this area.39.
• It’s intuitively plausible to us (and to our main advisors on the topic at this time, Dario Amodei and Jacob Steinhardt) that success on some items on the above research agendas could result in decreased risk.

Because the largest potential risks are probably still at least couple of decades away, a substantial risk of working in this area is that, regardless of what we do today, the most important work will be done by others when the risks become more imminent and comprehensible, making early efforts to prepare for the problem redundant or comparably inefficient. At the same time, it seems possible that some risks could come on a faster timeline. For example, many researchers in Bostrom’s survey described above assigned a 10% subjective probability to the creation of machine intelligences “that can carry out most human professions at least as well as a typical human” within 10 years.40 We have not vetted these judgments and believe it would be challenging to do so. We are highly uncertain about how much weight to put on the specific details of these judgments, but they suggest to us that very powerful artificial intelligence systems could exist relatively soon. Moreover, it may be important to have a mature safety-oriented research effort underway years or longer before advanced artificial intelligence (including advanced narrow artificial intelligence) is created, and nurturing that research effort could be a long-term project. Alternatively, even if advanced AI will not be created for decades, it’s possible that building up and shaping the field could have consequences decades later. So it seems possible that work today could potentially increase overall levels of preparation for advanced artificial intelligence.41

Finally, much of the research relevant to long-term problems may overlap with short-term problems, such as the role of artificial intelligence in surveillance, autonomous weapons systems, and unemployment. Even if work done in this field today does not affect very long-term outcomes with artificial intelligence, it could potentially affect these issues in the shorter term.42

#### 3.3 Could supporting this field lead to unwarranted or premature regulation?

Our opinion is that the potential risks and policy options in this field are currently poorly understood, and advocating for regulation would be premature. A potential risk of working in this field is that it could cause unwarranted or premature regulation to occur, which could be counterproductive. While supporting work in this space could potentially have that result, we would guess that working in this field would be more likely to reduce the risk of premature or unwarranted regulation for the following reasons:

• We would guess that more thoughtful attention to policy options would reduce the risk of unwarranted regulation, and make regulation more likely to occur only if it turns out to be needed.
• The field may eventually be regulated regardless of whether funders pay additional attention to it, and additional attention to this set of issues could potentially make the regulation more likely to be thoughtful and effective.
• We would guess that technical research (in contrast with social science, policy, law, and ethics research) on these issues would be particularly unlikely to increase regulation. While such work could potentially draw attention to potential safety issues and thereby make regulation more likely, it seems more plausible that if computer science researchers were perceived to pay greater attention to the relevant potential risks, this would decrease the perceived need for regulation.

## 4. Who else is working on this?

In 2015, Elon Musk announced a $10 million donation to support “a global research program aimed at keeping AI beneficial to humanity.” The program is being administered by the Future of Life Institute, a non-profit research institute in Boston led by MIT professor Max Tegmark.43 FLI issued a first call for proposals from researchers at the beginning of the year; we have a forthcoming write-up that further discusses this work. The sort of research they are funding is described above (see “What are the possible interventions?). #### 4.2 Organizations working in this space A few small non-profit/academic institutes work on risks from artificial intelligence, including: ORGANIZATION MISSION REVENUE OR BUDGET FIGURE Cambridge Center for the Study of Existential Risk “CSER is a multidisciplinary research centre dedicated to the study and mitigation of risks that could lead to human extinction.”44 Not available, new organization Future of Humanity Institute “The Future of Humanity Institute is a leading research centre looking at big-picture questions for human civilization. The last few centuries have seen tremendous change, and this century might transform the human condition in even more fundamental ways. Using the tools of mathematics, philosophy, and science, we explore the risks and opportunities that will arise from technological change, weigh ethical dilemmas, and evaluate global priorities. Our goal is to clarify the choices that will shape humanity’s long-term future.”45 About$1 million annual budget for 201346
Future of Life Institute “We are a volunteer-run research and outreach organization working to mitigate existential risks facing humanity. We are currently focusing on potential risks from the development of human-level artificial intelligence.”47 Not available, new organization
Machine Intelligence Research Institute “We do foundational mathematical research to ensure smarter-than-human artificial intelligence has a positive impact.”48 1,237,557 in revenue for 201449 One Hundred Year Study on Artificial Intelligence (AI100) “Stanford University has invited leading thinkers from several institutions to begin a 100-year effort to study and anticipate how the effects of artificial intelligence will ripple through every aspect of how people work, live and play.”50 Not available, new organization CSER, FHI, and FLI work on existential risks to humanity in general, but all are significantly interested in risks from artificial intelligence.51 ## 5. Questions for further investigation Amongst other topics, our further research on this cause might address: • Is it possible to get a better sense of how imminent advanced artificial intelligence is likely to be and the specifics of what risks it might pose? • What kinds of technical research are most important for reducing the risk of unexpected/undesirable outcomes from progress in artificial intelligence? Who are the best people to do this research? • What could be done—especially in terms of policy research or advocacy—to reduce risks from the weaponization/misuse of artificial intelligence? • Could a philanthropist help relevant fields develop by supporting PhD, postdoctoral, and/or fellowship programs? What would be the best form for such efforts to take? • To what extent could approaches and funding models for other fields—such as international peace and security or nuclear weapons policy—successfully be adapted to the risks posed by artificial intelligence? • What is the comparative size of the risk from intentional misuse of artificial intelligence (e.g. through weaponization) vs. loss of control of an advanced artificial intelligence agent with misaligned values? ## 6. Sources DOCUMENT SOURCE AI 100 About Page Source (archive) AI 100 Reflections and Framing Source (archive) Bostrom 2014 Source CSER about page, 2015 Source (archive) FHI about page, 2015 Source (archive) FLI about page, 2015 Source (archive) FLI blog, AI grant results, 2015 Source (archive) FLI conference page, 2015 Source (archive) FLI International Grants Competition, 2015 Source (archive) FLI Musk donation announcement, 2015 Source (archive) FLI Open Letter, 2015 Source (archive) FLI research priorities document, 2015 Source (archive) FLI survey of research questions, 2015 Source (archive) GiveWell’s conversations with Jaan Tallinn, 2011 Source (archive) GiveWell’s non-verbatim summary of a conversation with Jasen Murray and others from the Singularity Institute for Artificial Intelligence, February 2011 Source GiveWell’s non-verbatim summary of a conversation with Stuart Russell, February 28, 2014 Source GiveWell’s non-verbatim summary of a conversation with Tom Dietterich, April 28, 2014 Source GiveWell’s non-verbatim summary of a conversation with Tom Mitchell, February 19, 2014 Source Grace 2015 Source (archive) Holden Karnofsky, Thoughts on the Singularity Institute, 2012 Source (archive) MIRI blog, 2014 in review Source (archive) MIRI Existential Risk Strategy Conversation with Holden Karnofsky, 2014 Source (archive) MIRI home page, 2015 Source (archive) MIRI Research Agenda, 2015 Source (archive) MIRI strategy conversation with Steinhardt, Karnofsky, and Amodei, 2013 Source (archive) Muehlhauser objections and replies 2015 Source (archive) Muller and Bostrom 2014 Source (archive) Nick Beckstead’s non-verbatim summary of a conversation with Sean O hEigeartaigh, April 24, 2014 Source (archive) Steinhardt 2015 Source (archive) The Myth of AI Source (archive) Wait But Why on AI, Part 1 Source (archive) Wait But Why on AI, Part 2 Source (archive) Yudkowsky 2013 Source (archive) ## Geomagnetic Storms This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### What is the problem? A severe solar storm might have the potential to shut down power grids on a continental scale for months. #### Who is already working on it? Power companies, transformer makers, insurers, and governments all have an interest in protecting the grid from geomagnetic storms. As far as we know, there is little philanthropic involvement in this issue. #### What could a new philanthropist do? The grid can be protected through hardening and through the installation of ground-induced current blocking devices that would prevent the currents generated by a geomagnetic storm from flowing through the grid. A philanthropist could fund further research on the threat posed by geomagnetic storms or on mitigation possibilities, fund advocacy for dealing with the threat, or directly fund mitigation. ## 1. What is the problem? Eruptions from the sun bombard the earth with energetic matter, called coronal mass ejections (CMEs). CMEs damage satellites and, by temporarily disrupting Earth’s magnetic field, can disrupt the operation of power grids. In extreme cases, roughly once per decade, CMEs reach Earth within 24 hours.1 Whether a CME hits Earth depends on its direction and angular width. In July 2012, for instance, a powerful CME with an estimated angular width of 160° missed Earth because it launched during a week when its source region on the sun faced away from Earth.2 When a CME hits the earth, it can damage satellites, including ones critical for communication and navigation.3 It may also induce turbulence in the magnetic field on the planet’s surface, which in turn can generate abnormal currents in long-distance power lines.4 The intensity of these effects depends on the speed and magnetic strength and orientation of a CME.5 In March 1989, a major geomagnetic storm destabilized the grid in Québec enough to force it to shut down within minutes.6 This damaged equipment, including two major transformers, and blacked out most of the province. 83% of power was restored within nine hours.7 After, the Canadian government invested1.2 billion in equipment upgrades intended to make the Québec grid more robust to storms.8

In 1859, a storm approximately twice as powerful as the 1989 one occurred, though it caused no major damage because there was little electrical infrastructure at the time.9

According to John Kappenman, a consultant who works on geomagnetic storms, a 1 in 100-200 years worst-case geomagnetic storm could destroy large transformers throughout the world and cause a global power outage that would take years to fix. The knock-on effects for other infrastructure–hospitals, police, pipelines, food delivery—could cause a humanitarian disaster.10 Other estimates appear to be substantially less aggressive: the North American Electric Reliability Corporation, a power industry group, reported that only older transformers would be likely to be damaged in a severe geomagnetic storm, and the US Department of Homeland Security noted that Kappenman was the only source of more extreme estimates of damages from geomagnetic storms.11 A 2011 report for the OECD concluded that the threat from geomagnetic storms is not well understood.12

Besides the risk to the power grid, geomagnetic storms also threaten satellites and aviation.13

At the completion of this shallow review in May 2014, we did not feel that we had a good understanding of the degree of risk from geomagnetic storms, though we guessed, with low confidence, that Kappenman’s 1 in 100-200 year figure for a globally devastating storm is likely to overstate the degree of risk. The deep dive has made us more confident in this estimate.14 Nevertheless, the historical record from which to infer probabilities is short, and the responses of electric grids to storms are not well studied. Given the high humanitarian stakes, we believe the threat may well offer opportunities for philanthropy.

A high-altitude detonation of a single nuclear weapon, known as an electromagnetic pulse (EMP) attack, could cause similar effects as for a geomagnetic storm over an area the size of the continental US.15

## 2. Who is already working on it?

Power companies, state and federal governments, and insurance companies all have a stake in responding to geomagnetic storms:

• Power companies presumably want to protect their infrastructure from damage.16 We have not investigated the extent to which power companies have responded to geomagnetic storm risks.
• Insurance companies might be able to pressure power companies to protect their assets from geomagnetic storms by promising lower premiums for those who take this step.17 Insurance companies appear to be considering how to incorporate geomagnetic storms into their risk models.18
• The US federal government has been concerned by the risk of electromagnetic pulse attack (which would cause similar damage and require similar mitigation strategies to a geomagnetic storm) since the Cold War, although its commitment to mitigating the damage from such an attack may have lessened since then.19The SHIELD Act, which would require power companies to mitigate the threat from geomagnetic storms, has been introduced in the House but, as of July 2013, had not been voted on by either chamber.20
• The Federal Energy Regulatory Commission (FERC) is charged with enforcing geomagnetic storm-related regulations on the power industry. FERC works with an industry group, the North American Electric Reliability Corporation (NERC) to devise standards, which FERC can then either accept or remand to NERC for revision. As of May 2015, NERC had issued a detailed draft reliability standard relating to geomagnetic disturbances, for a 60-day period of public comment.21
• To our knowledge, Maine is the only state to have passed a law requiring power installations to be robust to EMPs and geomagnetic storms.22

A small number of advocates like Kappenman currently try to persuade NERC and the government to do more about the threat from geomagnetic storms, but as far as we know, there is very little philanthropic involvement in this issue.23

## 3. What could a new philanthropist do?

A philanthropist could potentially pursue a number of different approaches aiming to reduce risks from geomagnetic storms:24

• further research on the risks of geomagnetic storms and potential mitigation strategies
• advocacy for stronger geomagnetic safety standards for electric utilities, or for public funding to support mitigation efforts
• directly funding mitigation in partnership with electric utilities.

We do not have a strong sense of the likely returns to any of these strategies, though we would guess that the research and advocacy approaches would carry higher expected returns than direct support for mitigation.

#### 3.1 Approaches to mitigating geomagnetic storm risk

The two basic options available to protect the grid are operational mitigation and hardening. Operational mitigation entails operating the grid in such a way as to reduce the threat from geomagnetic storms.25 The most radical kind of operational mitigation would be to unplug grid components in advance of a predicted geomagnetic storm so that they are not vulnerable to the effects of the storm. This strategy is feasible because satellites can predict periods of a few days when geomagnetic storms are likely.26 However, the national grid would likely have to be shut down for several days, which would cause enormous economic damage.27

Kappenman believes that for $1 billion, the US grid could be hardened to resist the effects of geomagnetic storms with ground-induced current (GIC) blockers.28 However, GIC blocking devices may turn out to be more expensive than Kappenman estimates, and a more diversified hardening strategy might be necessary to protect the grid.29 It is possible to undertake some operational mitigation and hardening for satellites, though both approaches face challenges.30 ## 4. Questions for further investigation Our research in this area has yet to answer many important questions. Amongst other topics, our further research on this cause might address: • How much attention do national governments (including the U.S.) pay to the threat of geomagnetic storms and EMP attack? • How do power companies currently respond to the threat posed by geomagnetic storms? • How likely is further research to pay off in better estimates of the likely damage from severe geomagnetic storms and in better mitigation strategies? Are better estimates already available from experts we did not contact? • How significant is the risk of an EMP attack? ## 5. Our Process We initially decided to investigate geomagnetic storms because we thought that damage to the power grid from geomagnetic storms might be a serious risk that is relatively easy to quantify and interventions to mitigate the risk might be relatively tractable. The investigation that went into this shallow review has been very limited, consisting primarily of review of risk assessments by government agencies and other actors and a conversation with John Kappenman, the owner of Storm Analysis Consultants. In late 2014, we commissioned a “deep dive” investigation. The report focuses on assessing the probability of an extreme storm, with shallower coverage of the impacts on grids and no discussion of options for limiting them. Previous version of this page here. ## 6. Sources Kappenman Comments Before the FERC – Source Kappenman 2010 – Source Lloyd’s 2013 – Source NRC 2008 – Source NERC 1990 – Source Notes from a conversation with John Kappenman, 8/6/2013 – Source Odenwald 2000 – Source Roodman 2015 – Source Severe Space Weather Events 2008 – Source Siscoe, Crooker, and Clauer 2006 – Source World Data Center for Geomagnetism – Source Wales 2012 – Source DOCUMENT SOURCE Baker et al. 2013 Source CENTRA 2011 Source Cliver and Svalgaard 2004 Source DHS Office of Risk Management and Analysis 2011 Source FERC 2015 Source Foster et al. 2004 Source Geomagnetic Disturbance Task Force 2012 Source Gopalswamy 2006 Source Kappenman 2005 Source Kappenman 2006 Source Kappenman Comments Before the FERC Source Kappenman 2010 Source Lloyd’s 2013 Source NRC 2008 Source NERC 1990 Source Notes from a conversation with John Kappenman, 8/6/2013 Source Odenwald 2000 Source Roodman 2015 Source Severe Space Weather Events 2008 Source Siscoe, Crooker, and Clauer 2006 Source World Data Center for Geomagnetism Source Wales 2012 Source ## Risks from Atomically Precise Manufacturing This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research. ## In a nutshell #### Background: Atomically precise manufacturing is a proposed technology for assembling macroscopic objects defined by data files by using very small parts to build the objects with atomic precision using earth-abundant materials. There is little consensus about its feasibility, how to develop it, or when, if ever, it might be developed. This page focuses primarily on potential risks from atomically precise manufacturing. We may separately examine its potential benefits and development pathways in more detail in the future. #### What is the problem? If created, atomically precise manufacturing would likely radically lower costs and expand capabilities in computing, materials, medicine, and other areas. However, it would likely also make it substantially easier to develop new weapons and quickly and inexpensively produce them at scale with an extremely small manufacturing base. In addition, some argue that it would help make it possible to create tiny self-replicating machines that could consume the Earth’s resources in a scenario known as “grey goo,” but such machines would have to be designed deliberately and we are highly uncertain of whether it would be possible to make them. #### What are possible interventions? A philanthropist could seek to influence research and development directions or support policy research. Potential goals could include achieving consensus regarding the feasibility of atomically precise manufacturing, identifying promising development strategies, and/or mitigating risks from possible military applications. We are highly uncertain about how to weigh the possible risks and benefits from accelerating progress toward APM and about the effectiveness of policy research in the absence of greater consensus regarding the feasibility of the technology. #### Who else is working on it? A few small non-profit organizations have explicitly focused on research, development, and policy analysis related to atomically precise manufacturing. Atomically precise manufacturing receives little explicit attention in academia, but potential enabling technologies such as DNA nanotechnology and scanning probe microscopy are active fields of research. ## 1. Background #### 1.1 Terminology There are a number of related, but distinct, concepts discussed in the context of atomically precise manufacturing, including:1 • Nanotechnology • Molecular nanotechnology • Atomically precise manufacturing (APM, which is roughly synonymous with ‘molecular manufacturing’) #### 1.1.1 Nanotechnology According to the definition set by the U.S. National Nanotechnology Initiative:2 Nanotechnology is the understanding and control of matter at the nanoscale, at dimensions between approximately 1 and 100 nanometers, where unique phenomena enable novel applications. Encompassing nanoscale science, engineering, and technology, nanotechnology involves imaging, measuring, modeling, and manipulating matter at this length scale. ‘Nanotechnology’ is used in a broad sense to include APM, but also many rather different products and R&D projects. Nanomaterials (such as carbon nanotubes), DNA origami, and scanning tunneling microscopes are all considered nanotechnology, but they are not considered atomically precise manufacturing (as defined below) because they do not allow for programmable manufacturing of macroscopic structures. #### 1.1.2 Molecular nanotechnology ‘Molecular nanotechnology’ is a concept associated with Dr. Drexler’s 1986 book, Engines of Creation. Our understanding of this concept is highly limited, though we understand that: • It involves using very small, mobile ‘assemblers’ to bond atoms into desired stable patterns, and that they could be used to “build almost anything that the laws of nature allow to exist.”3 • It has been explained with less technical detail than APM, and Dr. Drexler regards his analysis of it as more uncertain than his analysis of APM.4 On the first point, Dr. Drexler has received criticism from Richard Jones (a Professor of Physics at the University of Sheffield), Richard Smalley (a Nobel laureate in Chemistry), and the Royal Society.5 However, Dr. Drexler has suggested to us that these criticisms are based on misunderstandings of his work.6 For example, he has described the idea of “literally building ‘atom by atom’ ” as a “technically inaccurate popularization of the idea of atomically precise manufacturing.”7 In Engines of Creation Dr. Drexler noted that molecular nanotechnology “will not be able to build everything that could exist.” In conversation with us, Dr. Drexler said that atomically precise manufacturing—which is the focus of this investigation—”does not include the concept of a ‘universal assembler’ capable of making any possible object.”8 #### 1.1.3 Atomically precise manufacturing In a conversation with us, Dr. Drexler characterized APM as follows:9 At a more mature stage in the development of APM, Dr. Drexler envisions desktop-sized (or larger), programmable “nanofactories” which would use earth-abundant materials (e.g., molecules composed of elements from the upper-right hand corner of the periodic table, including carbon, oxygen, nitrogen, and silicon) as inputs/feedstocks, and use arrays of molecule-binding nanoscale devices (using gearboxes, motors, and so on to implement positioning/transport mechanisms) to guide the motion of reactive molecules from the feedstocks to assemble objects and machines defined by data files (as in 3D printers today). Such a nanofactory would be capable of placing its inputs in controlled configurations in controlled sequences, providing a degree of control of chemical synthesis that cannot be achieved by means that rely on the diffusion of molecules in solution. Drexler suggests that systems of this general kind could produce a superset of the range of products that can be made by modern industry. The physical principles, mechanisms, and potential system architectures of such nanofactories are examined in greater detail in Nanosystems. In Nanosystems, Dr. Drexler proposes the following applications of APM:10 • Programmable positioning of reactive molecules with ~0.1 nm precision • Mechanosynthesis at > 10^6 operations/device [per] second • Mechanosynthetic assembly of 1 kg objects in < 10^4 s • Nanomechanical systems operating at ~10^9 Hz • Logic gates that occupy ~10^-26 m^3 (~10^-8 μ^3) • Logic gates that switch in ~0.1 ns and dissipate < 10^-21 J • Computers that perform 10^16 instructions per second per watt • Cooling of cubic-centimeter, ~10^5 W systems at 300 K • Compact 10^15 MIPS parallel computing systems • Mechanochemical power conversion at > 10^9 W/m^3 • Electromechanical power conversion at > 10^15 W/m^3 • Macroscopic components with tensile strengths > 5 x 10^10 Pa • Production systems that can double capital stocks in < 10^4 s Of these capabilities, several are qualitatively novel, and others improve on present engineering practice by one or more orders of magnitude. Dr. Drexler also argued that these nanofactories could be used to quickly make additional nanofactories.11 This helps clarify the definition of APM above, though we do not fully understand the significance (or even the meaning) of many of these proposed applications. However, our understanding is that these capabilities include extremely precise manufacturing, very powerful computers, very stiff materials, and fast assembly of macroscopic objects from raw materials. Our investigation focused on APM rather than nanotechnology or molecular nanotechnology because: • We are aware of arguments that APM and molecular nanotechnology could pose global catastrophic risks (see “What is the problem?”), but are not aware of arguments that other forms of nanotechnology could pose global catastrophic risks. • According to Dr. Drexler, there is stronger evidence for the feasibility of APM than the feasibility of molecular nanotechnology (as noted above). #### 1.2 Potential development pathways for APM Progress toward APM may proceed along two different pathways: • ‘Soft’ approaches using biomolecular materials capable of organizing themselves into desired three-dimensional structures, such as DNA nanotechnology. DNA origami, in which DNA self-assembles in solution to form desired 3D molecular structures, is one example of DNA nanotechnology.12 • ‘Hard’ pathways such as ‘scanning probe microscopy,’ where microscopes are used to pick up individual atoms and put them in desired arrangements, one by one. For example, IBM researchers used scanning tunneling microscopes (a special type of scanning probe microscope) to spell out “IBM” on a two-dimensional surface with individual atoms, as shown here. There is disagreement about which path is more promising. People who think the soft pathway is more likely to yield progress include: • Eric Drexler, • Richard Jones, and • Adam Marblestone, scientific advisor to the Open Philanthropy Project and Director of Scientific Architecting at the MIT Synthetic Neurobiology Group. Philip Moriarty, a Professor of Physics at the University of Nottingham, was more enthusiastic about the hard route.13 The following provides additional detail on Dr. Drexler’s preferred development pathway for APM:14 Dr. Drexler has a concept for a self-assembling, biomolecular, nano-resolution 3D printer operating in solution. The active head of this device (analogous to a printhead) could be moved by linear stepper motors with displacement increments of about a nanometer, operating along three axes and controlled by external optical inputs. Such a device could also be accurate to a resolution of a nanometer (though the system’s components would not be stiff enough to enable accurate positioning at the small-molecule length scale, due to thermal fluctuations), and would construct objects out of biomolecular materials. One way the active head of a 3D printer could work is by removing protective groups from active sites on a surface, allowing the feedstock materials in the solution to bind to the selected sites, transported by Brownian motion. Dr. Drexler envisions using these soft nanomachines to create the more mature form of APM described above.15 We are highly uncertain about how promising these development pathways are, and have not closely investigated them. #### 1.3 Will it eventually be possible to develop APM? There is no scientific consensus on whether APM is feasible in principle, and significant skepticism has been expressed in some quarters. We have not carefully considered the object-level merits of the arguments on both sides of this issue—which we believe would require substantial additional work—and therefore we focus on the perspectives of the people we interviewed and the scientific sources we considered. The feasibility of atomically precise manufacturing has been reviewed in a report published by the US National Academy of Sciences (NAS). The NAS report was initiated in response to a Congressional request, and the result was included in the first triennial review of the U.S. National Nanotechnology Initiative.16 It discusses APM for 4 pages under the heading, “Technical Feasibility of Site-Specific Chemistry for Large-Scale Manufacturing.”17 While the committee states that “many scientists foresee a long-term future in which a variety of strategies, tools, and processes allow nearly any stable chemical structure to be built atom by atom or molecule by molecule from the bottom up,”18 the report was inconclusive regarding the technical feasibility of APM. It noted that Dr. Drexler’s work was hard to evaluate because its questions—about the in-principle feasibility of potential future technologies—are “currently outside the mainstream of both conventional science (designed to seek new knowledge) and conventional engineering (usually concerned with the design of things that can be built more or less immediately).”19 The report did not identify specific technical flaws with Dr. Drexler’s theoretical calculations. However, it did not regard these calculations as a reliable basis for predicting the potential capabilities of future manufacturing systems, stating that “the eventually attainable range of chemical reaction cycles, error rates, speed of operation, and thermodynamic efficiencies of such bottom-up manufacturing systems cannot be reliably predicted at this time.”20 Despite this uncertainty, the NAS report recommended research funding for experimental demonstrations that link to abstract models of APM and guide long-term vision related to APM.21 A Royal Society report was dismissive of the feasibility of ‘molecular manufacturing,’ stating that they had “seen no evidence of the possibility of such nanoscale machines in the peer-reviewed literature, or interest in their development from the mainstream scientific community or industry.”22 However, like the NAS report, this report focused primarily on other aspects of nanotechnology rather than APM. Only pages 28 and 109 discuss concepts related to APM, and those pages only cite critical correspondence between Eric Drexler and Richard Smalley, one paper co-authored by Chris Phoenix (Co-Founder and Director of Research at the Center for Responsible Nanotechnology) and Eric Drexler, and the opinion of George Whitesides (a Professor of Chemistry at Harvard University).23 Moreover, these pages seem to be focused on concepts that we and Drexler associate with molecular nanotechnology rather than molecular manufacturing/APM,24 so it is unclear whether these critiques carry over to APM. The people we interviewed generally found it plausible that some form of atomically precise manufacturing was feasible in principle.25 However, some of them also expressed skepticism about the feasibility of some aspects of APM. For example: • Prof. Moriarty suggested that molecular manufacturing would only be feasible with a limited range of materials,26 though we are uncertain about the extent to which this is a disagreement with Dr. Drexler, who only discusses a limited range of materials in the context of APM/molecular manufacturing (see Drexler’s definition of APM above). • Prof. Jones was skeptical of the feasibility of developing ‘hard’ nanosystems from ‘soft’ nanosystems.27 We have an incomplete sense of which aspects of APM (such as range of materials, range of possible structures, size of structures created, speed of production, and capacity for self-replication) the people we spoke with thought were realistic, and which they did not. Dr. Drexler’s most notable individual critic was Richard Smalley, who had an open, critical correspondence with Dr. Drexler in Chemical & Engineering News. We did not thoroughly review the correspondence between Dr. Drexler and Dr. Smalley, but no one we spoke with suggested to us that the correspondence was conclusive regarding the feasibility of APM.28 We are not aware of any specific, generally accepted, published scientific proof or refutation regarding the feasibility of APM. #### 1.4 When might APM be developed? The timeline for APM development is controversial. Eric Drexler and Chris Phoenix—who had the shortest development timelines among people we spoke with—suggested that, given substantial investment and agreement about development pathways, it might be possible to develop atomically precise manufacturing—of a kind that could pose substantial risks or significantly change society—within a decade.29 The other people we spoke with (Philip Moriarty, Richard Jones, and Adam Marblestone) hold that atomically precise manufacturing advanced enough to pose substantial risks or significantly change society is further in the future.30 This is consistent with the NAS report’s conclusion that development pathways for APM were unclear. Because APM is a multifaceted concept that lacks a precise definition, we are uncertain about the extent to which the people we spoke with disagree about when different aspects of APM will reach different levels of capability. Unless APM is developed in a secret “Manhattan Project”—and there is disagreement about how plausible that is31 —the people we spoke with believe it would be extremely unlikely for an observer closely watching the field to be surprised by a sudden increase in potentially dangerous APM capabilities.32 ## 2. What is the problem? #### 2.1 Is there insufficient work on, and progress toward, APM? According to Dr. Drexler, lack of consensus about feasibility and implementation pathways is stalling progress in development toward APM.33 At the same time, Prof. Jones argues that experimental work by nanoscientists has a direct bearing on Dr. Drexler’s proposals in Nanosystems,34 and that progress in the field has been slow primarily because of the inherent difficulty of the science (though he also acknowledges some institutional challenges to receiving funding for ambitious, uncertain research projects).35 We are highly uncertain about the extent to which progress toward APM is held back by resolvable uncertainty about feasibility and implementation pathways and the extent to which it would be desirable to accelerate progress toward APM (given the potential risks discussed below). #### 2.2 What are the potential risks from APM? #### 2.2.1 APM and weapons development and production If APM were developed, it would likely be substantially easier to create new weapons and quickly and inexpensively produce them at scale. Our understanding is that APM would make this possible because:36 • As discussed above, APM would allow for the manufacturing of a superset of the products of modern industry using abundant feedstocks. • Nanofactories could be used to produce additional nanofactories (using the same feedstocks). • APM might speed prototyping and product development because factories could immediately build parts on site, leading to a faster design/prototype/test cycle. We would guess some especially concerning military applications would include new types of drones and centrifuges for enriching uranium that would be much easier to produce.37 In addition to the direct use of the weapons above, some related risks include: • The possibility that the above capabilities could disrupt geopolitics, including deterrence relationships. For example, Chris Phoenix suggested that there could be an arms race related to this technology, or that one nation might want to forcibly prevent another from gaining advanced APM.38 • The possibility that an individual or small group could use nanofactories to cheaply mass-produce weapons, enabling terrorist organizations.39 #### 2.2.2 Grey goo ‘Grey goo’ is a proposed scenario in which tiny self-replicating machines outcompete organic life and rapidly consume the earth’s resources in order to make more copies of themselves.40 According to Dr. Drexler, a grey goo scenario could not happen by accident; it would require deliberate design.41 Both Drexler and Phoenix have argued that such runaway replicators are, in principle, a physical possibility, and Phoenix has even argued that it’s likely that someone will eventually try to make grey goo. However, they believe that other risks from APM are (i) more likely, and (ii) very likely to be relevant before risks from grey goo, and are therefore more worthy of attention.42 Similarly, Prof. Jones and Dr. Marblestone have argued that a ‘grey goo’ catastrophe is a distant, and perhaps unlikely, possibility.43 We are highly uncertain about: • The in-principle feasibility and difficulty of grey goo, • The extent to which APM would assist in creating grey goo, and • Whether, if it is feasible, anyone would intentionally develop grey goo. ## 3. What are the possible interventions? A philanthropist working in this area might: • Help develop an academic consensus regarding the feasibility of APM and possible development pathways. This would likely be accomplished by convening meetings, commissioning feasibility research, and communicating findings. Similar efforts have faced challenges in the past.44 • Support policy research related to the development of atomically precise manufacturing. Such research could consider the goals of developing atomically precise manufacturing in addition to risks. However, such research may have limited impact in the absence of greater consensus about the feasibility and timeline of atomically precise manufacturing, and might better be left until such consensus is established.45 Topics of such research could include arms control, economic impacts of APM, the impact of APM on AI development, and the impact of APM on surveillance technology.46 • Support research and development of atomically precise manufacturing. This could include attempts to steer research in particular directions or to grow the field.47 Prof. Jones estimated that an investment of about$150 million over ten years would significantly grow the field.48
• Monitor progress toward atomically precise manufacturing, potentially supporting R&D or policy research as advanced capabilities become nearer.49

Dr. Drexler is not aware of any technical research agenda for this field—e.g. analogous to the technical research agendas for reducing possible risks from artificial intelligence that have been proposed by the Future of Life Institute or the Machine Intelligence Research Institute—that might help reduce the potential risks associated with APM. With respect to the risk of unauthorized use of nanofactories to manufacture weapons, he suggests that nanofactories could be designed so that they are only capable of making a limited range of products that does not include weapons.50

As stated above, we are uncertain about the desirability of faster progress toward APM. Before pursuing interventions that pushed forward its development, we think it would be important to weigh the possible risks and benefits of doing so.

## 4. Who else is working on this?

A few small non-profit organizations are explicitly focused on influencing and/or promoting the development of atomically precise manufacturing and/or molecular nanotechnology. Such organizations include:51

ORGANIZATION 2013 REVENUE 2013 ASSETS
The Foresight Institute $814,135$945,461
Center for Responsible Nanotechnology Not available Not available
Institute for Molecular Manufacturing $2,194$14,028

We have a very limited understanding of the activities of these organizations because our investigation of the field so far has been brief.

The 2015 US Federal Budget provides more than \$1.5 billion for the National Nanotechnology Initiative, a U.S. Government R&D initiative promoting and coordinating the development of nanotechnology.52 However, there is currently no focused R&D effort towards atomically precise manufacturing, though there is some relevant research toward a variety of shorter-term goals in applied and fundamental science.53 Although some academics work on ethical and legal issues associated with nanotechnology (e.g., the Center for Nanotechnology in Society at ASU), we have been told that little of this work is related to atomically precise manufacturing (as opposed to nanomaterials).54

Although atomically precise manufacturing currently receives little attention, it does not yet pose a significant risk. It is hard to know how much attention it will receive if/when the technology becomes more mature.

## 5. Questions for further investigation

Our investigation in this area left us with many open questions which could be addressed in further research, including:

1. Do academic organizations studying social issues related to nanotechnology (such as the ASU Center for Nanotechnology and Society) do work that is relevant to atomically precise manufacturing?
2. How would promoting progress in atomically precise manufacturing affect the potential risks posed by the technology and/or related technologies?
3. How effectively would convening meetings and commissioning research build consensus regarding the feasibility of atomically precise manufacturing and/or build support for specific development pathways?
4. Are there possible interventions that could be useful today in directly reducing risks, as opposed to simply improving the pace of progress toward APM, or our ability to forecast such progress?
5. How confident can we be that there will be substantial lead time between early signs that APM is feasible and the deployment of APM?
6. Are there promising proposals for advancing development toward atomically precise manufacturing? If so, how soon could atomically precise manufacturing be developed?
7. Are there any technologies we would be able to differentially accelerate to offset potential risks of APM? (For example, there may be some inherently defensive technology which could neutralize weapons produced by nanofactories.)
8. To what extent do critiques of the feasibility of molecular nanotechnology carry over to APM?
9. How costly would it be to develop APM?
10. Where would APM provide the largest additional benefits in comparison with other technologies currently in existence and under development?

## 6. Our process

We decided to look into this topic because:

• Atomically precise manufacturing is discussed as a possible global catastrophic risk in some comprehensive treatments of global catastrophic risks.55
• Our impression was—and continues to be—that atomically precise manufacturing receives little attention from government or philanthropy.

Our investigation to date has mainly consisted of conversations with five individuals with knowledge about atomically precise manufacturing and/or its potential risks:

• Eric Drexler – Academic Visitor, Oxford Martin Programme on the Impacts of Future Technology, University of Oxford
• Richard Jones – Pro-Vice-Chancellor for Research and Innovation, Professor of Physics, University of Sheffield
• Adam Marblestone – Director of Scientific Architecting, Massachusetts Institute of Technology Synthetic Neurobiology Group (scientific advisor to the Open Philanthropy Project)
• Philip Moriarty – Professor of Physics, University of Nottingham
• Chris Phoenix – Co-Founder and Director of Research, Center for Responsible Nanotechnology

We also had informal conversations with Eric Drexler, reviewed documents he provided, and listened to three audiobooks related to atomically precise manufacturing:

• Nano by Ed Regis
• Radical Abundance by Eric Drexler
• The Visioneers by Patrick McKray

The research on this page focuses substantially on Dr. Drexler’s perspective because work in this field has been very limited,56 and our understanding is that he has been responsible for a substantial fraction of the work on atomically precise manufacturing.

Relationship disclosure: This page was prepared by Nick Beckstead, who previously worked with Dr. Drexler at the Future of Humanity Institute at Oxford University.

## 7. Sources

DOCUMENT SOURCE
Chris Phoenix and Mike Treder 2008, Nanotechnology as global catastrophic risk Source (archive)
Drexler 1986, Engines of CreationEngines of Creation Source (archive)
Drexler 1992, NanosystemsNanosystems Source (archive)
Drexler-Smalley Debate Source (archive)
FLI survey of research questions, 2015 Source (archive)
Foresight Institute’s 990 for 2013 Source (archive)
GiveWell’s non-verbatim summary of a conversation with Adam Marblestone, August 27, 2014 Source
GiveWell’s non-verbatim summary of a conversation with Chris Phoenix, August 20, 2014 Source
GiveWell’s non-verbatim summary of a conversation with Eric Drexler, January 23, 2015 Source
GiveWell’s non-verbatim summary of a conversation with Eric Drexler, October 8, 2014 Source
GiveWell’s non-verbatim summary of a conversation with Philip Moriarty, September 3, 2014 Source
GiveWell’s non-verbatim summary of a conversation with Richard Jones, September 30, 2014 Source
“IBM” atoms Source
Institute for Molecular Manufacturing’s 990 for 2013 Source (archive)
Jones 2004, Soft MachinesSoft Machines Source (archive)
Jones 2004, Did Smalley deliver a killer blow to Drexlerian MNT? Source (archive)
Jones 2004, Molecular nanotechnology, Drexler and Nanosystems – where I standNanosystems – where I stand Source (archive)
Jones 2005, The mechanosynthesis debate Source (archive)
Jones 2007, Nanotechnology and visions of the future (part 1) Source (archive)
Jones 2008, Rupturing the Rapture Source (archive)
MIRI Research Agenda, 2015 Source (archive)
National Academy of Sciences 2006, A Matter of Size: Triennial Review of the National Nanotechnology InitiativeA Matter of Size: Triennial Review of the National Nanotechnology Initiative Source (archive)
National Nanotechnology Initiative Website Source (archive)
NNI Website, What It Is and How It Works Source (archive)
Nick Beckstead’s non-verbatim summary of a conversation with Miles Brundage, April 4, 2014 Source (archive)
Sander Olson interview with Philip Moriarty 2011 Source (archive)
The Royal Society and the Royal Academy of Engineering 2004 Source