• Partner With Us
  • Focus Areas
    • Cause Selection
    • Global Health & Wellbeing
      • Abundance & Growth
      • Effective Giving & Careers
      • Farm Animal Welfare
      • Global Aid Policy
      • Global Health & Development
      • Global Health R&D
      • Global Public Health Policy
      • Scientific Research
    • Global Catastrophic Risks
      • Biosecurity & Pandemic Preparedness
      • Forecasting
      • Global Catastrophic Risks Capacity Building
      • Potential Risks from Advanced AI
    • Other Areas
      • History of Philanthropy
  • Grants
  • Research & Updates
    • Blog Posts
    • In the News
    • Research Reports
    • Notable Lessons
  • About Us
    • Grantmaking Process
    • How to Apply for Funding
    • Careers
    • Team
    • Operating Values
    • Stay Updated
    • Contact Us
  • Partner With Us
  • Focus Areas
    • Cause Selection
    • Global Health & Wellbeing
      • Abundance & Growth
      • Effective Giving & Careers
      • Farm Animal Welfare
      • Global Aid Policy
      • Global Health & Development
      • Global Health R&D
      • Global Public Health Policy
      • Scientific Research
    • Global Catastrophic Risks
      • Biosecurity & Pandemic Preparedness
      • Forecasting
      • Global Catastrophic Risks Capacity Building
      • Potential Risks from Advanced AI
    • Other Areas
      • History of Philanthropy
  • Grants
  • Research & Updates
    • Blog Posts
    • In the News
    • Research Reports
    • Notable Lessons
  • About Us
    • Grantmaking Process
    • How to Apply for Funding
    • Careers
    • Team
    • Operating Values
    • Stay Updated
    • Contact Us

Meta-Research Innovation Center at Stanford (METRICS)

  • Content Type: Blog Posts
Published: April 23, 2014 | by Holden Karnofsky

Notes on this post:

  • Stuart Buck of the Laura and John Arnold Foundation, as well as Prof. Steven Goodman of METRICS, reviewed a draft of this post.
  • Note: The Open Philanthropy Project was formerly known as GiveWell Labs. Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

We’re very excited for the launch of Meta-Research Innovation Center at Stanford (METRICS), co-founded by John Ioannidis and Steven Goodman and supported by a grant from the Laura and John Arnold Foundation (LJAF). METRICS will bring together researchers to study the state of medical research quality, including questions such as how concerned we should be about publication bias (which the founders have published helpful papers on in the past), and to advocate for potential solutions. (Also see coverage in The Economist.)

Our work on GiveWell Labs was responsible for initially connecting the METRICS founders to LJAF, which is providing a commitment of up to ~$6 million to help METRICS through its initial years, during which time METRICS will be seeking more sources of support. We find it worthwhile to lay out the events that led to this connection, partly because they indicate some degree of impact on our part (though not of our usual kind) and partly because they make for an interesting case study in how to source giving opportunities.

Our role in connecting METRICS and LJAF

  • In 2012, we investigated the US Cochrane Center, in line with the then-high priority we placed on meta-research. As part of our investigation, we were connected – through our network – to Dr. Steven Goodman, who discussed Cochrane with us (notes). We also asked him about other underfunded areas in medical research, and among others he mentioned the idea of “assessing or improving the quality of evidence in the medical literature” and the idea of establishing a center for such work.
  • During our follow-up email exchange with Dr. Goodman, we mentioned that we were thinking of meta-research in general as a high priority, and sent him a link to our standing thoughts on the subject. He responded, “I didn’t know of your specific interest in meta-research and open science … Further developing the science and policy responses to challenges to the integrity of the medical literature is also the raison d’etre of the center I cursorily outlined, which is hard to describe to folks who don’t really know the area; I didn’t realize how far down that road you were.” He mentioned that he could send along a short proposal, and we said we’d like to see it.
  • At the same time, we were in informal conversations with Stuart Buck at the Laura and John Arnold Foundation (LJAF) about the general topic of meta-research. LJAF had expressed enthusiasm over the idea of the Center for Open Science (which it now supports), and generally seemed interested in the topic of reproducibility in the social sciences. I asked Stuart whether he would be interested in doing some funding related to the field of medicine as well, and in reviewing a proposal in that area that I thought looked quite promising. He said yes, and (after checking with Dr. Goodman) I sent along the proposal.
  • From that point on, LJAF followed its own process, though we stayed posted on the progress, reviewed a more fleshed-out proposal, shared our informal thoughts with LJAF, and joined (by videoconference) a meeting between LJAF staff and Drs. Ioannidis and Goodman.
  • Following the meeting with Drs. Ioannidis and Goodman, we told Stuart that we would consider providing a modest amount of co-funding (~10%) for the initial needs of METRICS. He said this wouldn’t be necessary as LJAF planned to provide the initial funding.

 

Takeaways

Case study in “active funding.” We’ve written before about the evolution of our thinking on active vs. passive funding. Specifically, while we used to imagine reviewing a large number of project proposals and recommending the best for funding, we now take the approach of choosing “causes” – broad areas/problems of interest, even when it isn’t obvious what the interventions or grantees would be – and looking for giving opportunities only after we’ve identified causes to focus on. Part of the reasoning behind this approach is that project proposals often emerge in response to funder interests, so it’s may not be possible to find all the possible projects to fund in an area until one starts actively expressing interest in the area.

The above case is a good example of this. The conversations we had with LJAF and with Dr. Goodman sprang out of a pre-declared interest in the cause of meta-research, and in Dr. Goodman’s case it seems he would never have sent the proposal if not for our clearly articulated interest in the subject.

Our impact. GiveWell never issued an official recommendation in favor of funding METRICS. While we shared informal opinions with LJAF, LJAF followed its own evaluation process and decided on its own to provide funding. So we don’t think the LJAF grant ought to be considered as GiveWell money moved in the usual sense. In addition, we don’t know what the counterfactual impact of our role was; it’s possible that LJAF would eventually have found and funded METRICS. With that said, it seems to us that by actively prioritizing the area of meta-research and following the lines of investigation we did, we raised the speed and probability of this project’s moving forward. On review of this post, Stuart stated “I’m certain that I wouldn’t have come across [the] initial proposal in any other way.”

Note that this post initially included a discussion of another project that we thought had had a similar dynamic. We introduced Elizabeth Iorns, founder of the Reproducibility Initiative, to Stuart Buck of LJAF in April 2013 (after a February 2013 conversation with Dr. Iorns as part of our continuing meta-research investigation); LJAF has since provided support for the Reproducibility Initiative through the Center for Open Science. However, Stuart corrected us on this point, pointing out that he was introduced to Elizabeth independently about a month after our intro; we now believe we played little to no role (relative to the counterfactual) in that connection.

Where we stand on meta-research

In mid-2013, we gave two updates on our thinking regarding meta-research: a landscape of the “open science” community and a more general update listing several things that “meta-research” could mean. We noted that we were planning to pause work on this front until we had examined some other causes.

At this time, our take is that:

  • There is more philanthropic interest in promoting reproducibility and open science (which had initially been the focus of our investigations) than we had initially guessed. (Note that LJAF became interested in these areas around the time that we did.)
  • Given the players that are already present, and having explored some other areas, our sense is that there are higher-leverage causes for us to enter.
  • We are trying to explore the broader concept behind “meta-research” – thinking about how to improve the incentives that academics face, both as they relate to reproducibility and as they relate to other matters – as part of our scientific research investigations.
  • We are extremely happy to see the launch of METRICS, which we believed to be probably the single most exciting giving opportunity we saw while exploring meta-research (and one we might have considered for a recommendation if not for LJAF’s interest).
Subscribe to new blog alerts
Open Philanthropy
Open Philanthropy
  • We’re Hiring!
  • Press Kit
  • Governance
  • Privacy Policy
  • Stay Updated
Mailing Address
Open Philanthropy
182 Howard Street #225
San Francisco, CA 94105
Email
info@openphilanthropy.org
Media Inquiries
media@openphilanthropy.org
Anonymous Feedback
Feedback Form

© Open Philanthropy 2025 Except where otherwise noted, this work is licensed under a Creative Commons Attribution-Noncommercial 4.0 International License.

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT