In October 2016, we wrote:
we are contracting [a developer] to build a simple online application for credence calibration training: training the user to accurately determine how confident they should be in an opinion, and to express this confidence in a consistent and quantified way.
That online application is now available:
Play “Calibrate Your Judgment”
Note that you must sign in with a GuidedTrack, Facebook, or Google account, so that the application can track your performance over time.
We expect many users will find this program to be the most useful free online calibration training currently available.
That said, we think there are several ways in which a calibration app could be more engaging and useful than ours, if someone were to invest substantially more development effort than we did. Some reflections on the challenges we encountered, and some lessons we learned, are available in this Google doc.
Spencer Greenberg, the lead developer of Calibrate Your Judgment, has released a paper describing the scoring rules used in the app, here.
Update April 23: In response to feedback, we have now improved the set of questions used for the app’s confidence intervals module, by removing hundreds of ill-formed or confusingly worded questions. We hope this leads to a better and more useful experience for users.