Basics

What is Metaculus?

Metaculus is an online forecasting platform and aggregation engine that brings together a global reasoning community and keeps score for thousands of forecasters, delivering machine learning-optimized aggregate forecasts on topics of global importance. The Metaculus forecasting community is often inspired by altruistic causes, and Metaculus has a long history of partnering with nonprofit organizations, university researchers and companies to increase the positive impact of its forecasts.

Metaculus therefore poses questions about the occurrence of a variety of future events, on many timescales, to a community of participating forecasters — you!

The name "Metaculus" comes from the Metaculus genus in the Eriophyidae family, a genus of herbivorous mites found in many locations around the world.

What is forecasting?

Forecasting is a systematic practice of attempting to answer questions about future events. On Metaculus, we follow a few principles to elevate forecasting above simple guesswork:

First, questions are carefully specified so that everyone understands beforehand and afterward which kinds of outcomes are included in the resolution, and which are not. Forecasters then give precise probabilities that measure their uncertainty about the outcome.

Second, Metaculus aggregates the forecasts into a community prediction based on the median of user forecasts weighted by recency. Surprisingly, the Community Prediction is often better than any individual predictor! This principle is known as the wisdom of the crowd, and has been demonstrated on Metaculus and by other researchers. Intuitively it makes sense, as each individual has separate information and biases which in general balance each other out (provided the whole group is not biased in the same way).

Third, we measure the relative skill of each forecaster, using their quantified forecasts. When we know the outcome of the question, the question is "resolved", and forecasters receive their scores. By tracking these scores from many forecasts on different topics over a long period of time, they become an increasingly better metric of how good a given forecaster is. These scores provide aspiring forecasters with important feedback on how they did and where they can improve.

When is forecasting valuable?

Forecasting is uniquely valuable primarily in complex, multi-variable problems or in situations where a lack of data makes it difficult to predict using explicit or exact models.

In these and other scenarios, aggregated predictions of strong forecasters offer one of the best ways of predicting future events. In fact, work by the political scientist Philip Tetlock demonstrated that aggregated predictions were able to outperform professional intelligence analysts with access to classified information when forecasting various geopolitical outcomes.

Why should I be a forecaster?

Research has shown that great forecasters come from various backgrounds—and oftentimes from fields that have nothing to do with predicting the future. Like many mental capabilities, prediction is a talent that persists over time and is a skill that can be developed. Steady quantitative feedback and regular practice can greatly improve a forecaster's accuracy.

Some events — such as eclipse timing and well-polled elections, can often be predicted with high resolution, e.g. 99.9% likely or 3% likely. Others — such as the flip of a coin or a close horse-race — cannot be accurately predicted; but their odds still can be. Metaculus aims at both: to provide a central generation and aggregation point for predictions. With these in hand, we believe that individuals, groups, corporations, governments, and humanity as a whole will make better decisions.

As well as being worthwhile, Metaculus aims to be interesting and fun, while allowing participants to hone their prediction prowess and amass a track-record to prove it.

Who created Metaculus?

Metaculus originated with two researcher scientists, Anthony Aguirre and Greg Laughlin. Aguirre, a physicist, is a co-founder of The Foundational Questions Institute, which catalyzes breakthrough research in fundamental physics, and of The Future of Life Institute, which aims to increase the benefit and safety of disruptive technologies like AI. Laughlin, an astrophysicist, is an expert at predictions from the millisecond predictions relevant to high-frequency trading to the ultra-long-term stability of the solar system.

What Are Metaculus Tournaments and Question Series?

Tournaments

Metaculus tournaments are organized around a central topic or theme. Tournaments are often collaborations between Metaculus and a nonprofit, government agency, or other organization seeking to use forecasting to support effective decision making. You can find current and archived tournaments in our Tournaments page.

Tournaments are the perfect place to prove your forecasting skills, while helping to improve our collective decision making ability. Cash prizes and Medals are awarded to the most accurate forecasters, and sometimes for other valuable contributions (like comments). Follow a Tournament (with the Follow button) to never miss new questions.

After at least one question has resolved, a Leaderboard will appear on the tournament page displaying current scores and rankings. A personal score board ("My Score") will also appear, detailing your performance for each question (see How are Tournaments Scored?).

At the end of a tournament, the prize pool is divided among forecasters according to their forecasting performance. The more you forecasted and the more accurate your forecasts were, the greater proportion of the prize pool you receive.

Can I donate my tournament winnings?

If you have outstanding tournament winnings, Metaculus is happy to facilitate donations to various non-profits, regranting organizations, and funds. You can find the list of organizations we facilitate payments to here.

Question Series

Like Tournaments, Question Series are organized around a central topic or theme. Unlike tournaments, they do not have a prize pool.

Question Series still show leaderboards, for interest and fun. However they do not award medals.

You can find all Question Series in a special section of the Tournaments page.

Is Metaculus a prediction market?

Metaculus has some similarities to a prediction market, but ultimately is not one. Metaculus aims to aggregate many people's information, expertise, and predictive power into high-quality forecasts. However, prediction markets generally operate using real or virtual currency, where participants buy (or sell) shares if they think that the standing prices reflect too low (or high) a probability of an event occurring. Metaculus, in contrast, directly solicits predicted probabilities from its users, then aggregates those probabilities. We believe that this sort of "prediction aggregator" has both advantages and disadvantages relative to a prediction market.

Advantages of Metaculus over prediction markets

Metaculus has several advantages over prediction markets, described below, but we want to preface this by saying that despite the potential issues with prediction markets that we describe here, we think prediction markets are valuable, are glad they exist, and would be glad to see more use of them.

1. Poor incentives for longer term forecasts. It's usually not a good use of your funds to lock them up in a prediction market for the long term, since you can usually get much better returns by investing, which means longer term markets are likely to have low liquidity. For an example see this plot from a Works in Progress article showing the trading volume on Betfair for the 2020 US presidential election. There was very little volume far in advance of the election, with most of the trading volume occurring only a month out from the election.
2. Problems with low probabilities. Prediction markets have market frictions that make them less useful for low probabilities. The return on using your money to bring a probability from 2% to 1% is negligible, or potentially negative if the prediction market extract a fee from traders. That's why you get weird results like Michelle Obama at 6% chance of becoming the Democratic nominee for the 2024 US presidential election in June of 2024, as was the case on Polymarket.
3. The focus isn't always forecasting. Prediction market incentives aren't always aligned with making the most accurate predictions. Consider that one potential use for prediction markets is hedging against risky outcomes. Additionally, people who are irrational but willing to put a ton of money behind their beliefs may skew the outcome. Sure, ideally a liquid market will correct for these skews, but it's possible that they could have an effect on the price. See this piece in Asterisk Magazine for more on "dumb money" in prediction markets.
4. What do individuals think will happen? Participants in prediction markets are expressing whether they think the probability is higher or lower than the market price, not making a forecast. If someone thinks the market is too low at 35% and bets accordingly, you don't know whether they think the true probability is 40% or 80%. This doesn't really impact the usefulness of the aggregate, but it does make the data less rich and informative, and harder to see the full distribution of forecasts like you can with the histograms for binary questions on Metaculus.
5. Individual market performance is not always a clear indication of forecasting skill. Excellent individual market performance might just signal proficiency at operating in markets, or ability to take advantage of bad bets made by others. For example, see this post about a tournament organized on Manifold where traders took a large early lead just due to intelligent use of limit orders. Since Metaculus elicits individual probabilities from every forecaster, we can better assess and recruit excellent forecasters.
6. Metaculus performs comparably to markets without the need to manage a portfolio. There are only a handful of apples-to-apples comparisons between platforms, but these find an advantage for Metaculus over prediction markets. Note that sample sizes tend to be small. However, there is also an indirect comparison (indirect because it does not consider the same questions across platforms) which found that prediction markets are more calibrated.

Are Metaculus Questions Polls?

No. Opinion polling can be a useful way to gauge the sentiment and changes in a group or culture, but there is often no single "right answer", as in a Gallup poll "How worried are you about the environment?"

In contrast, Metaculus questions are designed to be objectively resolvable (like in Will Brent Crude Oil top $140/barrel before May 2022?), and forecasters are not asked for their preferences, but for their predictions. Unlike in a poll, over many predictions, participants accrue a track record indicating their forecasting accuracy. These track records are incorporated into the Metaculus Prediction. The accuracy of the Metaculus track record itself is tracked here.

Metaculus Questions

What sorts of questions are allowed, and what makes a good question?

Questions should focus on tangible, objective facts about the world which are well-defined and not a matter of opinion. "When will the United States collapse?" is a poor, ambiguous question; What will be the US' score in the Freedom in the World Report for 2050? is more clear and definite. They generally take the form Will (event) X happen by (date) Y? or When will (event) X occur? or What will the value or quantity of X be by (date) Y?

A good question will be unambiguously resolvable. A community reading the question terms should be able to agree, before and after the event has occurred, whether the outcome satisfies the question's terms.

Questions should also follow some obvious rules:

1. Questions should respect privacy and not address the personal lives of non-public figures.
2. Questions should not be directly potentially defamatory or generally in bad taste.
3. Questions should never aim to predict mortality of individual people or even small groups. In cases of public interest (such as court appointees and political figures), the question should be phrased in other more directly relevant terms such as "when will X no longer serve on the court" or "will Y be unable to run for office on date X". When the topic is death (or longevity) itself questions should treat people in aggregate or hypothetically.
4. More generally, questions should avoid being written in a way that incentivizes illegal or harmful acts — that is, hypothetically, if someone were motivated enough by a Metaculus Question to influence the real world and change the outcome of a question's resolution, those actions should not be inherently illegal or harmful.

Who creates the questions, and who decides which get posted?

Many questions are launched by Metaculus staff, but any logged-in user can propose a question. Proposed questions will be reviewed by a group of moderators appointed by Metaculus. Moderators will select the best questions submitted, and help to edit the question to be clear, well-sourced, and aligned with our writing style.

Metaculus hosts questions on many topics, but our primary focus areas are Science, Technology, Effective Altruism, Artificial Intelligence, Health, and Geopolitics.

Who can edit questions?

• Admins can edit all questions at any time (however, once predictions have begun, great care is taken not to change a question's resolution terms unless necessary).
• Moderators can edit questions when they are Pending and Upcoming (before predictions have begun).
• Authors can edit their questions when they are Drafts and Pending.
• Authors can invite other users to edit questions that are in Draft or Pending.

How can I get my own question posted?

1. If you have a basic idea for a question but don't have time/energy to work out the details, you're welcome to submit it, discuss it in our question idea thread, or on our Discord channel.
2. If you have a pretty fully-formed question, with at least a couple of linked references and fairly careful unambiguous resolution criteria, it's likely that your question will be reviewed and launched quickly.
3. Metaculus hosts questions on many topics, but our primary focus areas are Science, Technology, Effective Altruism, Artificial Intelligence, Health, and Geopolitics. Questions on other topics, especially that require a lot of moderator effort to get launched, will be given lower priority and may be deferred until a later time.
4. We regard submitted questions as suggestions and take a free hand in editing them. If you're worried about having your name on a question that is altered from what you submit, or would like to see the question before it's launched, please note this in the question itself; questions are hidden from public view until they are given "upcoming" status, and can be posted anonymously upon request.

What can I do if a question I submitted has been pending for a long time?

We currently receive a large volume of question submissions, many of which are interesting and well-written. That said, we try to approve just enough questions that they each can get the attention they deserve from our forecasters. Metaculus prioritizes questions on Science, Technology, Effective Altruism, Artificial Intelligence, Health, and Geopolitics. If your question falls into one of these categories, or is otherwise very urgent or important, you can tag us with @moderators to get our attention.

What can I do if a question should be resolved but isn't?

If a question is still waiting for resolution, check to make sure there hasn't been a comment from staff explaining the reason for the delay. If there hasn't, you can tag @admins to alert the Metaculus team. Please do not use the @admins tag more than once per week regarding a single question or resolution.

What is a private question?

Private questions are questions that are not visible to the broader community. They aren't subject to the normal review process, so you can create one and predict on it right away. You can resolve your own private questions at any time, but points for private predictions won't be added to your overall Metaculus score and they won't affect your ranking on the leaderboard.

You can use private questions for anything you want. Use them as practice to calibrate your predictions before playing for points, create a question series on a niche topic, or pose personal questions that only you can resolve. You can even invite up to 19 other users to view and predict on your own questions!

To invite other forecasters to your private question, click the '...' more options menu and select 'Share Private Question'.

What are the rules and guidelines for comments and discussions?

We have a full set of community etiquette guidelines but in summary:

• Users are welcome comment on any question.
• Comments and questions can use markdown formatting
• Metaculus aims at a high level of discourse. Comments should be on topic, relevant and interesting. Comments should not only state the author's opinion (with the exception of quantified predictions). Comments which are spammy, aggressive, profane, offensive, derogatory, or harassing are not tolerated, as well as those that are explicitly commercial advertising or those that are in some way unlawful. See the Metaculus terms of use for more
• You can ping other users using "@username", which will send that user a notification (if they set that option in their notification settings).
• You are invited to upvote comments which contain relevant information to the question, and you can report comments that fail to uphold our etiquette guidelines.
• If a comment is spam, inappropriate/offensive, or flagrantly breaks our rules, please send us a report (under the "..."menu).

What do "credible source" and "before [date X]" and such phrases mean exactly?

To reduce ambiguity in an efficient way, here are some definitions that can be used in questions, with a meaning set by this FAQ:

1. A "credible source" will be taken to be an online or in-print published story from a journalistic source, or information publicly posted on a the website of an organization by that organization making public information pertaining to that organization, or in another source where the preponderance of evidence suggests that the information is correct and that there is no significant controversy surrounding the information or its correctness. It will generally not include unsourced information found in blogs, facebook or twitter postings, or websites of individuals.
2. The phrase "Before [date X] will be taken to mean prior to the first moment at which [date X] would apply, in UTC. For example, "Before 2010" will be taken to mean prior to midnight January 1, 2010; "Before June 30" would mean prior to midnight (00:00:00) UTC June 30.
   • Note: Previously this section used "by [date x]" instead of "before [date x]", however "before" is much clearer and should always be used instead of "by", where feasible.

What types of questions are there?

Binary Questions

Binary questions can resolve as either Yes or No (unless the resolution criteria were underspecified or otherwise circumvented, in which case they can resolve as Ambiguous). Binary questions are appropriate when an event can either occur or not occur. For example, the question "Will the US unemployment rate stay above 5% through November 2021?" resolved as No because the unemployment rate dropped below 5% before the specified time.

Range Questions

Range questions resolve to a certain value, and forecasters can specify a probability distribution to estimate the likelihood of each value occurring. Range questions can have open or closed bounds. If the bounds are closed, probability can only be assigned to values that fall within the bounds. If one or more of the bounds are open, forecasters may assign probability outside the boundary, and the question may resolve as outside the boundary. See here for more details about boundaries on range questions.

The range interface allows you to input multiple probability distributions with different weights. See here for more details on using the interface.

There are two types of range questions, numeric range questions and date range questions.

Numeric Range

Numeric range questions can resolve as a numeric value. For example, the question "What will be the 4-week average of initial jobless claims (in thousands) filed in July 2021?" resolved as 395, because the underlying source reported 395 thousand initial jobless claims for July 2021.

Questions can also resolve outside the numeric range. For example, the question "What will the highest level of annualised core US CPI growth be, in 2021, according to U.S. Bureau of Labor Statistics data?" resolved as > 6.5 because the underlying source reported more than 6.5% annualized core CPI growth in the US, and 6.5 was the upper bound.

Date Range

Date range questions can resolve as a certain date. For example, the question "When will the next Public Health Emergency of International Concern be declared by the WHO?" resolved as July 23, 2022, because a Public Health Emergency of International Concern was declared on that date.

Questions can also resolve outside the date range. For example, the question "When will a SpaceX Super Heavy Booster fly?" resolved as > March 29, 2022 because a SpaceX Super Heavy booster was not launched before March 29, 2022, which was the upper bound.

What are question groups?

Question groups are sets of closely related questions or question outcomes all collected on a single page. Forecasters can predict quickly and efficiently on these interconnected outcomes, confident that they are keeping all of their predictions internally consistent.

How do question groups facilitate more efficient, more accurate forecasting?

With question groups, it's easy to forecast progressively wider distributions the further into the future you predict to reflect increasing uncertainty. A question group collecting multiple binary questions on a limited set of outcomes or on mutually exclusive outcomes makes it easier to see which forecasts are in tension with each other.

What happens to the existing question pages when they are combined in a question group?

When regular forecast questions are converted into "subquestions" of a question group, the original pages are replaced by a single question group page. Comments that previously lived on the individual question pages are moved to the comment section of the newly created group page with a note indicating the move.

Do I need to forecast on every outcome / subquestion of a question group?

No. Question groups comprise multiple independent subquestions. For that reason, there is no requirement that you forecast on every outcome within a group.

How are question groups scored?

Each outcome or subquestion is scored in the same manner as a normal independent question.

Why don't question group outcome probabilities sum to 100%?

Even if there can only be one outcome for a particular question group, the Community Prediction functions as it would for normal independent questions. The Community Prediction will still display a median or a weighted aggregate of the forecasts on each subquestion, respectively. These medians and weighted aggregates are not constrained to sum to 100%

Feedback for question groups can be provided on the question group discussion post.

What are Conditional Pairs?

A Conditional Pair is a special type of Question Group that elicits conditional probabilities. Each Conditional Pair sits between a Parent Question and a Child Question. Both Parent and Child must be existing Metaculus Binary Questions.

Conditional Pairs ask two Conditional Questions (or "Conditionals" for short), each corresponding to a possible outcome of the Parent:

1. If the Parent resolves Yes, how will the Child resolve?
2. If the Parent resolves No, how will the Child resolve?

The first Conditional assumes that "The Parent resolves Yes" (or "if Yes" for short). The second conditional does the same for No.

Conditional probabilities are probabilities, so forecasting is very similar to Binary Questions. The main difference is that we present both conditionals next to each other for convenience:

Conditional questions are automatically resolved when their Parent and Child resolve:

• When the Parent resolves Yes, the "if No" Conditional is Annulled. (And vice versa.)
• When the Child resolves, the Conditional that was not annulled resolves to the same value.

Let's work through an example:

• The Parent is "Will it rain today?".
• The Child is "Will it rain tomorrow?".

So the two Conditionals in the Conditional Pair will be:

1. "If it rains today, will it rain tomorrow?"
2. "If it does not rain today, will it rain tomorrow?"

For simplicity, Metaculus presents conditional questions graphically. In the forecasting interface they are in a table:

And in the feeds, each possible outcome of the Parent is an arrow, and each conditional probability is a bar:

Back to the example:

It rains today. The parent resolves Yes. This triggers the second conditional ("if No") to be annulled. It is not scored.

You wait a day. This time it doesn't rain. The Child resolves No. This triggers the remaining Conditional ("if Yes") to resolve No. It is scored like a normal Binary Question.

How do I create conditional pairs?

You can create and submit conditional pairs like any other question type. On the 'Create a Question' page, select Question Type 'conditional pair' and select Parent and Child questions.

Note: You can use question group subquestions as the Parent or Child by clicking the Parent or Child button and then either searching for the subquestion in the field or pasting the URL for the subquestion.

To copy the URL for a subquestion, simply visit a question group page and click the '...' more options menu to reveal the Copy Link option.

How do I find certain questions on Metaculus?

Questions on Metaculus are sorted by activity by default. Newer questions, questions with new comments, recently upvoted questions, and questions with many new predictions will appear at the top of the Metaculus homepage. However, there are several additional ways to find questions of interest and customize the way you interact with Metaculus.

Search Bar

The search bar can be used to find questions using keywords and semantic matches. At this time it cannot search comments or users.

Filters

Questions can be sorted and filtered in a different manner from the default using the filters menu. Questions can be filtered by type, status and participation. Questions can also be ordered, for example by "Newest". Note that the options available change when different filters are selected. For example, if you filter by "Closed" questions you will then be shown an option to order by "Soonest Resolving".

Question Resolution

What are the "open date", "close date" and "resolve date?"

When submitting a question, you are asked to specify the closing date (when the question is no longer available for predicting) and resolution date (when the resolution is expected to occur). The date the question is set live for others to forecast on is known as the open date.

• The open date is the date/time when the question is open for predictions. Prior to this time, if the question is active, it will have "upcoming" status, and is potentially subject to change based on feedback. After the open date, changing questions is highly discouraged (as it could change details which are relevant to forecasts that have already been submitted) and such changes are typically noted in the question body and in the comments on the question
• The close date is the date/time after which predictions can no longer be updated.
• The resolution date is the date when the event being predicted is expected to have definitively occurred (or not). This date lets Metaculus Admins know when the question might be ready for resolution. However, this is often just a guess, and is not binding in any way.

In some cases, questions must resolve at the resolution date according to the best available information. In such cases, it becomes important to choose the resolution date carefully. Try to set resolution dates that make for interesting and insightful questions! The date or time period the question is asking about must always be explicitly mentioned in the text (for example, "this question resolves as the value of X on January 1, 2040, according to source Y" or "this question resolves as Yes if X happens before January 1, 2040)".

The close date must be at least one hour prior to the resolution date, but can be much earlier, depending upon the context. Here are some guidelines for specifying the close date:

• If the outcome of the question will very likely or assuredly be determined at a fixed known time, then the closing time should be immediately before this time, and the resolution time just after that. (Example: a scheduled contest between competitors or the release of scheduled data)
• If the outcome of a question will be determined by some process that will occur at an unknown time, but the outcome is likely to be independent of this time, then it should be specified that the question retroactively closes some appropriate time before the process begins. (Example: success of a rocket launch occurring at an unknown time)
• If the outcome of a question depends on a discrete event that may or may not happen, the close time should be specified as shortly before the resolve time. The resolve time is chosen based on author discretion of the period of interest.

Note: Previous guidance suggested that a question should close between 1/2 to 2/3 of the way between the open time and resolution time. This was necessary due to the scoring system at the time, but has been replaced by the above guidelines due to an update to the scoring system.

What timezone is used for questions?

For dates and times written in the question, such as "will event X happen before January 1, 2030?", if the timezone is not specified Coordinated Universal Time (UTC) will be used. Question authors are free to specify a different timezone in the resolution criteria, and any timezone specified in the text will be used.

For date range questions, the dates on the interface are in UTC. Typically the time of day makes little difference as one day is miniscule in comparison to the full range, but occasionally for shorter term questions the time of day might materially impact scores. If it is not clear what point in a specified period a date range question will be resolved as, it resolves as the midpoint of that period. For example, if a question says it will resolve as a certain day, but not what time of day, it will resolve as noon UTC on that day.

Who decides the resolution to a question?

Only Metaculus Administrators can resolve questions. Binary questions can resolve Yes, No, Ambiguous, or Annuled. Range questions can resolve to a specific value, an out-of-bounds value, Ambiguous, or Annuled.

What are "Ambiguous" and "Annulled" resolutions?

Sometimes a question cannot be resolved because the state of the world, the truth of the matter, is too uncertain. In these cases, the question is resolved as Ambiguous.

Other times, the state of the world is clear, but a key assumption of the question was overturned. In these cases, the question is Annulled.

In the same way, when a Conditional turns out to be based on an outcome that did not occur, it is Annulled. For example, when a Conditional Pair's parent resolves Yes, the if No Conditional is Annulled.

When questions are Annulled or resolved as Ambiguous, they are no longer open for forecasting, and they are not scored.

If you'd like to read more about why Ambiguous and Annulled resolutions are necessary you can expand the section below.

No Clear Consensus

Questions can also resolve Ambiguous when there is not enough information available to arrive at an appropriate resolution. This can be because of conflicting or unclear media reports, or because a data source that was expected to provide resolution information is no longer available. The following are some examples where there was no clear consensus.

• Will Russian troops enter Kyiv, Ukraine before December 31, 2022?
  • This question asked if at least 100 Russian troops would enter Ukraine before the end of 2022. It was clear that some Russian troops entered Ukraine, and even probable that there were more than 100 Russian troops in Ukraine. However there was no clear evidence that could be used to resolve the question, so it was necessary to resolve as Ambiguous. In addition to the lack of a clear consensus, this question is also an example of imbalanced outcomes and the need to preserve incentives. As an Admin explains here, due to the uncertainty around events in February the question could not remain open to see if a qualifying event would happen before the end of 2022. This is because the ambiguity around the events in February would necessitate that the question could only resolve as Yes or Ambiguous, which creates an incentive to forecast confidently in an outcome of Yes.
• What will the average cost of a ransomware kit be in 2022?
  • This question relied on data published in a report by Microsoft, however Microsoft's report for the year in question no longer contained the relevant data. It's Metaculus policy that by default if a resolution source is not available Metaculus may use a functionally equivalent source in its place unless otherwise specified in the resolution text, but for this question a search for alternate sources did not turn anything up, leading to Ambiguous resolution.

Annulment

Annulling a question is reserved for situations where reality is clear but the question is not. In other words, the question failed to adequately capture a method for clear resolution.

Note: Annulment was introduced in April of 2023, so while the following examples describe Annulment the questions in actuality were resolved as Ambiguous.

The Question Was Underspecified

Writing good forecasting questions is hard, and it only gets harder the farther the question looks into the future. To fully eliminate the potential for a question to be Annulled the resolution criteria must anticipate all the possible outcomes that could occur in the future; in other words, there must be clear direction for how the question resolves in every possible outcome. Most questions, even very well-crafted ones, can't consider every possible outcome. When an outcome occurs that does not correspond to the instructions provided in the resolution criteria of the question then that question may have to be Annulled. In some cases we may be able to find an interpretation that is clearly an appropriate fit for the resolution criteria, but this is not always possible.

Here are some examples of Annulment due to underspecified questions:

• What will Substack's Google Trends index be at end of 2022?
  • This question did not clearly specify how Google trends would be used to arrive at the average index for December of 2022, because the index value depends on the date range specified in Google Trends. An Admin provided more details in this comment.
• When will a fusion reactor reach ignition?
  • This question did not clearly define what was meant by “ignition”. As an Admin described in this comment, the definition of ignition may vary depending on the researchers using it and the fusion method, as well as the reference frame for what counts as an energy input and output.
• Will Russia order a general mobilization by January 1, 2023?
  • This question asked about Russia ordering a general mobilization, but the difficult task of determining that a general mobilization was ordered was not adequately addressed in the resolution criteria. The text of the question asked about a “general mobilization”, but the definitions used in the resolution criteria differed from the common understanding of a “general mobilization” and didn’t adequately account for the actual partial mobilization that was eventually ordered, as explained by an Admin here.

The Assumptions of the Question Were Subverted

Questions often contain assumptions in their resolution criteria, many of which are unstated. For example, assuming that the underlying methodology of a data source will remain the same, assuming that an organization will provide information about an event, or assuming that an event would play out a certain way. The best practice is to specify what happens in the event certain assumptions are violated (including by specifying that the question will be Annulled in certain situations) but due to the difficulty in anticipating these outcomes this isn't always done.

Here are some examples of Annulment due to subverted assumptions:

• Will a technical problem be identified as the cause of the crash of China Eastern Airlines Flight 5735?
  • This question relied on the conclusions of a future National Transportation Safety Board (NTSB) report. However, it was a Chinese incident so it was unlikely that the NTSB would publish such a report. Additionally, the question did not specify a date by which the report must be published resulting in a resolution of No. Since this was not specified and the assumption of a future NTSB report was violated the question was Annulled, as explained by an Admin here.
• What will the Federal Reserves' Industrial Production Index be for November 2021, for semiconductors, printed circuit boards and related products?
  • This question did not provide a description of how it should resolve in the event the underlying source changed its methodology. It anticipated the possibility of the base period changing, however, the entire methodology used to construct the series changed before this question resolved, not just the base period. Because the unwritten assumption of a consistent methodology was violated, the question was Annulled.
• When will Russia's nuclear readiness scale return to Level 1?
  • Media reporting about Russia's nuclear readiness level gave the impression that the level had been changed to level 2, leading to the creation of this question. However, a more thorough investigation found that Russia's nuclear readiness most likely did not change. This violated the assumption of the question leading to the question being Annulled, as explained by an Admin here.
• What will be the Biden Administration's social cost of 1 ton of CO2 in 2022?
  • This question specified that it would resolve according to a report published by the US Interagency Working Group (IWG), however the IWG did not publish an estimate before the end of 2022. This question anticipated this outcome and appropriately specified that it should be Annulled if no report was published before the end of 2022, and the question was resolved accordingly.

Imbalanced Outcomes and Consistent Incentives

Sometimes questions imply imbalanced outcomes, for example where the burden of proof for an event to be considered to have occurred is high and tips the scales toward a binary question resolving No, or where the question would require a substantial amount of research to surface information showing that an event occurred, which also favors a resolution of No. In certain circumstances these kinds of questions are okay, so long as there is a clear mechanism for the question to resolve as Yes and to resolve as No. However, sometimes questions are formulated such that there's no clear mechanism for a question to resolve as No, leading to the only realistic outcomes being a resolution of Yes or Annulled. This creates a bias in the question and also produces bad incentives if the question isn't Annulled.

The case of imbalanced outcomes and consistent incentives is best explained with examples, such as the following:

• Will any prediction market cause users to lose at least $1M before 2023?
  • This question asks whether certain incidents such as hacking, scams, or incorrect resolution lead to users losing $1 million or more from a prediction market. However, there's no clear mechanism specified to find information about this, as prediction markets aren't commonly the subject of media reports. Concretely proving that this did not occur would require extensive research. This creates an imbalance in the resolution criteria. The question would resolve as Yes if there was a clear report from credible sources that this occurred. However, to resolve as No it would require extensive research to confirm that it didn't occur and a knowledge of the happenings in prediction markets that most people do not possess. To resolve as No Metaculus would either have to do an absurd amount of research, or assume that a lack of prominent reports on the topic is sufficient to resolve as No. In this case the question had to be Annulled.
  • Now consider if there had been a clear report that this had actually occurred. In a world where that happened the question could arguably have been resolved as Yes. However, savvy users who follow our methods on Metaculus might realize that when a mechanism for a No resolution is unclear, that the question will then resolve as Yes or be Annulled. This creates bad incentives, as these savvy forecasters might begin to raise the likelihood of Yes resolution on future similar forecasts as they meta-predict how Metaculus handles these questions. For this reason, binary questions must have a clear mechanism for how they resolve as both Yes and No. If the mechanism is unclear, then it can create bad incentives. Any questions without a clear mechanism to resolve as both possible outcomes should be Annulled, even if a qualifying event occurs that would resolve the question as Yes.
• Will any remaining FTX depositor withdraw any amount of tradeable assets from FTX before 2023?
  • This question asked if an FTX depositor would withdraw assets where the withdrawal was settled by FTX. Unfortunately this question required a knowledge of the details of FTX withdrawals that was unavailable to Admins, resulting in there being no real mechanism to resolve the question as No. This led to an imbalance in possible outcomes, where the question could only truly resolve as Yes or be Annulled. The imbalance necessitated that the question be resolved as Ambiguous to preserve consistent incentives for forecasting.

How do I use the range interface?

Some Metaculus questions allow numeric or date range inputs, where you specify the distribution of probability you think is likely over a possible range of outcomes. This probability distribution is known as a probability density function and is the probability per unit of length. The probability density function can be used to determine the probability of a value falling within a range of values.

When you hover over the chart you see the probabilities at each point at the bottom of the chart. For example, in the image below you can see the probability density at the value 136, denoted by "P(x = 136)", and you can see the probability density that you and the community have assigned to that point (in the image the user has assigned a probability density of 1.40 to that value and the community has assigned a probability density of 2.97).

By selecting the "Probability Density" dropdown at the top of the chart you can change the display to "Cumulative Probability". This display shows the cumulative distribution function, or in other words for any point it shows you the probability that you and the community have assigned to the question resolving below the indicated value. For example, in the image below you can see the probability that you and the community have assigned to the question resolving below the value of 136, denoted by "P(x < 136)". The probability that the user has assigned is 7% to the question resolving below that value, while the community has assigned an 83% chance to the question resolving below that value.

The vertical lines shown on the graphs indicate the 25th percentile, median, and 75th percentile forecasts, respectively, of the user and the community. These values are also shown for the user and the community in the table at the bottom.

What is "Reaffirming" a prediction?

Sometimes you haven't changed your mind on a question, but you still want to record your current forecast. This is called "reaffirming": predicting the same value you predicted before, now.

It is also useful when sorting questions by the age of your latest forecast. Reaffirming a question sends it to the bottom of that list.

You can reaffirm a question from the normal forecast interface on the question page, or using a special button in feeds.

On question groups, reaffirming impacts all subquestions on which you had a forecast, but not the others.