Forecasting manifesto

Guidelines for rigorous forecasting research.

Christian Mueller 2025 June 6

The academic forecasting literature does not

The demand for economic forecasting is difficult to overstate. It is not only governments that rely on forecasts for budgeting, taxation, and expenditure, but also central banks, corporations, and even consumers who peer into the proverbial crystal ball in an attempt to anticipate the future. Fortunately, economists are generally enthusiastic about forecasting—and even more so about improving their forecasts. Consequently, academic journals regularly publish so-called ‘horse races’, pushing the boundaries of best practice ever further.

Or do they?

In truth, they do not. Rather than pushing boundaries, they run in circles—like a ship firmly anchored. The reasons for this shortcoming can be summarised by the confounding factors inherent in as-if ex-ante forecasting experiments:

• the use of revised data instead of real-time data
• a failure to distinguish between ‘actual’ and ‘pseudo’ forecasting
• publication bias in favour of models that win the horse race
• insufficient disclosure of modelling choices and parameter settings
• neglect of the impact of data vintages and benchmark revisions

Forecasting Study Guidelines

To remedy these systemic issues, journals should only publish contributions that:

• preregister the respective forecasting study in a public ledger.

Upon preregistration, authors should provide the following information, which will be made publicly available:

• precise model specification
• full code and dataset, including data vintages
• parameter values used for forecasting
• benchmarks for comparison
• evaluation method
• forecasting performance measures

Reasoning

Looking at prevailing forecasting practices, one feature still stands out: it is referred to as ‘quasi’, or more accurately, ‘pseudo’. A typical forecasting paper employs ‘quasi’ real-time data to support its claims. That is, the value of a variable at some point in the past, say \( t-p \), is estimated using data that was available only up to \( t-p-1 \), i.e., the period before the value of the variable was realised.

A 'horse race’ between competing estimation models subsequently determines the winning model by comparing the realisations of the variable in \( t-p, t-p+1, \dots, t \) with their estimated counterparts.

This ‘quasi real-time’ approach is intended to mimic an actual forecaster’s situation, and a brief review of the literature confirms that the method appears to satisfy editors and referees alike. However, from both methodological and practitioner perspectives, its shortcomings warrant serious scrutiny.

More formally, the distinction between actual and published forecasting can be illustrated by comparing the forecasting problems these two types of forecasters address: at time \( t \), both place their bets. However, while actual forecasters predict values for observations in the future—say, for \( t+1, \dots, t+f,\) where \( f>0 \) —published forecasters estimate values for the variable of interest in periods \( t-p, t-p+1, \dots, t-1,\) where \( p>0 \). These two scenarios may be visualised as follows:

The timing of published and of actual forecasting

The ‘quasi real-time’ method of published forecasting suffers from several shortcomings that render it unsuitable for informing best forecasting practices, even though it continues to be regarded as acceptable for publication in academic journals. This situation is untenable.

The main issue is that ‘quasi real-time’ forecasting draws on considerably more relevant information than is available to actual forecasters. The informational advantage typically includes knowledge of the following:

Tacit information advantage in ‘quasi real-time' forecasting

Researchers typically attempt to address their knowledge of the data to be forecast by restricting the data input, as described above, which constitutes the core of the ‘quasi real-time’ approach. Leaving aside the additional issue of data revisions, it is evident that this restriction remains insufficient. Knowing the realised values of forecast variables entails awareness of specific events during the forecast period and generally enables the researcher to tailor model choices to optimise the evaluation criterion. None of these circumstances apply to actual forecasting, nor does the actual forecaster have the authority to judge his or her own forecasts. Thus, while the winner of the horse race can be determined in ‘quasi real-time’, they cannot be identified when confronted with an actual forecasting task.

Recasting this issue in the terms of academic research, a meaningful horse race should take place in genuine real time; that is, the ‘quasi real-time’ exercise should be repeated post-publication to assess which method truly withstands the test of time. Notably, few journals — if any — follow up ‘quasi real-time' horse races with genuine real-time comparisons between competitors.

The issue of the availability of models and methods is more nuanced and therefore often overlooked. When conducting a horse race, the researcher often has in mind a novel forecasting technique that he or she wishes to promote. This is fair; however, if the novel technique is applied to historical data, comparing forecasts that were not available at the time does not inform best forecasting practice. If a novel method is to be compared to a legacy incumbent, the comparison should be made for data in the range \([t+1, t+f]\), not \([t-p, t]\). Once again, a genuine real-time approach is the only appropriate solution.

Occasionally—mostly in the context of industry benchmarking—genuine real-time forecasts are also compared. For instance, a newspaper or industry body may crown the best forecaster or forecasting institution. While such comparisons may appear to identify the best forecasting method, they suffer from notable limitations, chiefly a lack of detail.

In practice, forecasting often involves multiple individuals, sometimes linked by formal or informal hierarchies. Moreover, forecasts are frequently produced for distinct purposes. These two factors alone allow considerable scope for intervention in the forecasting process, as forecast values may trigger responses that are either desirable or undesirable. Likewise, senior staff may feel entitled—or obliged—to intervene in order to enhance or influence the results. Such interventions, as well as the many undocumented micro-decisions made during forecasting, largely remain unrecorded.

As a result, forecasts produced in practice often lack academic rigour and are affected by undocumented interference, rendering the replication and dissemination of best practice unfeasible.

Time to act is now

Though methodologically appealing, the forecasting literature offers little in the way of experimental evidence. Indeed, although both experimental economics and economic forecasting are active fields of research, their intersection remains rare. A notable exception is Woodard, Sornette and Fedorovsky (2011), who tested their asset pricing models with considerable rigour.1 It seems reasonable to assume that many successful forecasting models are shaped, at least in part, by the researcher’s unintentional use of information unavailable in a genuinely ex-ante setting. Other disciplines, such as the medical sciences, have long acknowledged these issues, and drugs are now routinely required to undergo prospective test studies. Moreover, to mitigate publication bias and prevent the suppression of unfavourable results, several leading medical journals now require clinical trials to be preregistered for consideration.

By contrast, in macroeconometrics, these confounding factors exert their full influence on the outcome.

For instance, a researcher may select a model that performs best in-sample and recommend it based on simulated forecasting success. In real-world settings, however, the future is unknown — as is the benchmark used for model selection.2 When combined with publication bias, one might suspect that a substantial proportion of published forecasting results may not withstand practical scrutiny.

In applied contexts, however, forecasters require robust scientific guidance to fulfil their tasks. Indeed, economic research institutions across the globe engage in genuine ex-ante forecasting, although their guiding principles are derived largely from theory and retrospective analysis. This manifesto seeks to bridge the fundamental gap between the needs of practitioners and the prevailing academic literature.

Support this manifesto—champion higher academic standards in forecasting research.