Study reviews human judgment ensemble during Omicron BA.1 wave in the USA

In a recent study posted to the medRxiv* preprint server, researchers demonstrated how well human judgment systems could complement computational models in responding to the rapidly evolving coronavirus disease 2019 (COVID-19) pandemic.

Background

The COVID-19 pandemic has claimed over six million lives globally and continues to wreak havoc. Unfortunately, much longer after its epidemiological effects will subside, the economic and societal burden of this pandemic will continue to impact the world. Epidemiological surveillance systems and policymaking improved significantly during the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic. Yet, they still lagged in anticipating emerging SARS-CoV-2 variants, vaccine uptake, and behavioral adaptations, such as mask use and vaccine uptake.

Six to 10 multi-model ensembles and over 40 short-term forecasts have yielded robust forecasts and projections concerning COVID-19 in the United States (US). However, there is room for improvement; for instance, these models could be adapted on-the-fly to report artifacts in the data.

Human judgment systems have emerged as a robust complementary approach to model-driven epidemiological forecasting. An early interactive platform based on human judgment systems predicted trajectories of an ongoing influenza season one to four weeks into the future (short-term). Although difficult to scale and operationalize, this method performed among the best among other computational models. More recently, human judgment systems were used to estimate cases, deaths, and the impact of monkeypox virus outbreaks across the US, Europe, and Canada.

About the study

In the present study, researchers conducted a Real-time Pandemic Decision Making (RPDM) tournament to evaluate the community forecasting platform Metaculus. It involved six forecasting rounds during the Omicron BA.1 wave in the US for 18 weeks between November 2021 and March 2022. Notably, Metaculus generated a weighted ensemble (the Metaculus Prediction) in addition to individual forecasts.

Although the primary study aim was to determine the performance of the Metaculus prediction across various questions during the Omicron BA.1 wave, the team also summarized insights from some key questions as part of the independent Omicron set.

Furthermore, the team modified some forecasting questions between the two phases of the study – the surge phase and the decline phase. In addition, they elicited multiple forecast horizons for each variable to compare forecast uncertainty over time. The current study was undertaken in collaboration with the Virginia Department of Health.

Study findings

The current pilot study demonstrated that human judgment ensembles could provide valuable signals for real-time pandemic decision-making during periods of high uncertainty, such as the Omicron BA.1 surge in the US. Compared to iqrCOV, the Metaculus performed marginally better for a forecast horizon of fewer than two weeks. Metaculus also had predictions comparable to medMAPE throughout the entire forecast horizon of 28 days. While these other two metrics underestimated the Omicron BA.1 surge, the Metaculus predictions had tighter uncertainty bounds post-peak. Furthermore, Metaculus demonstrated nearly 70% interquartile range (IQR) coverage through most of the forecast horizon. In addition to forecasts of comparable accuracy, Metaculus provided timely updates to a diverse set of questions. Further, it allowed for iterations on the questions with the decision maker in the loop, thus, focusing on the variables of interest during different Omicron surge phases.

The US Centers for Disease and Prevention (CDC) Nowcast estimated Omicron prevalence at 73.2% for the week of December 12-18, 2021. However, many forecasters found these estimates implausible, given the estimated Omicron growth rate. The CDC later revised their estimates which showed that Omicron prevalence for that week was much lower — with a point estimate of 22.5% outside the confidence interval of the previous estimate. Metaculus forecasters also predicted an 80% chance that Omicron would be less lethal than Delta by December 23, 2021.

Platforms like Metaculus do not operate in isolation. They require well-organized disease surveillance dashboards, publicly available model forecasts, active social and news media discussion, and rapid distribution of scientific findings via preprints. Only then do the human participants serve as effective information aggregators who can produce forecasts for variables of interest through mental models.

Conclusions

Human judgment-based platforms, such as Metaculus, go beyond providing direct forecasts and must be combined with computational models to make the most of such ensembles. Since they can aggregate information, interactive computational models and other analytical tools could help them assimilate domain knowledge and minimize certain cognitive biases. Remarkably, they could also provide ‘public pulse’ estimates of behavioral aspects, such as mask use and vaccine uptake. Overall, prediction tools, such as Metaculus, could serve as aggregators, trackers, and ensembles of such forecasts to be useful for policymakers.

*Important notice

medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Written by

Neha Mathur

Neha is a digital marketing professional based in Gurugram, India. She has a Master’s degree from the University of Rajasthan with a specialization in Biotechnology in 2008. She has experience in pre-clinical research as part of her research project in The Department of Toxicology at the prestigious Central Drug Research Institute (CDRI), Lucknow, India. She also holds a certification in C++ programming.