Mortality Watch API: Cumulative Endpoint Bug

by Alex Johnson 45 views

In the realm of statistical analysis, particularly when dealing with time-series data like mortality rates, accuracy and proper representation are absolutely paramount. Our exploration into the MortalityWatch stats platform has uncovered a peculiar issue with its /cum endpoint. This endpoint, designed to handle cumulative calculations, appears to be returning a flat baseline instead of the expected accumulating values. This seemingly small deviation has significant implications for how cumulative prediction intervals are displayed and interpreted, potentially leading to misleading visualizations on cumulative excess charts.

Understanding Cumulative Data and Prediction Intervals

Before diving into the specifics of the bug, let's set the stage by understanding what cumulative data and prediction intervals mean in this context. When we talk about cumulative values, we're referring to a running total. In the case of mortality rates, a cumulative endpoint would, ideally, show the total predicted excess mortality up to a certain point in time, building upon the values from previous periods. For example, if the baseline mortality for period 1 is X, for period 2 it should be X + the mortality of period 2, and so on. This creates a continuously increasing trend line representing the accumulated effect.

Prediction intervals, on the other hand, are crucial for understanding the uncertainty associated with our predictions. In any statistical model, there's an inherent degree of variability. Prediction intervals provide a range within which we expect the actual future values to fall, with a certain level of confidence (e.g., 95% confidence interval). When dealing with cumulative data, we expect these intervals to widen over time. Why? Because as we extend our predictions further into the future, the accumulation of uncertainty naturally increases. Small errors or variations in early periods can compound over time, leading to a broader range of possible outcomes for later periods. A widening prediction interval visually communicates this growing uncertainty, which is a fundamental aspect of statistical forecasting.

Imagine predicting the weather for the next hour versus the next month. Your prediction for the next hour is likely to have a tighter, more precise range of possible temperatures than your prediction for the next month. The same principle applies to cumulative mortality data. The /cum endpoint is intended to provide this cumulative view, and the associated prediction intervals should reflect the increasing uncertainty as we look further out in cumulative time.

The /cum Endpoint Issue: A Flat Baseline Problem

The core of the problem lies in how the /cum endpoint is currently functioning. When making a call to this endpoint, instead of receiving baseline (y) values that accumulate over time as expected, we are observing a consistent, flat baseline. This means that for each period requested in the cumulative mode, the endpoint returns the same baseline value. Let's illustrate with the example provided:

https://stats.mortality.watch/cum?y=470.7,471.5,464.1,458.4,520.7,571.2,494.7,456.3,434.1&bs=1&be=4&t=0&xs=2016

When this call is made, the expected behavior for cumulative data would be for the y values in the response to show an increasing trend. For instance, if the initial calculated baseline is 466.175, the subsequent y values should be approximately 466.175 * 2, 466.175 * 3, and so on. This cumulative increase is what accurately represents the total predicted baseline over successive periods.

However, the actual response reveals a different story:

{
  "y": [466.175, 466.175, 466.175, 466.175, 466.17, 466.18, 466.17, 466.18, 466.18, 2797.05],
  "lower": [null, null, null, null, 444.28, 454.15, 455.23, 455.65, 455.88, 2721.21],
  "upper": [null, null, null, null, 488.07, 478.2, 477.12, 476.7, 476.47, 2872.89],
  "zscore": [0.735, 0.865, -0.337, -1.264, 8.861, 17.068, 4.636, -1.605, -5.213, null]
}

As you can observe, the y values are largely repeating the same number (466.175 for the initial periods). This 'flat baseline' is the root cause of the subsequent visualization issues. It fails to represent the accumulating nature of the data when in cumulative mode. This is not just a theoretical problem; it has tangible consequences for users interpreting the data, especially when visualized.

The Impact on Visualizations: Misleading Prediction Intervals

The most immediate and concerning impact of this flat baseline is on the visual representation of prediction intervals in cumulative excess charts. When the y (baseline) values are flat, the calculated lower and upper bounds also tend to remain relatively constant or do not widen as they should. This creates an optical illusion where the prediction interval bands appear to have a consistent width across all periods, regardless of how far into the cumulative timeline we are looking.

In a correct cumulative prediction interval plot, we expect to see the interval bands widen as time progresses. This widening signifies the increasing uncertainty inherent in longer-term forecasts or cumulative totals. The flat baseline from the /cum endpoint prevents this natural widening. Consequently, users viewing cumulative excess charts might mistakenly believe that the uncertainty is constant, or even decreasing, when in reality, it should be accumulating.

This issue has been observed in the production environment, specifically on the MortalityWatch Explorer. When cumulative excess charts are generated, the prediction intervals do not expand as they should, failing to accurately communicate the growing uncertainty over time. This can lead to incorrect interpretations of risk, confidence in predictions, and the overall reliability of the cumulative data being presented. For researchers, public health officials, or anyone relying on this data, such misrepresentations can have significant consequences for decision-making.

Steps to Reproduce and Expected Behavior

To truly understand and verify this bug, let's walk through the steps to reproduce it and clarify what the expected outcome should be. The process is straightforward and involves making a specific API call and then analyzing the response.

Steps to Reproduce:

  1. Access the /cum Endpoint: The primary step is to make a request to the https://stats.mortality.watch/cum endpoint. This endpoint is specifically designed for cumulative calculations.

  2. Provide Input Data: You need to supply the necessary parameters. In the example that highlights the issue, the following parameters are used:

    • y=470.7,471.5,464.1,458.4,520.7,571.2,494.7,456.3,434.1: This is the input mortality data for several periods.
    • bs=1: Baseline start index.
    • be=4: Baseline end index.
    • t=0: Time parameter.
    • xs=2016: Start year.

    The full call looks like this:

    https://stats.mortality.watch/cum?y=470.7,471.5,464.1,458.4,520.7,571.2,494.7,456.3,434.1&bs=1&be=4&t=0&xs=2016

  3. Observe the Response: After making the call, examine the JSON response returned by the API. This response contains the calculated y, lower, upper, and zscore values.

Expected Behavior:

When the /cum endpoint is functioning correctly for cumulative mode, the output should reflect accumulated values. Let's break down what this means for the key fields:

  • y (Baseline Values): Instead of a flat line, the y values should represent the cumulative baseline. If the initial calculated baseline value (derived from the input y and other parameters) is, for example, 466.175, then:

    • The y value for Period 1 should be 466.175.
    • The y value for Period 2 should be 466.175 * 2 = 932.35.
    • The y value for Period 3 should be 466.175 * 3 = 1398.53.
    • And so on, for each subsequent period. This demonstrates a clear upward trend, signifying the accumulation of the baseline over time.

  • lower and upper (Prediction Interval Bounds): Correspondingly, the lower and upper bounds should also reflect this cumulative nature and the increasing uncertainty associated with it. While the exact calculation might be complex, the key principle is that these bounds should widen as the cumulative period increases. This means:

    • The gap between lower and upper for Period 2 should be larger than the gap for Period 1.
    • The gap for Period 3 should be larger than for Period 2, and so forth.

This widening of the prediction interval is crucial for accurately visualizing the confidence range of the cumulative predictions. It tells the user that as we extend further into the cumulative timeline, our prediction becomes less certain, and the range of possible outcomes expands.

Actual Behavior:

Contrasting with the expected behavior, the current implementation of the /cum endpoint exhibits the following:

  • Flat y Values: As seen in the provided example response, the y values are essentially static for the initial periods. They do not accumulate. For instance, [466.175, 466.175, 466.175, 466.175, ...]. This directly contradicts the purpose of a cumulative endpoint.

  • Nearly Identical Prediction Interval Widths: Because the baseline y values are flat, the calculated lower and upper bounds do not exhibit the expected widening. When plotted, this results in prediction interval bands that appear to have a consistent, or nearly consistent, width across all cumulative periods. The visual representation fails to convey the increasing uncertainty that should naturally occur with cumulative forecasting.

This deviation means that the output from the /cum endpoint is not suitable for generating accurate cumulative excess charts where the accumulation of values and the corresponding growth in uncertainty are critical components of the analysis.

Environmental Context

This issue has been confirmed to be present in the current setup of the MortalityWatch stats service. The problem is specifically tied to the /cum endpoint, which is the designated API for handling cumulative calculations. Importantly, the frontend applications are correctly identifying the need for cumulative data and are indeed calling the /cum endpoint as intended. This suggests that the bug resides within the backend logic of the /cum endpoint itself, rather than in how it's being invoked by the frontend.

  • Endpoint in Question: https://stats.mortality.watch/cum

  • Frontend Behavior: The frontend is correctly configured to use this endpoint when cumulative mode is selected (e.g., ce=1 parameter in the explorer URL). This means that users interacting with the MortalityWatch Explorer and selecting cumulative views are directly experiencing the effects of this bug.

  • Production Impact: The problem is visible and impactful on the live production environment. The URL provided, https://www.mortality.watch/explorer/?c=USA-FL&c=USA-CA&e=1&df=2020&dt=2023&ce=1&pi=1&p=0, serves as a direct example where cumulative excess charts display these incorrect, flat prediction intervals. This makes the data visualization misleading for actual users.

Understanding this environmental context is crucial for diagnosing and resolving the issue. It confirms that the problem is not an isolated test case but a live bug affecting the platform's users.

Conclusion and Next Steps

The /cum endpoint in MortalityWatch stats is a critical component for analyzing cumulative mortality trends. The discovery that it returns a flat baseline instead of accumulating values is a significant issue. This directly impacts the accuracy of prediction intervals displayed in cumulative excess charts, leading to potentially misleading visualizations that fail to represent the increasing uncertainty over time. The problem is confirmed in the production environment, affecting users who rely on these visualizations for insights.

The immediate next step is to investigate and rectify the logic within the /cum endpoint. The development team needs to ensure that the baseline (y) values are correctly calculated as cumulative sums and that the prediction interval bounds (lower and upper) accurately reflect the compounded uncertainty associated with these cumulative predictions. Fixing this bug will restore the integrity of cumulative data analysis on the MortalityWatch platform, providing users with accurate and trustworthy insights.

For further information on statistical modeling and time-series analysis, you can refer to resources like Google AI Blog for advancements in predictive technologies or explore The World Bank's data analysis tools and methodologies. These external resources can provide broader context on the importance of accurate statistical representations.