Learn how EHR timestamps, joins and preparation can leak future information, and how to test a prediction pipeline against what was actually known.
Figures open at full size. Wide tables scroll sideways.
A lab result can be an accurate record of a patient's condition and still be an invalid input to a prediction made earlier. The missing distinction is between when the sample was taken and when its result became available.
That distinction is central to Elena Albu and colleagues' practical guide to extracting and preparing electronic health record data for dynamic prediction—models that update their predictions during a hospital admission. Their paper identifies more than forty challenges across cohort definition, outcome definition, feature engineering and cleaning.
For clinical machine-learning teams, the contribution is a detailed account of how an apparently sound dataset can diverge from the information available in a working hospital. It is a guide based on selected literature and extraction experience, not a new model with demonstrated improvements in patient outcomes.
The paper separates three stages. Hospital staff and devices collect data; extraction processes move records into a usable structure; preparation code converts them into model inputs. Each stage can introduce a different error.
An observation can be missing from the source record. A correct observation can be linked to the wrong admission during extraction. Correctly linked records can then be aggregated using information from the future. Treating all three as “dirty data” makes it harder to locate and correct the problem.
The authors' original diagram compares the retrospective model-building pipeline with the live prediction path. In the first, researchers can see a completed admission. In the second, a scheduled time or new event triggers a request using the information available then.
The engineering challenge is to make those paths agree at each prediction time. A cleaner retrospective dataset is not necessarily a more realistic one if its corrections or information cannot be reproduced during use.
The paper's timestamp recommendations distinguish a lab order, planned collection, actual sample collection, preliminary result and final validated result. Those events can be separated by substantial time. A positive microbiology result, for example, may become available more than a day after blood was collected.
The collection timestamp describes the patient's state when the sample was taken. It does not mean the eventual result was already known. If a training pipeline attaches that result to the collection time and lets earlier predictions use it, the model receives information that a deployed service would not yet have had. This is temporal leakage.
The same lab record can legitimately serve two different roles. To construct a retrospective outcome, researchers may need the clinically relevant collection time. To decide whether its value was eligible as a predictor, they need the time the information became available to the prediction system. Using one timestamp indiscriminately for both roles confuses the target with the inputs.
The problem also appears in routine documentation. The paper describes a nurse recording an observation at 10:13 but assigning it an observation time of 8:20. A replay of the hospital record at 9:00 must not treat that entry as already available merely because its clinical timestamp is earlier.
This is why timestamp names alone are insufficient. The extraction contract needs to explain whether a field represents an event, a planned task, entry into the system, validation or a later modification. If the relevant history was never stored, the team may be unable to reconstruct the intended prediction task reliably.
The guide focuses on structured data from a single hospital, including ICU and hospital-wide sources. It does not validate a cross-hospital platform or cover the extraction of notes and images. Its value lies in the concrete failure modes and remedies, not a headline accuracy improvement.
Patient and admission identifiers link records across systems, but historical merges, transfers and anonymization can create inconsistencies. A selection rule can also depend on information that becomes known only later. The question is whether the training cohort represents patients who could actually have been identified at the intended prediction time.
The authors recommend checking the completeness of inclusion criteria and the way admission episodes are constructed. A large dataset does not compensate for systematically selecting the wrong population.
A diagnosis code, clinical assessment and rule-derived outcome are not interchangeable labels. Coding can occur after discharge, documentation practices vary between wards, and a derived outcome may require several data sources.
The paper recommends checking those sources and testing the implemented definition with clinical experts. Otherwise, the model may learn differences in documentation rather than the intended event. It may also appear to predict an event that has already occurred if post-event records remain in the evaluation.
The same observation may live in different tables across wards or move when a software module changes. Mapping records into a common vocabulary helps organize what was found; it does not establish that every relevant source field was discovered.
The clinical-mapping recommendations therefore preserve source context and historical changes. A sudden fall in recorded measurements might reflect a new documentation workflow, an extraction mistake or an actual change in care. Those explanations should not be collapsed into one missing-value policy.
The paper cautions against case-by-case corrections that cannot be reproduced in the live pipeline. A researcher reviewing an entire admission can use later measurements to recognize an earlier error. A prediction service at that earlier time cannot.
The useful rule is to make transformations reproducible using information already available. This also argues for investigating surprising missingness before filling it in: missing ages or unusually sparse ICU measurements may reveal broken joins or mappings rather than a statistical imputation problem.
One of the paper's strongest implementation recommendations is to compare retrospective processing with incremental processing. Build the input at a chosen trigger using the historical pipeline, then build it again by revealing only the records available up to that trigger. The two should agree under the same feature definitions.
The original table excerpt below states that test in the authors' own engineering context. It is a selected row from a longer recommendations table, not a measured experiment.
| Problem | Description and recommendation |
| Temporal leaks due to incorrect date/time handling or incorrect linking during data preparation. |
DESCRIPTION: Incorrect time zone handling, incorrect joins, time based filtering or grouping and ordering items temporally can introduce temporal leaks during data preparation. Such leaks can distort predictions by providing information that would not be available at the time of prediction in clinical implementation. For example, misalignment of time zones may shift timestamps incorrectly, making future events appear as if they occurred earlier. Incorrect joins between tables (e.g., linking lab results to patient records without proper date/time filtering) can introduce data that should not be available at prediction time.
RECOMMENDATION: Thorough unit testing of the data preparation pipeline is essential to prevent temporal leaks. For example, when predictions are generated at specific triggers (every 6 or 24 hours), testing should ensure that processing a full admission retrospectively in the data pipeline produces the same results as processing data incrementally up to each trigger time. This step helps confirm that only information available at a given prediction time is used, preserving the model’s real-world applicability. METRIC: Timeliness - currency; WW: Currency |
A useful acceptance suite would include a late-entered observation, a revised lab result, a transfer between systems and a record whose timestamp lacks time-of-day precision. For each case, inspect the input record that became eligible, the feature produced and the prediction trigger. A mismatch should be explained before model scores are treated as representative of deployment.
This test does not establish clinical usefulness. It establishes a narrower prerequisite: the offline evaluation is using an information path the live system could reproduce.
OMOP's Common Data Model provides a common structure and vocabulary for observational health data. It is useful when a team needs consistent tables and concept mappings. Retain the original source identifiers and mapping history needed to investigate errors; standardization cannot recover information that was never extracted.
OHDSI's Data Quality Dashboard runs configurable checks against an OMOP database. Use it for systematic checks of fields, relationships and plausible values, then investigate failures in the source context. Passing these checks does not establish that every feature respects prediction-time availability.
For exchange with clinical systems, FHIR R4's Observation definition distinguishes the clinically relevant time from the time a version was issued. Map those fields to the hospital's actual recording and delivery process; neither field name should be assumed to capture every delay in a local integration.
The remaining work belongs in the existing extraction and preparation code. Preserve timestamp meanings, version transformations, test joins and run the incremental replay comparison. Keep patients separated appropriately across development and evaluation, and fit learned preprocessing on training data rather than the evaluation set. The paper's broader principle is consistent preparation, not adoption of a particular model library.
Before funding a more complex clinical prediction model, ask whether the team can reproduce the information available at each prediction time. If it cannot, a better offline score may be rewarding a better view of the future.
We would use this guide as a joint review between data engineers, researchers and clinical experts. Its strongest contribution is making failures specific enough to investigate. Successful replay and data-quality checks justify moving forward to clinical validation; they do not replace it. For the separate question of whether observed differences between groups imply unfair treatment, our causal-fairness analysis explains why a model audit needs explicit assumptions.
Elena Albu and colleagues, Challenges and Recommendations for Electronic Health Records Data Extraction and Preparation for Dynamic Prediction Modelling in Hospitalized Patients — a Practical Guide: Tutorial. arXiv version 2, posted 17 March 2025. Original diagram and selected table row reproduced under CC BY 4.0.
Investigate time leakage, revised data and the backtest behind an economic forecast.
Compare the cost of communication between stages with the parallelism lost by combining them.
