The subgroup that will fail in a Phase II trial is rarely identifiable from a single data modality. Clinical characteristics captured in EHR data tell you about disease presentation and comorbidity burden. Genomic data tells you about inherited variant effects on pathway activity. Imaging-derived biomarkers tell you about structural and functional disease state at a point in time. Each modality captures a different projection of the patient. The signal for which projection predicts non-response lives in the conditional dependencies between them.
Building virtual patient distributions that are useful for protocol decisions requires integrating all three modalities, not because the data is available and we want to use it, but because single-modality models systematically miss the conditioning dimensions that drive subgroup structure. This post describes how we approach multimodal integration at Valinor Discovery, what the technical challenges actually are, and where the method produces reliable inference versus where the uncertainty is genuinely unresolvable.
What each modality contributes and what it cannot do alone
EHR-derived cohort data is typically the most volumetrically rich source for a given indication. Large disease registries and claims databases cover thousands of patients with structured longitudinal records: diagnosis codes, procedure history, lab values, vital signs, prior medication exposure. The coverage is broad, and the data reflects real-world treatment patterns rather than the selected populations in clinical trials.
What EHR data rarely contains is the molecular characterization needed to distinguish subgroups defined by pathway-level mechanisms. Two patients with identical ICD code profiles, comparable lab values, and similar prior treatment histories may differ substantially in the upstream pathways driving their disease, and that difference is what determines their likely response to a mechanism-targeted compound. EHR data cannot tell you which of those patients is in the responsive subgroup.
Genomic variant data, typically variant allele frequencies from population databases and, where available, indication-specific sequencing studies, provides the molecular layer. It tells you about the frequency distribution of variants relevant to the compound's mechanism in the disease population. The challenge is that population-level variant frequencies are often not matched to the clinical profiles in your EHR cohort. You have genomic frequencies from one source and clinical characteristics from another, and integrating them requires statistical bridging rather than a direct match.
Imaging-derived biomarkers, whether quantitative volumetric measurements, radiomic features, or functional imaging parameters, add a real-time disease state dimension that neither EHR structured data nor germline genomics captures. For conditions where disease burden or tissue state at baseline is a determinant of response, imaging features can be among the strongest conditioning variables in the virtual patient model. The limitation is that imaging data is often the most sparsely available and the hardest to aggregate consistently across cohort sources.
The alignment problem at the center of multimodal integration
The core technical challenge is that these three modalities are almost never available from the same patients in the same study. You have EHR data from one registry. You have genomic frequencies from a population genetics database that sampled a partially overlapping but not identical population. You have imaging biomarker statistics from a published natural history study with its own inclusion criteria.
Attempting to concatenate these sources as if they described the same patients would produce a virtual population with fundamentally incorrect conditional structure. The dependencies between genomic features and clinical variables in your integrated model would reflect statistical artifacts of how the sources relate to each other, not how the underlying biology relates clinical state to molecular profile.
The approach we use treats each data source as a partial view of the underlying patient population, with its own selection function. The integration layer builds a hierarchical model in which the true joint distribution over all modalities is the shared latent quantity, and each observed data source is a conditional distribution over that latent quantity given its source-specific selection. This is the same probabilistic framework that underlies the Bayesian cohort reconstruction we described in an earlier post, extended to handle the alignment problem across modality-specific sources.
In practice this means we maintain separate likelihood terms for each data source, parameterized by source-specific selection models that we estimate from what we can observe about how each source was constructed: inclusion criteria, geographic coverage, collection time window. The posterior over the full joint distribution integrates evidence across all sources while accounting for the fact that they selected their patients differently.
Format-level challenges that consume most of the engineering effort
The statistical framework described above is the conceptually important part of multimodal integration. In practice, a large fraction of the effort in any real project goes into format-level alignment before the statistical work can even begin.
EHR data structured to HL7 FHIR standards uses a different terminological layer than CDISC SDTM/ADaM clinical trial data, which in turn uses a different coding structure than the registry data that comes as flat file cohort summaries. Variant data from population genomics databases arrives as VCF-format allele frequency tables that require crosswalk to whatever variant classification scheme the indication's published literature uses. Imaging biomarkers may be reported in different measurement conventions across the studies you are drawing from.
None of this terminological and format alignment is intellectually interesting. All of it has to be done correctly for the integration to produce valid results. A mapping error between a lab value code and its intended clinical variable will silently propagate through the Bayesian model and produce a virtual patient distribution that looks plausible but has a systematic bias in the conditioning structure for that variable. The model will not alert you to the mapping error; it will incorporate it as if it were ground truth.
We have invested heavily in the data validation infrastructure that catches these errors before they enter the probabilistic model, specifically because the downstream consequences of silent mapping errors are much harder to detect than the upfront cost of rigorous data validation.
What the integrated model enables that single-modality models cannot
The value of multimodal integration for virtual patient modeling shows up most clearly in enrichment scenario evaluation. Consider a candidate eligibility stratifier that is partially correlated with clinical severity score, which is available in EHR data, and also correlated with a variant frequency in a gene that affects drug metabolism. If you model only the EHR layer, you see the clinical severity correlation but miss the pharmacogenomic dimension. If you model only the genomic layer, you see the variant effect but cannot condition on the clinical profile the protocol will actually use as eligibility criteria.
The integrated model lets you ask: among patients who meet the clinical eligibility criteria I am considering, what is the conditional distribution of the pharmacogenomic variant? That is the question that matters for protocol design, and it can only be answered from a joint model of both modalities.
We are not suggesting that multimodal integration resolves all uncertainty in enrichment scenario prediction. Gaps in data coverage, particularly for imaging-derived biomarkers in indications where prospective imaging cohorts are small, produce genuine uncertainty that no statistical method eliminates. The integrated model quantifies that uncertainty explicitly, which is itself useful: it tells the development team which dimensions of their enrichment strategy are well-supported by the available data and which dimensions have wide posterior uncertainty that a targeted data collection effort might be worth addressing before the protocol is finalized.