Healthcare AI teams rarely fail on model architecture. They fail on data they cannot defend. A dataset without documented consent, clear provenance and disciplined de-identification is a liability no matter how large it is, and that liability tends to surface at the worst moment: during a regulatory submission, a hospital procurement review or a partner's due diligence.
The usual response is to treat compliance as a filter applied at the end. Strip the names, hash the record numbers, ship it. That produces data that is technically de-identified and practically useless, because the same process that removes identifiers also removes the links between records that made them clinically meaningful. Building clinical datasets that are both defensible and useful requires designing for both from the start.
01The dataset you cannot defend
Consider what a reviewer at a regulator or a health system will ask about the data behind a diagnostic model:
- Where did each record come from, and under what agreement?
- Did patients consent to this use, and can you show it?
- How were identifiers removed, and who checked the result?
- Who had access to the data before de-identification, and under what controls?
- Does the training population resemble the population the model will be used on?
If the answer to any of these is a shrug or a reference to a third-party aggregator's terms of service, the model's performance numbers do not matter. The data has to be explainable record by record, which means provenance has to be captured at collection, not reconstructed afterwards.
02Provenance, step by step
Our clinical datasets are built around a documented chain from collection to delivery. The specifics vary by engagement, but the structure does not:
Consent and licensing first
Data is collected under informed consent through partner institutions, with the terms of use documented per source before anything moves. If consent does not cover the intended use, the data does not enter the dataset.
De-identification with a human in the loop
Automated de-identification catches most identifiers in structured fields and a large share in free text. It misses the rest: a relative's name in a clinical note, a hospital name burned into an image header, a date that combined with a rare diagnosis points to one person. Every pass is followed by human review, and records that cannot be cleaned confidently are rejected rather than patched.
HIPAA allows two routes. Safe Harbor removes 18 listed identifiers, such as names, dates other than the year and geographic units smaller than a state. Expert Determination has a qualified statistician certify that the risk of re-identifying anyone is very small, which can keep more useful detail, such as exact dates, when the analysis supports it. Either way, data de-identified to the HIPAA standard is no longer protected health information. GDPR sets a higher bar: pseudonymised data still counts as personal data, and only data that is truly anonymised falls outside it.
Controlled handling
Access controls, documented handling procedures and project-specific contractual safeguards apply throughout. HIPAA and GDPR requirements are addressed according to the data, the processing roles and the agreements for each engagement, rather than with a single generic claim of compliance.
Because properly de-identified data is no longer protected health information, licensing it does not generally require a business associate agreement. A data use agreement should still set what the data can be used for and prohibit any attempt to re-identify patients.
De-identification is the floor. Provenance is what makes the data defensible.
03Why patient-level linkage matters
Clinical reality is multimodal. A diagnosis is reached by combining imaging, pathology, labs, genetics and history, often over years. A dataset that contains each of those as an isolated pool can support narrow benchmarks, but it cannot teach a model to reason the way clinicians do.
We link records at the patient level before de-identification, so the links survive it. In oncology, that means whole-exome sequencing, digital pathology whole-slide images, FFPE tissue, DNA and RNA, lab reports and structured records across 13 categories travel together for more than 5,000 longitudinal patients. In dermatology, clinic-sourced photographs track the progression of a condition across multi-year follow-up, alongside redacted labs and prescriptions.
That linkage is what makes the data useful for real diagnostic and risk-prediction models: a model can learn how a lesion's appearance relates to its pathology, or how a tumour's genetics relate to its response to treatment, instead of learning each signal in isolation.
04The population gap
Most widely available clinical datasets come from a small number of health systems in North America and Western Europe. Models trained on them inherit their demographics, and the consequences are measurable: dermatology models that perform worse on darker skin, oncology models calibrated on populations with different disease prevalence and genetic profiles.
We focus deliberately on non-Caucasian populations, with data across 15 countries including South and Southeast Asia, Latin America and the Middle East. This is not a diversity checkbox. Models trained on demographically narrow data underperform exactly where health systems are already weakest, and those are increasingly the markets where clinical AI is being deployed.
05Expert validation, not just labels
Clinical labels are only as good as the people who made them. Healthcare professionals continuously validate annotations across imaging, notes and records, flag inconsistencies and escalate cases that need a second opinion. Across projects we maintain annotation accuracy above 99%, and the review criteria each label was made under ship with the data.
The same reviewers write and grade clinical evaluation sets: case-based questions with expert rubrics and a held-out answer key, scored the same way every run. For teams building medical assistants, that is often the most valuable deliverable, because it shows where a model's clinical reasoning breaks before a clinician finds out the hard way.
06Questions to ask any clinical data vendor
- Can you show consent and usage terms per source?
- Is de-identification checked by a person, and what happens to records that fail?
- Are records linked at the patient level after de-identification?
- What is the demographic and geographic make-up of the population?
- Who validated the labels, and what were their qualifications?
- Which formats do you deliver? DICOM, WSI, FHIR and HL7 should not require a conversion project.
If you are building in this space, our healthcare vertical covers what is available today. Samples are available under NDA.
