Every frontier lab has already trained on the open web. Public repositories, permissively licensed or otherwise, are saturated, and they skew toward tutorials, abandoned side projects and code that never shipped. What models have barely seen is how software actually gets built inside companies: long-lived codebases with CI pipelines, code review threads, migration histories and the messy fixes that follow real incidents.
That gap is now the binding constraint on coding models. Benchmarks keep climbing while engineers keep reporting the same failures in production: agents that write plausible code that does not fit the system around it, refactors that pass a local test and break three services downstream, fixes that silence an error instead of resolving it. Those are not reasoning failures so much as data failures. The model has never seen enough of the situations it is being asked to handle.
01What public code is missing
Public code is a snapshot of finished files. Even when a repository has history, most public projects are small, short-lived and maintained by one or two people. The median public repository has no meaningful review culture, a thin or absent test suite and a commit log that reads like a diary rather than an engineering record.
Production codebases look different along almost every axis a coding model cares about:
- Scale and age. Systems that have been maintained for a decade carry layers of decisions, deprecations and migrations. Our deepest repository has more than 43,500 commits, and the longest continuous timeline runs ten years.
- Team structure. Dozens of contributors with different habits, conventions enforced in review and ownership boundaries between modules.
- Constraints. Real dependencies, real deployment targets and real performance budgets, rather than code written to illustrate a concept.
- Consequences. Code that broke for customers, and the fixes that followed.
A model trained mostly on public code learns to write code that looks right in isolation. A model trained on production code has a chance to learn what makes code right in context.
02The lifecycle is the signal
The source files are the least interesting part of a licensed codebase. The value is in everything around them, the record of how the code got to its current state and why. We deliver repositories with that record intact:
Commit sequences
A bug fix is rarely one commit. It is a failing test, a first attempt, a revert, a narrower fix and a follow-up that handles the edge case nobody saw. That sequence teaches a model something no single diff can: how an engineer narrows down a problem.
Review threads
Pull request discussions are where engineering judgment is written down. A reviewer explaining why a change is rejected, whether it is the wrong abstraction, an unsafe migration or a missing index, is exactly the reasoning a coding agent needs and almost never gets. Across the collection there are more than 500,000 pull and merge requests, with merged and unmerged changes both kept.
Tests
A test suite is a definition of correct for that system. Pairing source with its tests, tracked separately per repository, lets teams train for test generation, coverage-aware completion and, most importantly, build tasks where success is checked by something other than a model's opinion.
An agent that has only seen finished code can imitate style. An agent trained on the full lifecycle can imitate judgment.
03Licensed, not scraped
None of this works if the data cannot be used. We work directly with software companies to license their private production repositories with consent and contracts. Every repository in the collection is a real commercial system. Forks, tutorials and toy projects are excluded at intake.
Open datasets take the other route. The large public code datasets behind most open code models are built from public code filtered by licence, and ask users to respect each file's original licence terms, including attribution. That makes them a strong, free base for pre-training. It does not give you a contract confirming the right to train, and it cannot include code that was never public.
Licensing is not just a legal formality. It shapes what a lab can do with the model afterwards. A model trained on data of unknown provenance carries that uncertainty into every deployment, every enterprise contract and every audit. Documented rights per repository remove the question entirely.
Before delivery, secrets are removed and repository metadata is structured for dataset use. Specific sanitization requirements, such as removing internal hostnames, customer names in fixtures or proprietary comments, are agreed with each customer during licensing, because the right answer depends on how the data will be used.
04What it unlocks for coding agents
The biggest shift is not in pre-training but in what becomes possible downstream. Real repositories with real tests are the raw material for agentic environments that actually resemble the job.
A task built from a production repository looks like this: start from the repository at a specific commit, give the agent a shell, an editor and a test runner, and ask it to fix the failing test that a real engineer fixed. Success is the test passing with no new failures, checked by the test suite and an engineer rubric. A human reference solution comes with the task, and the grader is verified against it, passing the reference and failing broken attempts.
Because the repositories are private, tasks can be checked against public repositories and benchmarks for leakage. That matters more every year. Popular public benchmarks are increasingly contaminated, and scores on them say less about whether a model can do real work. You can read more about how we build these in our private code vertical.
05How to evaluate a codebase dataset
If you are sourcing code data, from us or anyone else, these are the questions that separate useful data from expensive noise:
- Is history included, or only snapshots? Without commits and review threads you are buying source files you could approximate with public code.
- What is the depth distribution? A collection of 5,000 shallow repositories teaches less than 500 deep ones. Ask for commit counts and years active per repository.
- Are tests tracked separately? Source-to-test ratios tell you which repositories can support verifiable tasks.
- What is the language mix? Ask for a percentage breakdown, not a list of languages that appear somewhere.
- How were rights obtained? Consent and a contract per source, or a terms-of-service interpretation?
- What was removed, and how was it checked? Secrets, credentials and personal data in fixtures and logs.
Every repository we deliver carries the same metadata fields, so the collection can be filtered by language, depth, activity and test coverage before you commit. A sample under NDA is the fastest way to check whether it fits your training plan.
