Your model passes every test in CI. The pipeline glows green. You merge to trunk, deploy to production, and within 48 hours, prediction accuracy drops 12%. Your monitoring finally catches it, but thousands of users have already received degraded recommendations. The code didn't change. The tests didn't fail. The infrastructure held steady. What happened?
This is the trap of adopting trunk-based development (TBD) for ML systems without rethinking what "production-ready" means. CTOs embrace trunk based development because it works brilliantly for traditional software—faster feedback loops, fewer merge conflicts, continuous deployment. But ML systems operate under different rules. Speed without the right safety infrastructure doesn't make you faster; it makes you fragile.
Why TBD's Core Assumption Collapses for ML
Trunk-based development rests on one powerful assumption: if your code passes tests, it's safe to ship. This works because software correctness is largely deterministic. A sorting algorithm either works or it doesn't. A database query either returns the right rows or not. Tests catch failures.
ML systems shatter this assumption entirely. A model can pass validation, achieve high accuracy on test data, and fail silently in production because real-world data has shifted. It can make confident predictions on inputs it's never encountered. It can degrade as upstream feature pipelines introduce subtle changes. None of this triggers a test failure or stops your CI/CD pipeline.
This is concept drift, data drift, and feature entanglement—three failure modes that traditional testing frameworks don't detect. When you combine these risks with trunk-based velocity, you've built a high-speed pipeline for high-risk deployments, disguised as mature engineering.
The problem isn't TBD itself. Most organizations adopting TBD for ML treat it as a process change rather than an architectural requirement. They inherit the speed benefits without building the infrastructure needed to make speed safe.
Last Time the Market Was This Expensive, Investors Waited 14 Years to Break Even
In 1999, the S&P 500 peaked. Then it took 14 years to gradually recover by 2013.
Today? Goldman Sachs sounds crazy forecasting 3% returns for 2024 to 2034.
But we’re currently seeing the highest price for the S&P 500 compared to earnings since the dot-com boom.
So, maybe that’s why they’re not alone; Vanguard projects about 5%.
In fact, now just about everything seems priced near all time highs. Equities, gold, crypto, etc.
But billionaires have long diversified a slice of their portfolios with one asset class that is poised to rebound.
It’s post war and contemporary art.
Sounds crazy, but over 70,000 investors have followed suit since 2019—with Masterworks.
You can invest in shares of artworks featuring Banksy, Basquiat, Picasso, and more.
24 exits later, results speak for themselves: net annualized returns like 14.6%, 17.6%, and 17.8%.*
My subscribers can skip the waitlist.
*Investing involves risk. Past performance is not indicative of future returns. Important Reg A disclosures: masterworks.com/cd.
Risk 1: Silent Model Degradation
Your model was trained on six months of historical data and performs beautifully in offline evaluation. Then user behavior shifts. Seasonality changes. A competitor launches a feature that affects your input distribution. Your model keeps making predictions—confidently and incorrectly—and nothing in your CI/CD pipeline notices.
This happens because validation tests measure model performance on static test sets, not on live data. Traditional software fails loudly in production; logs capture errors and alarms fire. ML models fail silently. They serve bad predictions for days or weeks while infrastructure health looks perfect.
A common anti-pattern I’ve observed in the industry involves relying too heavily on initial validation. For example, in fintech, it’s entirely possible for a credit model to test at 90% accuracy but fail silently due to concept drift. If the team lacks continuous monitoring, they might miss that the market has shifted, leading to weeks of unreliable predictions despite the application technically staying 'up'.
Risk 2: Feature-Model Entanglement
Your data engineering team refactors a feature transformation pipeline to improve performance. The change is backward-compatible at the API level—the output schema is identical. But the feature's statistical properties have shifted subtly. Maybe normalization changed. Maybe a bucketing strategy shifted. Maybe null handling is different.
Your models load successfully and run without errors. But they're making predictions based on feature distributions they were never trained on. Unlike code dependencies, where breaking changes surface as runtime errors, model dependencies fail silently. Performance drops 7%, and your monitoring system takes two days to flag it.
Risk 3: Reproducibility Collapse
High-velocity deployment destroys your ability to trace cause and effect. Which training data version produced model v47? Which hyper-parameters? When a model fails, your team asks: was it a code change, a data change, or model retraining?
At high velocity with TBD, you've got dozens of models in flight, multiple data pipelines feeding them, and feature code changing constantly. You cannot answer "what changed?" because too many things changed simultaneously.
This is more than operational frustration. It's a compliance nightmare. Regulated industries require audit trails. Your auditors want to know: when this model made this decision for this customer, what data was it trained on? If you can't answer that, you have a regulatory problem hiding beneath TBD's speed gains.
What Traditional TBD Doesn't Cover
Trunk-based development assumes:
Tests verify correctness – Passing tests mean code is production-ready
A single deployment axis – Change code, deploy code, system behaves correctly
Deterministic failure modes – Bugs manifest as errors, crashes, or incorrect outputs that are detectable
ML systems operate on three axes simultaneously:
Code changes – Model inference code, feature engineering logic, serving infrastructure
Model changes – New model versions, different hyper-parameters, alternative architectures
Data changes – Training data distributions, feature distributions, input characteristics, pipeline logic
A change on any axis can degrade model performance in production without triggering any test failure. Traditional TBD's safety guarantees don't extend to any of this.
From the Trenches: What "Safe TBD for ML" Actually Requires
Organizations that successfully run TBD for ML at scale build an additional operational stack that traditional software teams don't need.
Layer 1: Feature Flags and Traffic Control
You cannot deploy a model to 100% of traffic and monitor it. You need gradual rollout with traffic control separate from code deployment:
Use a feature management platform (Unleash, LaunchDarkly, Growthbook) to route traffic to model versions independently of code deployment
Run model A for 10% of users and model B for 90% while both are deployed to the same infrastructure
Kill switches that disable a model in seconds without redeploying code
You can deploy silently, run it on a canary cohort, and observe behavior before routing more traffic to it.
# Feature flag-driven model selection
def get_prediction(user_id, features):
if feature_flag.is_enabled("model_v2_canary", user_id):
model_version = "v2_canary"
model = load_model("model_v2")
else:
model_version = "v1_stable"
model = load_model("v1")
prediction = model.predict(features)
log_prediction_event(user_id, model_version, prediction, features)
return prediction
This pattern lets you deploy new models constantly while minimizing risk.
Layer 2: Continuous Model Monitoring
Application monitoring tools track latency, error rates, and throughput. These are insufficient for models. You need model-specific observability:
Prediction accuracy – Compare predictions against ground truth as labels arrive (hours to weeks after prediction)
Feature drift – Monitor input distributions continuously; alert when features deviate from training distribution
Confidence scores – Track prediction confidence spread; increasing uncertainty signals concept drift
Prediction distribution – Does your model suddenly predict 80% positive when it historically predicted 20%?
Latency by cohort – Does prediction latency increase for specific user segments or feature value ranges?
These metrics require instrumentation separate from application monitoring.
# Log prediction event with monitoring context
def log_prediction_for_monitoring(user_id, features, prediction, model_version):
event = {
"timestamp": datetime.now(),
"user_id": user_id,
"model_version": model_version,
"prediction": prediction,
"feature_values": features,
"confidence": prediction.confidence,
}
monitoring_client.log_event(event)
Your monitoring system then computes metrics: prediction accuracy by cohort, feature distribution statistics compared to training distribution, and alerts on sudden shifts.
Layer 3: Data Versioning and Model Registry
You need to know, for any production model, what data it was trained on and what code was running. This requires:
Model registry – Central tracking of every model version, performance metrics, training date, and lineage (MLflow, Hugging Face Model Hub)
Data versioning – Which dataset snapshot was used for training? (DVC, Pachyderm, Delta Lake)
Experiment tracking – Which hyperparameters produced which model? (Weights & Biases, MLflow, Neptune)
Feature versioning – What was the schema and transformation logic for features at training time?
When a model degrades, you diagnose whether it was a data change, feature change, or concept drift.
# Model registry entry
model_v47:
created: 2024-01-15T14:32:00Z
training_dataset: "training_data_v23"
code_commit: "abc1234def567"
feature_schema_version: "v5"
hyperparameters:
learning_rate: 0.001
max_depth: 8
performance_metrics:
accuracy: 0.923
deployed_to_production: 2024-01-16T09:00:00Z
canary_traffic_percent: 5
Layer 4: The Operational Cost
Most CTOs underestimate this infrastructure stack's complexity. It's roughly 2-3x more complex than running TBD for traditional software. You manage:
Feature flag infrastructure and policies
Model serving infrastructure (distinct from application servers)
Continuous monitoring pipelines (data drift detection, accuracy tracking, cohort analysis)
Model registries and experiment tracking systems
Data versioning infrastructure
Automated retraining pipelines
Audit logging for regulatory compliance
This isn't optional. Without this stack, you're not actually running TBD safely—you're running a high-velocity deployment pipeline with low visibility into model behavior.
The Real Question
Here's what you must decide as an Engineering Leader: Is the speed gain from trunk-based development worth the infrastructure investment required to make it safe for ML?
The honest answer: it depends on your risk tolerance and business model. For real-time recommendations, dynamic pricing, and personalization, the speed benefit may justify the infrastructure investment because slowness costs more than calculated risk. For lending decisions, medical diagnostics, and safety-critical systems, TBD may not be appropriate at all.
If you do decide that TBD is right for your organization, commit to building the infrastructure. Don't adopt TBD for speed alone. Adopt it because you've deliberately built the monitoring, versioning, feature management, and observability layers that make it safe.
The trap isn't TBD itself. The trap is adopting the speed benefits without accepting the infrastructure costs. Do one or the other—but not the speed without the safety.