Why the AI project is central to AI success
Steve Sperandeo ·

Watch the six-minute film
Nobody can promise that an AI project will work. The model may not learn the pattern you need. Your records may not contain the information it would have to learn from. The world the model meets in use may differ from the world it was trained on. What you can do is manage the ways a project fails, one at a time, and every failure you manage raises the odds that the project succeeds.
That is the whole job of AI implementation, and it is why the AI project matters more than any model inside it. In Orama Lake, the AI project is where that job is done.
This article answers two questions. Why does the data project, more than the model, decide whether AI succeeds? And what must a project hold to manage the ways data projects fail?
The data project is the AI
In AI for Everyone, Andrew Ng explains most of today's AI with one idea, supervised learning, which "just learns input, output, or A to B mappings."1 A house's size and number of bedrooms go in; its price comes out. A chest X-ray goes in; a diagnosis comes out. The system learns the mapping from examples, and "data means both the input A and the output B."2 A large language model works the same way at vast scale: it learns to predict the next word.
Every useful AI system, then, is a data project underneath: the right inputs, the right outputs, enough of both, and a way to check that what the system learned holds up. Without the data, a model has nothing to learn from.
Models, meanwhile, are getting easier to come by. In March 2026, the top four models on the public Arena leaderboard were separated by fewer than 25 points; a year earlier, the spread was about 97.3 Anyone can rent the same frontier model. What sets one organization's AI apart is less often its model than whether its data project worked.
Engineers have said so for a decade. In 2015, a team at Google wrote that the machine learning code is "a small fraction" of a real machine learning system, and that a mature system "might end up being (at most) 5% machine learning code and (at least) 95% glue code."4 The rest collects the data, checks it, moves it, serves the results, and monitors them. In 2021, Ng named the idea data-centric AI, which he defined as "the discipline of systematically engineering the data needed to successfully build an AI system." For many practical applications, he argued, "it's now more productive to hold the neural network architecture fixed," and to improve the data instead.5
The model still matters on the hardest problems. For most problems a business brings to AI, though, the model is the part you rent. The data project is the part you run.
How data projects fail
Andrew Ng states the starting point plainly in AI for Everyone: every successful data project starts with two things, a real problem to solve and the right data to solve it.6 In a talk in April, I added what follows from that. "Not all data. Not more data. The right data. A terabyte of the wrong data is worthless."7 Most failed projects start from the other end: "We have data, what can we do with it?" or "We have AI, where should we apply it?" The team spends months building infrastructure and collecting everything it can, then goes looking for insights. It finds noise. The project is shelved, and AI fatigue sets in.
Research says the same in more careful terms. In 2024, RAND interviewed 65 experienced data scientists and engineers about why AI projects fail and identified five root causes.8 The people involved misunderstand, or fail to communicate, the problem the AI is meant to solve. The organization "lacks the necessary data to adequately train an effective AI model." The work focuses "more on using the latest and greatest technology than on solving real problems." The organization lacks the infrastructure "to manage their data and deploy completed AI models." And the technology is applied "to problems that are too difficult for AI to solve."
Those causes appear at the start. Others surface later.
Bad data spreads. Researchers at Google interviewed 53 practitioners building high-stakes AI and took their title from one of them: "Everyone wants to do the model work, not the data work." Small problems in the data compounded into what the authors called data cascades, and 92% of the practitioners had lived through at least one.9
The score lies. When test data leaks into training data, a model looks better than it is. Sayash Kapoor and Arvind Narayanan found leakage in 294 papers across 17 fields. In civil war prediction, complex models were believed to beat simple ones by a wide margin; once the errors were corrected, they did not perform substantively better than logistic regression, a method decades old.10
The world moves. A model learns the world its data came from. In 2018, John Zech and his colleagues trained a pneumonia detector on chest X-rays from two hospital systems. On new X-rays from those hospitals, it scored an area under the curve of 0.931; on X-rays from a third hospital, 0.815. It may also have learned the hospitals themselves. The model could tell which hospital an X-ray came from, and the hospitals saw very different rates of pneumonia: sorting the images by hospital alone, without looking for pneumonia at all, scored 0.861.11 A sepsis model in use at hundreds of American hospitals scored 0.63 when Michigan Medicine tested it on 38,455 of its own hospital stays. It missed 67% of the patients who had sepsis.12

The steps don't repeat. A language model can give different answers to the same question. Thinking Machines put one prompt to a large language model 1,000 times at temperature 0, the setting meant to make its output deterministic, and got 80 different completions.13 A workflow that lets a model choose each step cannot promise the same result twice.
The work can't be found again. Researchers tried to rerun 863,878 Jupyter notebooks from public repositories. Only 24.11% ran without errors, and only 4.03% produced the same results.14 Nobody can build on a result that nobody can reproduce.
And these failures stay hidden. The news covers AI's successes, Ng tells his students, and "we hardly ever see any failure stories because they just aren't as interesting to report on." For someone who works on one new AI project a year, he adds, "to see three examples would take you three years."1516 So most organizations learn how data projects fail the slow way: by running one.

Manage the failure, and success follows
Since no one can promise success, ask a more useful question: what makes data projects fail, and what stops each cause?
Ng's advice points the same way. Before committing to a project, he says, "I'll usually have either myself or engineers do technical diligence on a project to make sure that it is feasible."2 His playbook for companies new to AI says "it is more important for your first few AI projects to succeed rather than be the most valuable AI projects."17 In 2020, he wrote that "we as a community need to get much better at bridging proofs of concept (PoCs) and production deployments."18 And in 2025, writing about AI agents, he named "the single biggest predictor of how rapidly a team makes progress": a disciplined process for evaluations and error analysis.19 Each piece of advice moves attention from the model to the work around it.
Process documents have tried to carry this discipline for decades. CRISP-DM and its successors lay out the stages of a data project. Yet when 237 data science professionals were surveyed in 2021, only 25% said they followed any project methodology.20 A practice written in a document works only when people remember it. A practice built into the place where the work runs happens by default.
The AI project is that place. Because the work runs there, the ways it fails can be managed there.
What an AI project is
In Orama Lake, the AI project is the main container. It holds:
- the data it draws on, in its workspace;
- its execution graphs, which Lake calls pipelines: steps joined by arrows that connect to data, clean it, train models, and index, serve, and act on the results;21
- many models, each a step in a graph, working toward the project's goal;22
- every run of every graph, with its report, lineage (where the data came from), outputs, log, and trace;23
- every version of every model, with its score on data it never saw, and experiments that try many settings and keep the best.2223

If you have built workflows in n8n, the execution graph will look familiar. n8n calls itself "the platform for agents and workflows you can follow," and it lives up to the name.24 Lake can import an n8n workflow export as a pipeline, and it can do the same with a pipeline written in Python, which it reads but never runs.21 A graph you can follow is the right shape for this work. The AI project keeps that graph beside the data it reads, the models it trains, and the record of every run.
Many models, one goal
A project usually holds more than one model, because a business problem rarely comes down to a single prediction. One model might sort customers into segments; another, trained within each segment, might predict which customers will leave.22 Each model does one job that can be checked on its own, and together they serve one goal.
The case for combining models is old and strong. In the M4 forecasting competition, which tested 61 methods on 100,000 time series, "all of the top-performing methods" were combinations.25 Researchers at Berkeley see the same shift in today's AI, from single models to "compound AI systems" that tackle tasks "using multiple interacting components," including "multiple calls to models, retrievers, or external tools."26 An ensemble needs a home that keeps its models, the graph that joins them, and the measurements that judge them together. In Lake, that home is the project.
Each failure, and what stops it
Here is how the project answers each way a data project fails. Lake does everything in the right-hand column today.212327
| How a data project fails | What the AI project does |
|---|---|
| The problem is unclear. No one has said what goes in and what should come out. | A pipeline starts from a stated question: What should it predict or answer? You name the dataset and the column to predict, so A and B are written down before anything trains. |
| The data is wrong for the job. | Auto-Suggest reads the workspace's datasets and recommends projects the data can support. A canned project names the columns it needs, including those that rule an item out; if one is missing, the run is blocked rather than the value assumed false. |
| The problem is too hard for the method. | The engine refuses a method it can't run and names the issue that tracks it. A pipeline whose method doesn't train yet says This Pipeline Trains Nothing Yet. |
| The score lies. | A training run holds back a share of the data (a fifth, by default), and its report leads with the score on rows the model never saw. A run without a hold-out says so. |
| The steps don't repeat. | The graph is fixed. It runs only after approval, and any change needs approval again. The engine checks the graph before every run, and a failed check stops the run. Agents that help design a graph only propose changes; a person applies them. |
| The work can't be found again. | Every run keeps its lineage, outputs, log, and trace. Every model keeps its versions, with each version's held-out score charted, so you can see whether the newest is better or worse. |
| The world moves. | Retraining adds a new version of the model, scored on held-out data like the last, so a decline shows in the comparison. A canned solution also monitors what it delivers. |
Each row closes one way a project can fail, and the rows add up.
A word on approval. It means the team reads the graph before it runs, the way a colleague reviews a change to code before it ships. Unlike a model's answer, a graph can be read, checked, and approved once, then run the same way every time. That is how a probabilistic model can work inside a process that must be repeatable: the model makes predictions, and the approved graph around it decides what happens to them.
The AI solution: a canned data project
If every project meets the same failures, it helps to start from one that has already met them.
An AI solution in Lake is a canned data project. Orama's catalogue holds solutions to common problems, each built from the modelling methods it combines, with levels that set how far it goes.27 When you start one, Lake creates an AI project in your workspace with its pipeline already designed. Before anything else is created, the Project Agent reads your data and proposes the foundation the solution will run on: the snapshot date, what each row describes, the column to predict, the columns to learn from, and the conditions that rule an item out. You review the proposal, correct it, and confirm it.27
A canned project carries what someone learned the first time: the data the problem needs, the methods that work, and how to hold out data to measure the result. It is one way to follow Ng's advice and make your first projects ones likely to succeed.
It still runs on your data. Zech's pneumonia model slipped on X-rays from another hospital, and a packaged sepsis model used at hundreds of hospitals did poorly on Michigan's patients. A canned project in Lake keeps its own holdout and is measured on your data, and if your data lacks a condition the solution needs, the run stays blocked until the data supplies it.27 The template brings the design. Your data must earn the result.
You can also write your own solutions, or duplicate one of Orama's and adapt it, so each new project in your organization starts from what the last one learned.27
The test
Here is a test for any AI effort, in Lake or anywhere else. Ask where its project lives: the one place that holds its data, workflows, models, runs, and scores. Then go down the list of ways data projects fail, and for each one, ask what in that project stops it.
If every row has an answer, the effort has a real chance. If the project lives in a notebook, a slide deck, and someone's memory, it is running on hope.
Notes
- Andrew Ng, AI for Everyone, week 1, "Machine learning" (DeepLearning.AI and Coursera). ↩
- Ng, AI for Everyone, week 1, "What machine learning can and cannot do." ↩
- Stanford HAI, The 2026 AI Index Report, chapter 2, "Technical Performance." ↩
- Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (2015), figure 1. ↩
- Eliza Strickland, "Andrew Ng: Unbiggen AI," IEEE Spectrum, February 9, 2022. ↩
- Andrew Ng, AI for Everyone (DeepLearning.AI and Coursera), as I quoted him in "AI Agents: From Theory to Implementation," a talk given April 16, 2026. ↩
- Steve Sperandeo, "AI Agents: From Theory to Implementation," a talk given April 16, 2026, section I, "Data Project Success and Data Project Failure." ↩
- Ryseff, De Bruhl, and Newberry, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (RAND, 2024), summary. ↩
- Sambasivan et al., "'Everyone Wants to Do the Model Work, Not the Data Work': Data Cascades in High-Stakes AI" (2021). ↩
- Kapoor and Narayanan, "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science" (2023), abstract. ↩
- Zech et al., "Variable Generalization Performance of a Deep Learning Model to Detect Pneumonia in Chest Radiographs" (2018), abstract. ↩
- Wong et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients" (2021), abstract. ↩
- Horace He and Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference," September 10, 2025. ↩
- Pimentel et al., "A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks" (2019). ↩
- Ng, AI for Everyone, week 1, "Week 1 introduction." ↩
- Ng, AI for Everyone, week 1, "More examples of what machine learning can and cannot do." ↩
- Landing AI, AI Transformation Playbook, step 1, "Execute pilot projects to gain momentum." ↩
- Andrew Ng, "From Proof of Concept to Production," The Batch, May 6, 2020. ↩
- Andrew Ng, "Improve Agentic Performance with Evals and Error Analysis, Part 1," The Batch, October 15, 2025. ↩
- Martinez, Viles, and Olaizola, "A Survey Study of Success Factors in Data Science Projects" (2021). ↩
- Orama Lake documentation, "AI Projects and Pipelines," lake v0.9.0. ↩
- Orama Lake documentation, "Concepts and Glossary," lake v0.9.0. ↩
- Orama Lake documentation, "Train a Model," lake v0.9.0. ↩
- n8n, home page, read November 19, 2026. ↩
- Makridakis, Spiliotis, and Assimakopoulos, "The M4 Competition: 100,000 Time Series and 61 Forecasting Methods" (2020), finding 1. ↩
- Zaharia et al., "The Shift from Models to Compound AI Systems," Berkeley Artificial Intelligence Research (BAIR), February 18, 2024. ↩
- Orama Lake documentation, "AI Solutions, Methodologies and Opportunities," lake v0.9.0. ↩
Bibliography
- He, Horace, and Thinking Machines Lab. 2025. "Defeating Nondeterminism in LLM Inference." Thinking Machines Lab, September 10, 2025. thinkingmachines.ai, the post. Accessed 2026-11-19.
- Kapoor, Sayash, and Arvind Narayanan. 2023. "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns 4 (9): 100804. https://doi.org/10.1016/j.patter.2023.100804.
- Landing AI. n.d. AI Transformation Playbook: How to Lead Your Company into the AI Era. https://landing.ai/wp-content/uploads/2020/05/LandingAI_Transformation_Playbook_11-19.pdf. Accessed 2026-11-19.
- Makridakis, Spyros, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2020. "The M4 Competition: 100,000 Time Series and 61 Forecasting Methods." International Journal of Forecasting 36 (1): 54–74. https://doi.org/10.1016/j.ijforecast.2019.04.014.
- Martinez, Iñigo, Elisabeth Viles, and Igor G. Olaizola. 2021. "A Survey Study of Success Factors in Data Science Projects." arXiv:2201.06310. https://arxiv.org/abs/2201.06310.
- n8n. 2026. "n8n." https://n8n.io/. Accessed 2026-11-19.
- Ng, Andrew. 2020. "From Proof of Concept to Production." The Batch, DeepLearning.AI, May 6, 2020. https://www.deeplearning.ai/the-batch/from-proof-of-concept-to-production. Accessed 2026-11-19.
- Ng, Andrew. 2025. "Improve Agentic Performance with Evals and Error Analysis, Part 1." The Batch, DeepLearning.AI, October 15, 2025. https://www.deeplearning.ai/the-batch/improve-agentic-performance-with-evals-and-error-analysis-part-1. Accessed 2026-11-19.
- Ng, Andrew. n.d. AI for Everyone. DeepLearning.AI and Coursera. https://www.coursera.org/learn/ai-for-everyone.
- OramaSearch, Inc. 2026a. "AI Projects and Pipelines." `docs/guide/design.md`, lake v0.9.0. GitHub. https://github.com/oramasearch/lake/blob/v0.9.0/docs/guide/design.md.
- OramaSearch, Inc. 2026b. "AI Solutions, Methodologies and Opportunities." `docs/guide/solutions.md`, lake v0.9.0. GitHub. https://github.com/oramasearch/lake/blob/v0.9.0/docs/guide/solutions.md.
- OramaSearch, Inc. 2026c. "Concepts and Glossary." `docs/guide/concepts.md`, lake v0.9.0. GitHub. https://github.com/oramasearch/lake/blob/v0.9.0/docs/guide/concepts.md.
- OramaSearch, Inc. 2026d. "Train a Model." `docs/guide/train.md`, lake v0.9.0. GitHub. https://github.com/oramasearch/lake/blob/v0.9.0/docs/guide/train.md.
- Pimentel, João Felipe, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019. "A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks." https://leomurta.github.io/papers/pimentel2019a.pdf. Accessed 2026-11-19.
- Ryseff, James, Brandon F. De Bruhl, and Sydne J. Newberry. 2024. The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Lessons from Interviews with Data Scientists and Engineers. RR-A2680-1. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/research_reports/RRA2680-1.html.
- Sambasivan, Nithya, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M. Aroyo. 2021. "'Everyone Wants to Do the Model Work, Not the Data Work': Data Cascades in High-Stakes AI." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM. https://doi.org/10.1145/3411764.3445518.
- Sculley, D., Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. "Hidden Technical Debt in Machine Learning Systems." In Advances in Neural Information Processing Systems 28. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf.
- Sperandeo, Steve. 2026. "AI Agents: From Theory to Implementation." Talk, April 16, 2026.
- Stanford Institute for Human-Centered Artificial Intelligence (HAI). 2026. The 2026 AI Index Report. Chapter 2, "Technical Performance." https://hai.stanford.edu/assets/files/ai_index_report_2026_chapter_2_technical.pdf. Accessed 2026-11-19.
- Strickland, Eliza. 2022. "Andrew Ng: Unbiggen AI." IEEE Spectrum, February 9, 2022. https://spectrum.ieee.org/andrew-ng-data-centric-ai.
- Wong, Andrew, E. Otles, J. P. Donnelly, A. Krumm, J. McCullough, O. DeTroyer-Cooley, J. Pestrue, M. Phillips, J. Konye, C. Penoza, M. Ghous, and K. Singh. 2021. "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients." JAMA Internal Medicine 181 (8): 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626.
- Zaharia, Matei, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. 2024. "The Shift from Models to Compound AI Systems." Berkeley Artificial Intelligence Research (BAIR), February 18, 2024. BAIR, the post.
- Zech, John R., M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann. 2018. "Variable Generalization Performance of a Deep Learning Model to Detect Pneumonia in Chest Radiographs: A Cross-Sectional Study." PLOS Medicine 15 (11): e1002683. https://doi.org/10.1371/journal.pmed.1002683.
