Why the AI project is central to AI success

Steve Sperandeo ·

On the left, the loose pieces of an AI effort drift apart: a notebook, two charts, a slide, and two steps whose link has broken. A dashed path carries them into one window named ai-project, where the work sits in place: a graph of six steps, three models feeding a fourth that feeds the goal, and a stack of runs, each with its check.

Watch the six-minute film

Nobody can promise that an AI project will work. The model may not learn the pattern you need. Your records may not contain the information it would have to learn from. The world the model meets in use may differ from the world it was trained on. What you can do is manage the ways a project fails, one at a time, and every failure you manage raises the odds that the project succeeds.

That is the whole job of AI implementation, and it is why the AI project matters more than any model inside it. In Orama Lake, the AI project is where that job is done.

This article answers two questions. Why does the data project, more than the model, decide whether AI succeeds? And what must a project hold to manage the ways data projects fail?

The data project is the AI

In AI for Everyone, Andrew Ng explains most of today's AI with one idea, supervised learning, which "just learns input, output, or A to B mappings."1 A house's size and number of bedrooms go in; its price comes out. A chest X-ray goes in; a diagnosis comes out. The system learns the mapping from examples, and "data means both the input A and the output B."2 A large language model works the same way at vast scale: it learns to predict the next word.

Every useful AI system, then, is a data project underneath: the right inputs, the right outputs, enough of both, and a way to check that what the system learned holds up. Without the data, a model has nothing to learn from.

Models, meanwhile, are getting easier to come by. In March 2026, the top four models on the public Arena leaderboard were separated by fewer than 25 points; a year earlier, the spread was about 97.3 Anyone can rent the same frontier model. What sets one organization's AI apart is less often its model than whether its data project worked.

Engineers have said so for a decade. In 2015, a team at Google wrote that the machine learning code is "a small fraction" of a real machine learning system, and that a mature system "might end up being (at most) 5% machine learning code and (at least) 95% glue code."4 The rest collects the data, checks it, moves it, serves the results, and monitors them. In 2021, Ng named the idea data-centric AI, which he defined as "the discipline of systematically engineering the data needed to successfully build an AI system." For many practical applications, he argued, "it's now more productive to hold the neural network architecture fixed," and to improve the data instead.5

The model still matters on the hardest problems. For most problems a business brings to AI, though, the model is the part you rent. The data project is the part you run.

How data projects fail

Andrew Ng states the starting point plainly in AI for Everyone: every successful data project starts with two things, a real problem to solve and the right data to solve it.6 In a talk in April, I added what follows from that. "Not all data. Not more data. The right data. A terabyte of the wrong data is worthless."7 Most failed projects start from the other end: "We have data, what can we do with it?" or "We have AI, where should we apply it?" The team spends months building infrastructure and collecting everything it can, then goes looking for insights. It finds noise. The project is shelved, and AI fatigue sets in.

Research says the same in more careful terms. In 2024, RAND interviewed 65 experienced data scientists and engineers about why AI projects fail and identified five root causes.8 The people involved misunderstand, or fail to communicate, the problem the AI is meant to solve. The organization "lacks the necessary data to adequately train an effective AI model." The work focuses "more on using the latest and greatest technology than on solving real problems." The organization lacks the infrastructure "to manage their data and deploy completed AI models." And the technology is applied "to problems that are too difficult for AI to solve."

Those causes appear at the start. Others surface later.

Bad data spreads. Researchers at Google interviewed 53 practitioners building high-stakes AI and took their title from one of them: "Everyone wants to do the model work, not the data work." Small problems in the data compounded into what the authors called data cascades, and 92% of the practitioners had lived through at least one.9

The score lies. When test data leaks into training data, a model looks better than it is. Sayash Kapoor and Arvind Narayanan found leakage in 294 papers across 17 fields. In civil war prediction, complex models were believed to beat simple ones by a wide margin; once the errors were corrected, they did not perform substantively better than logistic regression, a method decades old.10

The world moves. A model learns the world its data came from. In 2018, John Zech and his colleagues trained a pneumonia detector on chest X-rays from two hospital systems. On new X-rays from those hospitals, it scored an area under the curve of 0.931; on X-rays from a third hospital, 0.815. It may also have learned the hospitals themselves. The model could tell which hospital an X-ray came from, and the hospitals saw very different rates of pneumonia: sorting the images by hospital alone, without looking for pneumonia at all, scored 0.861.11 A sepsis model in use at hundreds of American hospitals scored 0.63 when Michigan Medicine tested it on 38,455 of its own hospital stays. It missed 67% of the patients who had sepsis.12

One pneumonia model, three scores (AUC): 0.931 on new X-rays from the two hospital systems it learned from, 0.815 on X-rays from a third hospital system, and 0.861 for sorting the images by hospital alone, without looking for pneumonia.
Area under the curve (AUC) of one model trained on chest X-rays from two hospital systems; 0.5 is chance and 1.0 is perfect. Source: Zech et al., PLOS Medicine 15 (11), 2018, abstract. Internal: Mount Sinai and NIH combined. External: Indiana University. Hospital sorting: on the joint Mount Sinai and NIH data.

The steps don't repeat. A language model can give different answers to the same question. Thinking Machines put one prompt to a large language model 1,000 times at temperature 0, the setting meant to make its output deterministic, and got 80 different completions.13 A workflow that lets a model choose each step cannot promise the same result twice.

The work can't be found again. Researchers tried to rerun 863,878 Jupyter notebooks from public repositories. Only 24.11% ran without errors, and only 4.03% produced the same results.14 Nobody can build on a result that nobody can reproduce.

And these failures stay hidden. The news covers AI's successes, Ng tells his students, and "we hardly ever see any failure stories because they just aren't as interesting to report on." For someone who works on one new AI project a year, he adds, "to see three examples would take you three years."1516 So most organizations learn how data projects fail the slow way: by running one.

Six cards, the known places a data project fails: 1, the problem: nobody wrote down what goes in and what should come out; 2, the data: the right data was never there or was never checked; 3, the score: test data leaked into training and the score flattered the model; 4, the world: the data moved on and the model did not; 5, the steps: a model decided each step and decided differently each time; 6, the record: nobody can find the work again or run it again.
A data project fails in known places, and each one can be managed, one at a time. Sources: RAND, Ryseff et al. (2024); Sambasivan et al. (2021); Kapoor and Narayanan (2023); Zech et al. (2018); Wong et al. (2021); Thinking Machines (2025); Pimentel et al. (2019).

Manage the failure, and success follows

Since no one can promise success, ask a more useful question: what makes data projects fail, and what stops each cause?

Ng's advice points the same way. Before committing to a project, he says, "I'll usually have either myself or engineers do technical diligence on a project to make sure that it is feasible."2 His playbook for companies new to AI says "it is more important for your first few AI projects to succeed rather than be the most valuable AI projects."17 In 2020, he wrote that "we as a community need to get much better at bridging proofs of concept (PoCs) and production deployments."18 And in 2025, writing about AI agents, he named "the single biggest predictor of how rapidly a team makes progress": a disciplined process for evaluations and error analysis.19 Each piece of advice moves attention from the model to the work around it.

Process documents have tried to carry this discipline for decades. CRISP-DM and its successors lay out the stages of a data project. Yet when 237 data science professionals were surveyed in 2021, only 25% said they followed any project methodology.20 A practice written in a document works only when people remember it. A practice built into the place where the work runs happens by default.

The AI project is that place. Because the work runs there, the ways it fails can be managed there.

What an AI project is

In Orama Lake, the AI project is the main container. It holds:

  • the data it draws on, in its workspace;
  • its execution graphs, which Lake calls pipelines: steps joined by arrows that connect to data, clean it, train models, and index, serve, and act on the results;21
  • many models, each a step in a graph, working toward the project's goal;22
  • every run of every graph, with its report, lineage (where the data came from), outputs, log, and trace;23
  • every version of every model, with its score on data it never saw, and experiments that try many settings and keep the best.2223
An AI project drawn as one container with four parts: data (accounts, support tickets, usage); an execution graph of six steps, connect, clean, model, index, serve, and act; many models, one goal, where a model that segments the customers feeds a model that predicts who will leave; and the record of every run: every run, held-out scores, model versions, and lineage.
An AI project holds the whole data project in one place: its data, its execution graphs, the models that work together toward one goal, and the record of every run. Source: Orama Lake documentation, lake v0.9.0 (AI Projects and Pipelines; Concepts and Glossary; Train a Model). Dataset names are illustrative.

If you have built workflows in n8n, the execution graph will look familiar. n8n calls itself "the platform for agents and workflows you can follow," and it lives up to the name.24 Lake can import an n8n workflow export as a pipeline, and it can do the same with a pipeline written in Python, which it reads but never runs.21 A graph you can follow is the right shape for this work. The AI project keeps that graph beside the data it reads, the models it trains, and the record of every run.

Many models, one goal

A project usually holds more than one model, because a business problem rarely comes down to a single prediction. One model might sort customers into segments; another, trained within each segment, might predict which customers will leave.22 Each model does one job that can be checked on its own, and together they serve one goal.

The case for combining models is old and strong. In the M4 forecasting competition, which tested 61 methods on 100,000 time series, "all of the top-performing methods" were combinations.25 Researchers at Berkeley see the same shift in today's AI, from single models to "compound AI systems" that tackle tasks "using multiple interacting components," including "multiple calls to models, retrievers, or external tools."26 An ensemble needs a home that keeps its models, the graph that joins them, and the measurements that judge them together. In Lake, that home is the project.

Each failure, and what stops it

Here is how the project answers each way a data project fails. Lake does everything in the right-hand column today.212327

Each way a data project fails, and what the AI project does about it
How a data project failsWhat the AI project does
The problem is unclear. No one has said what goes in and what should come out.A pipeline starts from a stated question: What should it predict or answer? You name the dataset and the column to predict, so A and B are written down before anything trains.
The data is wrong for the job.Auto-Suggest reads the workspace's datasets and recommends projects the data can support. A canned project names the columns it needs, including those that rule an item out; if one is missing, the run is blocked rather than the value assumed false.
The problem is too hard for the method.The engine refuses a method it can't run and names the issue that tracks it. A pipeline whose method doesn't train yet says This Pipeline Trains Nothing Yet.
The score lies.A training run holds back a share of the data (a fifth, by default), and its report leads with the score on rows the model never saw. A run without a hold-out says so.
The steps don't repeat.The graph is fixed. It runs only after approval, and any change needs approval again. The engine checks the graph before every run, and a failed check stops the run. Agents that help design a graph only propose changes; a person applies them.
The work can't be found again.Every run keeps its lineage, outputs, log, and trace. Every model keeps its versions, with each version's held-out score charted, so you can see whether the newest is better or worse.
The world moves.Retraining adds a new version of the model, scored on held-out data like the last, so a decline shows in the comparison. A canned solution also monitors what it delivers.

Each row closes one way a project can fail, and the rows add up.

A word on approval. It means the team reads the graph before it runs, the way a colleague reviews a change to code before it ships. Unlike a model's answer, a graph can be read, checked, and approved once, then run the same way every time. That is how a probabilistic model can work inside a process that must be repeatable: the model makes predictions, and the approved graph around it decides what happens to them.

The AI solution: a canned data project

If every project meets the same failures, it helps to start from one that has already met them.

An AI solution in Lake is a canned data project. Orama's catalogue holds solutions to common problems, each built from the modelling methods it combines, with levels that set how far it goes.27 When you start one, Lake creates an AI project in your workspace with its pipeline already designed. Before anything else is created, the Project Agent reads your data and proposes the foundation the solution will run on: the snapshot date, what each row describes, the column to predict, the columns to learn from, and the conditions that rule an item out. You review the proposal, correct it, and confirm it.27

A canned project carries what someone learned the first time: the data the problem needs, the methods that work, and how to hold out data to measure the result. It is one way to follow Ng's advice and make your first projects ones likely to succeed.

It still runs on your data. Zech's pneumonia model slipped on X-rays from another hospital, and a packaged sepsis model used at hundreds of hospitals did poorly on Michigan's patients. A canned project in Lake keeps its own holdout and is measured on your data, and if your data lacks a condition the solution needs, the run stays blocked until the data supplies it.27 The template brings the design. Your data must earn the result.

You can also write your own solutions, or duplicate one of Orama's and adapt it, so each new project in your organization starts from what the last one learned.27

The test

Here is a test for any AI effort, in Lake or anywhere else. Ask where its project lives: the one place that holds its data, workflows, models, runs, and scores. Then go down the list of ways data projects fail, and for each one, ask what in that project stops it.

If every row has an answer, the effort has a real chance. If the project lives in a notebook, a slide deck, and someone's memory, it is running on hope.

Notes

  1. Andrew Ng, AI for Everyone, week 1, "Machine learning" (DeepLearning.AI and Coursera). ↩
  2. Ng, AI for Everyone, week 1, "What machine learning can and cannot do." ↩
  3. Stanford HAI, The 2026 AI Index Report, chapter 2, "Technical Performance." ↩
  4. Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (2015), figure 1. ↩
  5. Eliza Strickland, "Andrew Ng: Unbiggen AI," IEEE Spectrum, February 9, 2022. ↩
  6. Andrew Ng, AI for Everyone (DeepLearning.AI and Coursera), as I quoted him in "AI Agents: From Theory to Implementation," a talk given April 16, 2026. ↩
  7. Steve Sperandeo, "AI Agents: From Theory to Implementation," a talk given April 16, 2026, section I, "Data Project Success and Data Project Failure." ↩
  8. Ryseff, De Bruhl, and Newberry, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (RAND, 2024), summary. ↩
  9. Sambasivan et al., "'Everyone Wants to Do the Model Work, Not the Data Work': Data Cascades in High-Stakes AI" (2021). ↩
  10. Kapoor and Narayanan, "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science" (2023), abstract. ↩
  11. Zech et al., "Variable Generalization Performance of a Deep Learning Model to Detect Pneumonia in Chest Radiographs" (2018), abstract. ↩
  12. Wong et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients" (2021), abstract. ↩
  13. Horace He and Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference," September 10, 2025. ↩
  14. Pimentel et al., "A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks" (2019). ↩
  15. Ng, AI for Everyone, week 1, "Week 1 introduction." ↩
  16. Ng, AI for Everyone, week 1, "More examples of what machine learning can and cannot do." ↩
  17. Landing AI, AI Transformation Playbook, step 1, "Execute pilot projects to gain momentum." ↩
  18. Andrew Ng, "From Proof of Concept to Production," The Batch, May 6, 2020. ↩
  19. Andrew Ng, "Improve Agentic Performance with Evals and Error Analysis, Part 1," The Batch, October 15, 2025. ↩
  20. Martinez, Viles, and Olaizola, "A Survey Study of Success Factors in Data Science Projects" (2021). ↩
  21. Orama Lake documentation, "AI Projects and Pipelines," lake v0.9.0. ↩
  22. Orama Lake documentation, "Concepts and Glossary," lake v0.9.0. ↩
  23. Orama Lake documentation, "Train a Model," lake v0.9.0. ↩
  24. n8n, home page, read November 19, 2026. ↩
  25. Makridakis, Spiliotis, and Assimakopoulos, "The M4 Competition: 100,000 Time Series and 61 Forecasting Methods" (2020), finding 1. ↩
  26. Zaharia et al., "The Shift from Models to Compound AI Systems," Berkeley Artificial Intelligence Research (BAIR), February 18, 2024. ↩
  27. Orama Lake documentation, "AI Solutions, Methodologies and Opportunities," lake v0.9.0. ↩

Bibliography