The kickoff meeting usually begins with optimism. There is a budget, someone has seen a competitor announce an AI success, and the team creates a slide titled “Phase 1: Model Development.” In many projects I have worked on, that is where the roadmap begins to fail. The team has started in the middle, before defining the process, checking the data, or agreeing on how success will be measured. A 2024 Gartner survey found that only 48% of AI projects reached production. A RAND Corporation study cites estimates suggesting that more than 80% of AI projects fail.
For the past two years, I have been building AI automation roadmaps and the data pipelines behind them. In the projects I have seen stall, model accuracy was rarely the main issue. The team was usually trying to automate an unclear process, using data without a clear owner, against a goal nobody had defined in measurable terms. The model was often the easiest part to build. What mattered more was whether the organisation had a stable process and usable data to back it up. Most AI roadmaps should therefore begin with a readiness assessment, not model selection.
Starting with the model means starting in the middle
A model automates a decision. If the decision is unclear, the model inherits that ambiguity and presents it as a probability score. I have seen teams try to automate approval processes where experienced managers reached different conclusions after reviewing the same case. That is not a modelling problem; it means the business rule has not been defined. Training a model on those decisions only reproduces the inconsistency at scale. Until the rule is clear, adding a model gives an undefined process a technical interface.
The first question should not be, “Which model do we need?” It should be whether a qualified person can consistently explain the correct output for a given input. If experienced employees disagree about the same case, the team needs to define the rule and document the exceptions before building anything. It must also decide who has the authority to resolve cases of uncertainty. Otherwise, the model will automate disagreement rather than a reliable decision.
The strongest automation candidates have enough volume to justify the investment, stable rules, and a manageable error profile. A 3% error rate may be acceptable when categorising internal documents, but the same rate could create serious consequences when a system affects lending or employment decisions. The UK’s Information Commissioner’s Office guidance on automated decision-making places additional requirements on decisions that have legal or similarly significant effects. A roadmap needs to account for those requirements before development begins. In practice, the best automation opportunities are often high-volume processes with clear answers and limited consequences per error.
Data readiness is not a yes-or-no question
Most teams ask whether their data is “good enough,” but that question is too broad. In my work, I review four areas: consistency, ownership, representation, and relevance to the business question. A dataset can look complete while still failing in one of them. Problems arise when definitions change without documentation, for example after a form or CRM update. The records may appear complete while representing different things across different periods. Unless the team identifies those changes, the model may learn from administrative noise rather than a real business pattern.
Ownership is another common risk. A team may know where the data is stored but still be unable to identify who can approve access or explain how it was collected. This dependency often appears after the pipeline has already been designed. If nobody can take responsibility for the dataset, the project is not ready. Data ownership determines whether the team can legally use the information and whether someone can answer questions when problems arise.
Representation is tested through edge cases. The initial data pull usually describes normal operations well, while rare cases may be stored in support tickets or separate spreadsheets. A model trained only on routine cases can perform well during evaluation and fail when employees need it most. Before modelling begins, the team should identify where exceptions are recorded and whether there are enough examples to test the system properly. The team must also check whether the data can answer the business question. More complex modelling cannot fix a mismatch between the available records and the problem being solved.
Turning requirements into a measurable roadmap
A credible AI roadmap states what result will show that the project worked. “Improve efficiency” is not enough. A useful target would be: “The current process requires 40 hours of manual work per week, and the project should reduce this to fewer than 10.” The roadmap should explain how often the result will be measured and who will track it. Without a baseline and a target, the project will eventually be judged through opinions or demo performance. A strong model score does not prove that the business problem has been solved.
I design roadmaps so the riskiest assumption is tested early, while it is still cheap to challenge. The team first confirms that the decision is clearly defined, then checks whether the available data can support it. Only after those conditions are met should it build the smallest usable version and put it in front of the people who will rely on it. Early user feedback often creates more value than additional modelling work. Employees quickly identify where the output does not fit the real workflow. The main time saving often comes from finding these mismatches before the team builds the wrong solution twice.
Model accuracy is not the business outcome
A model can reach 95% accuracy and still produce little value if the operations team does not trust it. Employees may ignore the recommendation or check every result manually. In either case, the system has failed to produce the expected operational change. That is why I track adoption, processing time, and the override rate. A high override rate may indicate weak performance, but it can also show that the model is solving the wrong problem or delivering its output at the wrong point in the workflow. Accuracy should support the business target rather than replace it.
AI readiness checklist
Before committing resources to model development, the team should be able to confirm:
- Is the business process clearly defined and consistently followed?
- Is the required data accessible, representative and owned by someone?
- What level of risk and error is acceptable for this use case?
- What measurable business outcome are we trying to improve?
- Who will use the system, and how will it fit into existing workflows?
If one of these points cannot be confirmed, the team should investigate before writing code. Sometimes the assessment leads into development. In other cases, it exposes a process problem, a data gap, or a use case that should not be automated. Both outcomes are useful. The purpose of an AI roadmap is to determine whether building a model is the right decision and whether the investment can produce a result people will use.
