There is a pattern to AI projects that stall. The model works. The interface is fine. Six weeks in, the team is not tuning prompts — they are arguing about which of two tables is authoritative, and whether the export from the old CRM ever included cancelled orders.
This is not a detour from the project. It is the project. Data preparation consistently accounts for 30 to 60% of total AI project budgets, more than the model work. Anyone who has shipped this kind of system expects that. It only feels like a surprise from the buying side, where the data looked fine because nobody had asked it a hard question yet.
Here is what "ready" actually means, in the order it tends to bite.
1. Can a program reach it at all?
Not "do we have this data" — can code get it, repeatedly, without a person? A weekly CSV someone exports by hand is not an integration; it is a dependency on that person's calendar.
Worth confirming early: is there an API or direct database access, what does it cost, is there rate limiting, and is there a non-production environment to develop against? The absence of a staging environment is one of the most reliable predictors of timeline slip we see, and it never appears in a proposal.
2. Does the same thing have the same name?
Entity resolution is unglamorous and decides whether the system works. If a customer exists as ACME Corp in billing, Acme Corporation in the CRM, and acme-corp-ltd in support tickets, then any question spanning those systems returns a partial answer with total confidence. That is worse than an error, because nobody investigates a confident answer.
You need one canonical identifier per real-world thing, and a documented rule for reconciling the rest. This is ordinary data engineering. It is also the step most often skipped in the rush to demo something.
3. Who is allowed to see what?
This is the one that causes incidents rather than delays.
Retrieval systems are extremely good at surfacing exactly the document someone should not have seen. If salary reviews, board notes or another client's records sit in the same store as general documentation, a well-phrased question will find them. And because the answer is generated rather than linked, the usual audit trail — "who opened this file?" — does not exist.
Getting this right means permissions travel with the content into the index, and every retrieval is filtered by the asking user's identity, not the service account's. Retrofitting that into a working prototype is close to a rebuild, which is why it belongs in week one. Data security when agents access multiple systems is repeatedly flagged as a top enterprise risk, and this is the specific mechanism behind it.
4. How stale is too stale?
"Real-time" is expensive and usually unnecessary; "whenever someone remembers to re-run the job" is not acceptable either. Pick the number deliberately per source. Pricing might need to be minutes fresh. Policy documents can be daily. An archive can be static.
The failure mode to design against is the silent one: the sync breaks, and the system keeps answering confidently from a stale index. Monitor freshness as a first-class metric and alert on it, the same way you would on error rate.
5. Are the documents actually machine-readable?
A large share of useful institutional knowledge is in PDFs, and a meaningful share of those are scans. If the text layer is missing or bad, retrieval quality collapses in a way that is hard to diagnose — the system is not wrong, it simply never sees the relevant passage.
Same story for tables. A table flattened into a text blob loses the row-column relationship that made it meaningful, and models will happily read across the wrong row. Budget for document processing as its own line item when scans and tables are in scope.
A readiness checklist
Before committing to an AI build, we would want yes-or-no answers to these. Run it yourself; it takes an afternoon and it is the highest-leverage afternoon in the project.
- Can code retrieve this data on a schedule without a human?
- Is there a non-production environment to develop against?
- Does every important entity have one canonical ID across systems?
- Can we answer "is this user allowed to see this record?" programmatically?
- Do we know how fresh each source must be, and would we notice if it stopped updating?
- Do our PDFs have a real text layer?
- Is there a named owner for each source who can answer questions about it?
- Do we have 50+ real examples of the questions this system must answer well?
Three or more "no" answers does not mean do not build. It means the data work is the first phase, and pretending otherwise just moves the cost later, when it is more disruptive.
Common questions
Can we not just point an LLM at our database?
For a narrow, well-modelled schema, sometimes yes. It breaks down when the schema encodes business rules that live in people's heads — status codes that mean different things by region, soft-deleted rows, historical fields that stopped being maintained. The model has no way to know which columns to distrust.
Does more data improve results?
Usually the opposite. Retrieval degrades as the store fills with near-duplicates and superseded versions. A curated set of current, authoritative documents outperforms an indiscriminate dump of everything, reliably.
How long does data preparation take?
For a single workflow with two or three sources, two to six weeks is typical, driven mostly by access and permissions rather than volume. Getting credentials and a staging environment often takes longer than the engineering.
Start with the assessment
The cheapest version of this is a short, fixed-scope assessment before any build commitment: what exists, what is reachable, what is authoritative, and what it would take to make it usable. It de-risks the largest line item in the budget for a fraction of it.
That is a large part of what our data engineering work involves, and it feeds directly into AI integration once the foundations hold. If you are weighing a build, start with a conversation — we would rather tell you the data is not ready than bill you to discover it.