AI Agents in Operations: What Actually Works, and What Still Does Not
A practical account of where AI agents deliver measurable value in operations-heavy businesses, where they fail, and what has to be true before you deploy one.
The short answer
AI agents deliver reliable value in operations today for document extraction, structured question answering over governed data, exception triage, and drafting routine correspondence, while they remain unreliable for autonomous financial decisions, unsupervised system writes, and any task where a plausible wrong answer is expensive and hard to detect.
Key takeaways
- The determining factor is not model quality. It is whether the data the agent reads is governed, current, and correct.
- Agents work well where a wrong answer is cheap and obvious. They fail where a wrong answer is expensive and plausible.
- Document extraction is the highest return, lowest risk deployment available to most operations businesses today.
- Natural language querying only works on a modeled semantic layer. Pointed at raw tables, it produces confident nonsense.
- Keep a human in the loop on anything that writes to a system of record or moves money. That is a design decision, not a maturity stage.
The gap between AI demos and AI in production has narrowed considerably, but it has not closed, and where it remains open it remains open for structural reasons rather than model limitations. This is a field report on what holds up in operations environments.
The test that predicts success
That question sorts nearly every proposed use case correctly, and it sorts them better than any assessment of model capability. An agent that drafts a first-pass email is safe because the sender reads it. An agent that adjusts a revenue forecast feeding a board pack is not, because nobody will independently recompute it.
What works reliably today
Document extraction and classification
This is the clearest win available to operations businesses. Contracts, subcontracts, change orders, insurance submissions, invoices, inspection reports, and prior authorizations all arrive as unstructured documents that humans currently read and retype into systems.
Extraction works because the ground truth is in the document. The model is not reasoning about the world, it is locating and structuring text that exists. Verification is cheap: show the extracted field next to the highlighted source passage and a human confirms in seconds rather than reading forty pages.
Question answering over a governed corpus
Asking a system what the force majeure terms are in a specific contract, or which subcontracts include a particular indemnity clause, works well when the corpus is bounded and the answer cites its source. The citation is not a nicety. It converts an unverifiable claim into a verifiable one, which is what makes the tool trustworthy enough to use.
Exception triage
Agents are good at reading a queue of anomalies, clustering them, and proposing a priority order with reasoning. A human still decides. The agent removes the sorting work, which is often 70 percent of the effort, without owning the decision.
Structured querying on a semantic layer
Natural language analytics genuinely works, with one hard prerequisite: the agent must query a modeled semantic layer where utilization, margin, and backlog each have exactly one definition, not raw source tables. On a governed model, the agent translates a question into a query against known metrics. On raw tables, it invents joins and produces numbers that look right and are not.
What still does not work
| Use case | Why it fails | Viable alternative |
|---|---|---|
| Autonomous financial decisions | Errors are plausible, expensive, and discovered late | Agent proposes with reasoning, a human approves |
| Unsupervised writes to a system of record | A wrong write propagates silently through downstream reporting | Write to a staging queue with human release |
| Analytics on ungoverned raw data | The agent invents joins and definitions, producing confident wrong numbers | Build the semantic layer first, then point the agent at it |
| Replacing judgment in client communication | Tone and relationship context are not in the training data | Draft generation with human editing before sending |
| Long autonomous multi-step chains | Error compounds at each step with no checkpoint | Short chains with verification between stages |
The prerequisite nobody wants to hear
Most failed AI deployments in operations businesses failed for the same reason: the underlying data was not ready. The model was fine. It was reading from a source where utilization meant three different things depending on which office entered the timesheet.
- 1Data is centralized. The agent can reach everything relevant without a human assembling a spreadsheet first.
- 2Definitions are governed. Each metric has one implementation, documented, in a modeling layer.
- 3Data is current. If it refreshes monthly, the agent gives monthly-quality answers regardless of how good the model is.
- 4Permissions are enforced upstream. The agent must inherit the user's data access, not bypass it. This is the most commonly skipped requirement and the most dangerous.
- 5Outputs are traceable. Every answer cites the records it came from, so it can be checked.
Deployment surface matters more than expected
Where the agent lives strongly predicts whether it gets used. A separate application requires a person to remember it exists and change their habits. An agent in Slack or Microsoft Teams meets people where they already work.
| Surface | Adoption | Best for |
|---|---|---|
| Slack or Teams | Highest | Ad hoc questions, alerts, quick lookups |
| Embedded in an existing internal tool | High | Task-specific assistance in context |
| Inside the BI tool | Medium | Follow-up questions on a dashboard someone is already reading |
| Standalone web application | Lowest | Deep, specialized workflows worth a context switch |
How to sequence a first deployment
- 1Pick a use case where wrong answers are cheap and obvious. Document extraction with source highlighting is the canonical starting point.
- 2Measure the baseline first. Hours currently spent, current error rate. Without this you cannot demonstrate value and the project dies at budget review.
- 3Keep a human in the loop for the first cycle, and instrument how often they override the agent. That override rate is your real accuracy metric.
- 4Expand only where the override rate is low and stable. Rising overrides mean the underlying data drifted, not that the model got worse.
- 5Do not automate the human out of the loop on anything that writes to a system of record or moves money, regardless of how good the numbers look.
The firms getting real value from AI in operations are not the ones that deployed the most sophisticated agents. They are the ones that did the unglamorous data work first and then applied AI to a narrow problem where being wrong was survivable.
Questions we get on this topic
What are the best AI use cases for operations businesses?
Document extraction and classification, question answering over a bounded document corpus with source citations, exception triage, and natural language querying against a governed semantic layer. These share a common property: a wrong answer is cheap to detect and cheap to correct.
Can AI answer questions about our business data?
Yes, provided it queries a modeled semantic layer where each metric has exactly one definition rather than raw source tables. Pointed at ungoverned data, an AI assistant will infer joins and definitions and produce numbers that are confidently wrong, which is worse than no answer.
Do we need a data warehouse before deploying AI?
For anything that answers questions about business performance, effectively yes. The agent can only be as accurate, current, and consistent as the data it reads. Document-focused use cases are the exception, because the ground truth lives in the documents themselves rather than in your systems.
Should AI agents be allowed to take actions automatically?
For low-stakes, easily reversible actions such as tagging a document or drafting a message, yes. For anything that writes to a system of record, moves money, or commits the firm, keep a human approval step. This is a permanent design decision rather than a stage you graduate out of.
How do you measure whether an AI deployment is working?
Track the human override rate, meaning how often a reviewer corrects or rejects the agent's output, against a baseline you measured before launch. A low and stable override rate justifies expanding scope. A rising rate usually signals that the underlying data drifted rather than that the model degraded.
Founder and CEO of VisualFlow Analytics. Former data analyst at Pratt & Whitney Canada, computer science and mathematics at McGill University. Leads technical delivery and client strategy across engineering, construction, and industrial data programs.