How to Evaluate Business AI ROI and Adoption
For enterprise leaders and operators evaluating AI pilots, a practical scorecard connects quality, effort, adoption and costs to evidenced benefits and investment decisions.
Measure whether AI actually improves business performance—not just how much time it saves. Use the Dainin AI Value & Adoption Scorecard to assess output quality, productivity, adoption, financial return and operational risk before deciding whether to expand an AI deployment.
An AI assistant reduces proposal-drafting time from three hours to one. The team is enthusiastic, usage is increasing, and management wants to expand the pilot.
But the proposals still need specialist review. Some require extensive corrections. Nobody has measured whether total effort has fallen, whether the output meets the same standard, or what the business has gained from the time saved.
These are not minor details. They determine whether AI has delivered measurable value or simply shifted work from one part of the process to another.
Evaluating business AI ROI requires five things: reliable evidence, acceptable output quality, improved operating performance, sustained adoption and demonstrable economic value.
The Dainin AI Value & Adoption Scorecard brings these together in a structured evaluation method. It measures the whole workflow, distinguishes financial savings from released capacity, and prevents a promising productivity figure from concealing poor adoption or unacceptable risks.
The method below includes measurement definitions, a complete illustrative assessment, ROI calculations and decision rules for expanding, redesigning or stopping an AI initiative.
- Establish what successful work looks like before measuring AI
The first mistake in many AI evaluations is measuring the tool rather than the work.
If an employee spends 45 minutes prompting AI to produce a document that previously took three hours, the apparent saving is substantial. But if another employee spends two hours correcting the result, the end-to-end improvement may be small or nonexistent.
Start with one business workflow and follow it from initial request to accepted outcome.
For a proposal-writing process, that might mean:
Opportunity intake → approved source information → proposal preparation → specialist review → commercial approval → client delivery → corrections or follow-up.
Every stage matters, including work that fails, needs repeating or falls back to a manual process.
Before introducing AI, establish the following:
| Evaluation requirement | What must be defined |
|---|---|
| Workflow boundary | Where the work begins and what counts as completed delivery |
| Acceptance criteria | What makes the result accurate, usable and compliant with business requirements |
| Eligible workload | Which tasks qualify for AI assistance, including tasks where AI is subsequently not used |
| Quality requirements | Which claims, prices, calculations or commitments require exact validation |
| Authority | Who may review, approve, reject or execute consequential actions |
| Baseline period | Which existing results and costs provide the comparison |
| Evaluation period | When the AI pilot begins and ends |
| Decision owner | Who determines whether the deployment expands, changes or stops |
Measure labour effort separately from elapsed turnaround time. Waiting two days for approval is not equivalent to 16 hours of work.
Use comparable workloads. A baseline containing complex enterprise tenders cannot fairly be compared with an AI pilot handling only short, standard proposals.
Where possible, evaluate comparable AI-assisted and non-assisted workflows during the same period. If the only available comparison is before and after deployment, document changes in demand, staffing, complexity and operating conditions.
A before-and-after difference is evidence of a change. It is not automatically proof that AI caused it.
Do not replace missing baseline records with vendor productivity estimates. If historical records are inadequate, collect a short prospective baseline before making a financial claim.
- The Dainin AI Value & Adoption Scorecard
The Dainin AI Value & Adoption Scorecard evaluates an AI initiative through five decision gates.
These are not five scores to average together. Each answers a different question that matters to the investment decision.
| Decision gate | Question it must answer | What constitutes useful evidence |
|---|---|---|
| 1. Evidence reliability | Can we trust the comparison? | Defined populations, consistent records, complete costs and documented assumptions |
| 2. Quality and control | Is the resulting work acceptable and sufficiently safe? | Acceptance records, review outcomes, incident severity and approval compliance |
| 3. Operating performance | Does AI improve the whole workflow? | Accepted output, end-to-end effort, turnaround and rework |
| 4. Adoption and coverage | Is AI being used where intended, and do people continue using it? | Eligible-work coverage, repeat use, non-use and fallback records |
| 5. Realised economic value | Do supported benefits justify the total incremental cost? | Avoided expenditure, attributable contribution, cost records and other separately reported benefits |
An initiative can perform well on one gate and fail another.
High adoption does not compensate for inaccurate output. Faster completion does not establish financial savings. A positive ROI estimate does not justify expanding a system that breaches an important approval requirement.
The gates should be assessed together, with quality and safety requirements acting as constraints rather than numbers that can be offset by favourable financial results.
What the scorecard measures
The following definitions can be copied into a spreadsheet or evaluation system.
| Measure | Calculation or recording rule |
|---|---|
| Accepted completions | Number of tasks completed to the agreed standard, after any required corrections |
| First-pass acceptance | Tasks accepted without substantive revision ÷ tasks first submitted for review |
| End-to-end effort | Total hours spent on preparation, AI interaction, review, approval, correction and fallback |
| Effort per accepted completion | Total workflow hours ÷ accepted completions |
| Workflow coverage | Eligible tasks that used AI ÷ all eligible tasks entering the workflow |
| Repeat use | Initially active users who use AI again ÷ initially active users with another eligible opportunity during the follow-up period |
| Exceptions | Rejected outputs, overrides, incidents and unauthorised actions, recorded by type and severity |
| Monetised benefits | Financial improvements supported by evidence and attributable to the intervention |
| Incremental costs | Additional implementation and operating costs relative to the existing approach |
| Investment decision | Expand, redesign, continue bounded evaluation, or stop |
For repeat use, exclude users who had no further eligible opportunity. Otherwise, the evaluation may incorrectly classify people as abandoning the system when they simply had no reason to use it again.
For coverage, retain the entire eligible population, including people who did not use AI. Measuring only enthusiastic adopters creates a misleading picture of operational impact.
Exceptions should not be reduced to one aggregate percentage. A formatting error and an unauthorised commercial commitment have very different consequences.
Set the decision conditions before reviewing the results
There are no universal thresholds that make an AI initiative successful.
A proposal-writing assistant, medical documentation system and financial approval agent have different acceptable error rates, operating requirements and risk profiles.
The organisation should define its own thresholds before examining pilot results.
For example, it may require a minimum reduction in effort, no increase in substantive errors, specified adoption coverage and zero critical approval violations.
Recording these conditions in advance reduces the temptation to reinterpret a disappointing outcome as success.
- Worked example: Evaluating an AI proposal-writing pilot
Consider a hypothetical mid-market professional-services company evaluating AI-assisted proposal preparation.
The business wants to reduce the time spent preparing proposals without compromising commercial accuracy, approval requirements or quality.
It measures a baseline over eight weeks and compares it with an eight-week AI pilot. Both periods contain 120 eligible proposals with a comparable complexity mix.
The following dataset is synthetic and exists solely to demonstrate the evaluation method.
Step 1: Measure the operating results
| Metric | Baseline | AI pilot |
|---|---|---|
| Eligible proposals | 120 | 120 |
| Proposals ultimately accepted | 100 | 108 |
| Accepted on first review | 72 | 84 |
| First-pass acceptance rate | 60% | 70% |
| Total end-to-end workflow effort | 480 hours | 378 hours |
| Effort per accepted proposal | 4.8 hours | 3.5 hours |
| AI-assisted workflow coverage | Not applicable | 75% |
| Repeat use among returning eligible users | Not applicable | 80.8% |
| Severe AI-related incidents | Not applicable | 0 observed |
For this example, every eligible proposal reached an initial review, so first-pass acceptance uses 120 as the denominator in each period.
The total-effort figures include unsuccessful attempts, corrections, reviews and fallback work—not just work associated with proposals that were ultimately accepted.
During the pilot, 90 of the 120 eligible proposals received AI assistance. That produces 75% coverage.
Of 26 initially active users who had another eligible opportunity during the follow-up window, 21 used AI again. That produces an 80.8% repeat-use rate.
Step 2: Calculate the productivity change
The baseline required:
480 hours ÷ 100 accepted proposals = 4.8 hours per accepted proposal
The AI pilot required:
378 hours ÷ 108 accepted proposals = 3.5 hours per accepted proposal
The observed reduction in effort per accepted proposal is:
(4.8 − 3.5) ÷ 4.8 × 100 = 27.1%
Total recorded workflow effort also declined by 102 hours, or 21.3%, while accepted completions increased from 100 to 108.
First-pass acceptance improved by 10 percentage points.
These are promising operating results. The same work population was evaluated, quality did not deteriorate according to the selected acceptance measure, and the organisation completed more acceptable work with less recorded effort.
But three questions remain.
First, did AI cause the improvement, or did other changes contribute?
Second, what happened to the 102 hours no longer spent on the evaluated workflow?
Third, did the company actually save money or generate additional financial value?
A productivity result is not yet an ROI result.
Step 3: Examine adoption and reliability
The pilot achieved 75% coverage and 80.8% repeat use among previously active users with another eligible opportunity.
Those are different measures.
Coverage tells management how much of the intended workload the technology reached.
Repeat use indicates whether people who tried the system continued using it when the opportunity arose.
A product might achieve excellent repeat use among a handful of enthusiastic employees while leaving most of the intended workload untouched. Conversely, compulsory usage could produce high coverage while concealing dissatisfaction, heavy review effort or frequent manual corrections.
The company should examine why the remaining 25% of eligible proposals did not use AI. Some may have required specialist judgement, involved unsuitable source material or encountered technical barriers.
A coverage target should not encourage staff to apply AI to work for which it is inappropriate.
The absence of severe incidents during this eight-week pilot is encouraging, but it does not demonstrate that future incidents are impossible. The organisation must continue monitoring the relevant failure conditions.
Step 4: Evaluate the financial result
Suppose the company also has documented evidence that it reduced an external proposal-support contract, avoiding £12,000 in expenditure during the evaluation period.
Assume the reduction is attributable to the changed workflow, does not conceal replacement expenditure elsewhere, and is supported by the revised contract and actual invoices.
Further suppose that the AI pilot incurred £8,000 in complete incremental costs, including licensing, configuration, training, usage, review and administration.
The resulting financial assessment is:
| Financial measure | Amount |
|---|---|
| Documented expenditure avoided | £12,000 |
| Complete incremental AI costs | £8,000 |
| Net monetised benefit | £4,000 |
| ROI | 50% |
The 102 hours of released capacity are not added to the £12,000.
They may have operational value, but until the organisation demonstrates how those hours were used and whether they created further economic benefits, counting them as additional cash savings would overstate the result.
Similarly, the eight additional accepted proposals are not automatically counted as new revenue or profit. That would require evidence that they led to incremental commercial outcomes, together with an appropriate contribution calculation.
The example therefore produces two separately useful findings:
Operating result: The workflow required less effort per accepted proposal while maintaining or improving the selected quality measures.
Financial result: Under the stated hypothetical evidence and cost assumptions, the initiative generated a 50% ROI during the evaluation period.
These findings are related, but they should not be combined into an inflated benefit figure.
- Calculate ROI without confusing time savings with financial value
The ROI calculation itself is straightforward.
The difficult part is determining which benefits legitimately belong in it.
The US Office of Personnel Management's Evaluation Guide: Evidence-Based Strategies to Capture the Benefits and Costs of Work-Life Programs uses net programme benefits divided by programme costs, expressed as a percentage.
For a business AI evaluation, the equivalent calculation is:
ROI (%) = (Attributable monetised benefits − Incremental costs) ÷ Incremental costs × 100
Use a consistent evaluation period, currency and comparison baseline.
Incremental costs should include the additional spending required to obtain the claimed benefits, relative to the existing approach.
Do not calculate a percentage when the cost denominator is zero or inadequately established.
Four benefit categories that should remain distinct
1. Actual expenditure avoided
These are genuine reductions in payments or expenditure that would otherwise have occurred.
Examples include a cancelled external service, a reduced software subscription or a demonstrable reduction in purchased processing capacity.
2. Redeployed capacity
AI may release employee hours that are then used for other work.
That can be valuable, particularly where capacity constraints were limiting output, but hours saved do not automatically translate into lower payroll expenditure.
Report the released capacity and its subsequent use. Monetise it only when there is a defensible financial mechanism.
3. Additional delivered work
AI may enable a business to serve additional customers, prepare more proposals or complete more work.
The relevant financial benefit is attributable incremental contribution, not gross revenue. Any associated delivery costs must be accounted for without double-counting.
4. Nonfinancial improvements
Quality, employee experience, customer response times, accessibility and operational resilience can justify an investment.
They should be reported clearly, even when a reliable monetary value is unavailable.
Do not invent a financial equivalent merely to produce an ROI percentage.
Maintain a benefit-evidence ledger
For each claimed benefit, record what changed, how it was valued and what proves it occurred.
| Benefit claimed | Evidence required | Treatment |
|---|---|---|
| Reduced supplier expenditure | Contract changes, invoices and replacement-cost checks | Count supported net expenditure avoided |
| Released employee time | Comparable stage-level time records and evidence of redeployment | Report capacity separately unless financial value is substantiated |
| Additional profitable output | Incremental accepted work, contribution and attribution evidence | Count supported incremental contribution |
| Better output quality | Acceptance, defect and correction records | Report improvement; monetise only evidenced economic effects |
| Reduced software costs | Cancelled or reduced licences and actual invoices | Count net expenditure avoided |
| Lower operational risk | Recorded incidents, exposure and valid cost evidence | Report risk changes; monetise only with defensible assumptions |
Count every economic effect once.
The same released employee hours cannot simultaneously be treated as payroll savings and as the capacity that generated additional revenue.
A software subscription saving already counted as a benefit should not also be deducted from incremental costs unless the accounting treatment explicitly avoids duplication.
Do not attribute a sales win solely to an AI-generated proposal without examining other contributing factors.
Test the financial result against uncertainty
In the worked example, the ROI is 50% only because the assumed £12,000 avoided expenditure is supported and attributable.
If later review shows that only £7,000 of those savings can be substantiated, the calculation changes:
(£7,000 − £8,000) ÷ £8,000 × 100 = −12.5%
The operating performance may still have improved, but the claimed financial return would no longer be positive.
This is why a credible evaluation must retain its underlying evidence rather than publishing an isolated percentage.
Do not automatically annualise an eight-week pilot. Costs, adoption, capacity constraints and the value of subsequent work may change as deployment expands.
- What research tells us about AI productivity and adoption
Research supports assessing AI outcomes at the level of real work rather than assuming that faster individual tasks create equivalent business returns.
In Generative AI at Work, published in The Quarterly Journal of Economics in 2025, Erik Brynjolfsson, Danielle Li and Lindsey Raymond examined AI assistance using data from 5,172 customer-support agents.
The study found a 15% average increase in issues resolved per hour following access to AI assistance, with substantial differences between worker groups.
Less experienced and lower-skilled workers improved both speed and quality. The most experienced and highest-skilled workers saw small speed gains and small quality declines.
The implication is not that the same results will occur in every business. It is that an average productivity figure can conceal important differences in who benefits and under what conditions.
An AI evaluation should therefore examine relevant groups, roles and task complexity rather than rely exclusively on one aggregate result.
A second study, Shifting Work Patterns with Generative AI, by Eleanor W. Dillon, Sonia Jaffe, Nicole Immorlica and Christopher T. Stanton, examined a six-month randomised experiment involving 7,137 knowledge workers across 66 firms.
During the second half of the experiment, the 80% of treated workers who used the AI tool spent two fewer hours per week on email and reduced work outside regular hours.
The researchers did not detect changes in the quantity or composition of workers' tasks attributable to individual-level AI provision.
That distinction is directly relevant to ROI assessment.
AI can improve an individual's experience and reduce time spent on particular activities without immediately producing a corresponding increase in output or measurable profitability.
Neither result is unimportant. They simply represent different kinds of value.
- Decide whether to expand, redesign or stop
An AI pilot should end with an authorised decision, not merely a presentation of favourable metrics.
The Dainin AI Value & Adoption Scorecard uses four possible outcomes.
| Decision | When it applies |
|---|---|
| Expand | Evidence is sufficiently reliable; required quality and safety conditions pass; operating performance and adoption support the proposed scope; and the financial or explicitly agreed nonfinancial case is justified |
| Redesign | A remediable problem prevents expansion, and the team can identify the correction, owner, cost and next evaluation |
| Continue bounded evaluation | Important evidence remains incomplete, but additional testing is justified within defined limits, budget and review date |
| Stop or revert | Critical requirements fail, material risks remain unacceptable, or further investment is unjustified |
Apply the gates in a disciplined order.
First, determine whether the evidence is reliable enough to make a decision. Missing cost or acceptance records should not silently be treated as zero.
Next, check critical safety, quality and authority conditions. A consequential breach may require immediate suspension regardless of the financial result.
Then assess operating performance, adoption and economic value against the thresholds agreed before the pilot.
Where the evidence supports only a narrow use case, expansion should remain within that scope.
Applying the rules to the proposal-writing example
Suppose the business agreed in advance that expansion would require:
- First-pass acceptance no worse than the 60% baseline, with at least 100 ultimately accepted proposals across a comparable workload.
- Average effort of no more than four hours per accepted proposal.
- At least 70% eligible-work coverage and 70% repeat use among returning eligible users.
- No severe approval or information-integrity incidents.
- Complete incremental cost records and a positive, supportable financial case.
The illustrative pilot meets these conditions.
Its first-pass acceptance is 70%, average effort is 3.5 hours per accepted proposal, coverage is 75%, repeat use is 80.8%, no severe incident was observed, and the assumed documented financial evidence supports a positive ROI.
That would justify controlled expansion to comparable proposal types, provided the evaluation was sufficiently reliable and the relevant risks remained acceptable.
It would not establish that the same results apply to complex legal documents, financial commitments or unfamiliar workloads.
The expansion decision should specify the authorised use cases, ongoing monitoring measures, accountable owner and circumstances that would trigger review or suspension.
If the £12,000 benefit could not be substantiated, the financial gate would remain unresolved. Management might continue a bounded evaluation to obtain the missing evidence rather than approve broad deployment on the strength of an unsupported ROI estimate.
Keep monitoring after expansion
Evaluation does not end when the pilot succeeds.
The NIST AI Risk Management Framework's Generative AI Profile includes guidance under MANAGE 4.1–4.3 relating to monitoring, improvement, escalation and incident response.
It provides a useful governance reference, not a financial ROI benchmark.
The organisation should continue tracking the measures that justified expansion, especially accepted output, exceptions, operating costs and adoption across the wider population.
If circumstances change, the original ROI estimate should be revisited.
- Make AI evaluation part of normal business operations
One of the difficulties with AI ROI measurement is that the evidence is often spread across different systems.
Usage information sits in the AI tool. Time and delivery records sit elsewhere. Costs appear in finance systems. Approval histories are held in messages or project applications. Business outcomes are reviewed separately.
This fragmentation makes it difficult to connect AI activity with actual business results.
An effective operating model should preserve the relationship between the original business objective, the work performed, the approvals required, the costs incurred and the resulting outcome.
That principle is central to Dainin's Cognitive Enterprise Operating System.
Dainin brings business context, AI agents, work management and governed decisions into a connected environment. Its Digital Double is designed to maintain organisational knowledge and context across functions, while defined authority determines which actions agents may perform and which require human approval.
This creates a foundation for evaluating AI within the operational processes it supports, rather than relying only on model usage statistics.
Where the relevant data and integrations are available, organisations can examine work, approvals, exceptions and outcomes using the same business context.
The Dainin AI Value & Adoption Scorecard is deliberately platform-independent. It can be applied to Dainin, another AI product, or a combination of systems.
Its purpose is to help organisations determine whether the capability they are purchasing is producing acceptable work, being used appropriately and creating enough value to justify continued investment.
The real test of business AI value
An AI tool that saves time is useful. An AI deployment that produces better work, operates reliably, earns sustained adoption and delivers demonstrable business value is an investment.
Those are not the same thing.
Before expanding an AI initiative, establish the baseline, measure the complete workflow, account for review and correction costs, distinguish released capacity from monetary savings, and examine whether the benefits survive scrutiny.
Use the scorecard to make a decision that can be revisited as evidence improves.
The objective is not to prove that AI works. It is to determine where AI works well enough, safely enough and economically enough to deserve a larger role in the business.
Explore how Dainin connects business intelligence, decisions and execution
About the methodology
Dainin AI Value & Adoption Scorecard — Version 1.0
Developed by Dainin Research & Insights.
This methodology combines established business evaluation principles into five decision gates: evidence reliability, quality and control, operating performance, adoption and coverage, and realised economic value.
Its contribution is an integrated evaluation and decision process rather than a new financial ROI formula.
The method is intended for practical business evaluation. It is not an independently validated predictive model, industry benchmark or certification standard.
Thresholds must be adapted to the workflow, risk profile and organisational requirements. The completed proposal-writing assessment uses synthetic data and does not represent measured Dainin customer performance.
Suggested citation: Dainin Research & Insights (2026), How to Evaluate Business AI ROI and Adoption: The Dainin AI Value & Adoption Scorecard, Version 1.0.
References
- Brynjolfsson, E., Li, D. and Raymond, L. (2025). Generative AI at Work. The Quarterly Journal of Economics, 140(2), 889–942.
- Dillon, E. W., Jaffe, S., Immorlica, N. and Stanton, C. T. Shifting Work Patterns with Generative AI. American Economic Review: Insights.
- U.S. Office of Personnel Management. Evaluation Guide: Evidence-Based Strategies to Capture the Benefits and Costs of Work-Life Programs.
- National Institute of Standards and Technology (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1.
Run your own AI ROI and Adoption Assessment
You don't need specialist software to begin evaluating your AI investment.
The Dainin AI Value & Adoption Scorecard can be applied using your preferred AI assistant. Copy the prompt below into ChatGPT, Claude, Gemini or another conversational AI tool.
The assistant will guide you through the assessment, ask questions relevant to your situation, identify missing information and produce a structured report covering productivity, adoption, financial value and deployment readiness.
You can complete an initial assessment even if you don't have every figure available. Missing information will be identified rather than replaced with invented estimates.
Note: These are simplified single agent sharable versions of Dainin AI's instruction sets with proipretory IP stripped out.
# DAININ::AVA-ADVANCED v2.1
## Enterprise AI Value, ROI & Adoption Evaluation Engine
ROLE=Principal[AI_Transformation,Management_Consulting,Financial_Analysis,Econometrics,Implementation_Science,AI_Governance].
STANDARD=Dainin_AI_Value_Adoption_Scorecard_v1.0.
MODE=Adaptive|Stateful|Evidence_Governed|Decision_Oriented.
OBJECTIVE=Evaluate_AI_Investment{Business_Value,Operational_Performance,Quality,Adoption,Governance,Economics,Scale_Readiness}.
OUTPUT=Executive_Investment_Report+Technical_Evaluation_Annex.
CONSTRAINT=No_Fabricated_Evidence|No_Arbitrary_Scores|No_Unsupported_Causality|No_Double_Counting.
## 0::INVARIANTS
I01: ACTIVITY≠OUTPUT≠ACCEPTED_OUTPUT≠BUSINESS_OUTCOME≠ECONOMIC_VALUE.
I02: TIME_SAVED≠CAPACITY_RELEASED≠COST_AVOIDED≠CASH_SAVED.
I03: USAGE≠ADOPTION≠EFFECTIVE_USE≠VALUE_REALISATION.
I04: OBSERVATION≠INFERENCE≠FORECAST≠CAUSAL_EFFECT.
I05: UNKNOWN≠ZERO≠FALSE≠N/A.
I06: MODEL_QUALITY≠WORKFLOW_QUALITY≠BUSINESS_PERFORMANCE.
I07: ROI_POSITIVE cannot override CRITICAL_RISK.
I08: ADOPTION_HIGH cannot compensate for QUALITY_FAILURE.
I09: BASELINE/INTERVENTION must share equivalent boundaries, populations, units and outcome definitions.
I10: BENEFITS/COSTS must share comparator, period, scope and accounting basis.
I11: No fabricated inputs, benchmarks, measurements, certifications, sources or statistical precision.
I12: No retrospective success thresholds represented as pre-agreed.
I13: Missing evidence does not automatically mean failure; separate inadequacy of evidence from demonstrated underperformance.
I14: Financial benefits require identifiable realisation and attribution mechanisms.
I15: Count each economic effect exactly once.
I16: Do not infer causality from before-after observations.
I17: Do not disclose confidential information unnecessarily.
I18: No weighted composite score across independent decision gates.
I19: Recommendations must be decision-relevant, proportionate to evidence and operationally executable.
I20: Established theoretical frameworks retain attribution; Dainin provides the integrated evaluation protocol.
## 1::ASSESSMENT_CONTROLLER
STATE={INIT,CONTEXT,THEORY_OF_CHANGE,WORKFLOW,BASELINE,EVIDENCE,VALIDATION,QUALITY,PERFORMANCE,ADOPTION,CAUSALITY,ECONOMICS,RISK,GATES,DECISION,REPORT}.
MODE∈{PRE_DEPLOYMENT,PILOT,OPERATIONAL,RETROSPECTIVE,VENDOR_COMPARISON,PORTFOLIO,SCALE_READINESS}.
CLASSIFY_MODE from user inputs; update dynamically.
TRANSITIONS:
- Missing decision-critical data -> request minimum sufficient information.
- Contradictory data -> reconcile or flag dependent conclusions.
- Adequate data -> analyse without further questioning.
- PRE_DEPLOYMENT -> pilot/business-case design; prohibit realised-ROI claims.
- POST_DEPLOYMENT without comparator -> current-state analysis; causal improvement unresolved.
- Material risk -> assess containment before expansion.
- User requests completion -> produce evidence-bounded report immediately.
- Available tools -> use for reliable computation/research where appropriate; never imply unavailable tool usage.
INTERVIEW_POLICY:
Ask <=3 related questions/turn. Optimise by {Decision_Impact,Uncertainty_Reduction,Expected_Information_Value,Response_Burden}. Avoid duplicate questions. Prefer business language in interactions; technical precision in analysis. Accept anonymised records, aggregates, estimates and unknowns. Classify each appropriately. Explain unavailable evidence collection methods. Allow user to finish early.
FIRST_QUESTION="What AI tool, agent or workflow are you evaluating, and is it planned, being piloted or already operating?"
## 2::FRAMEWORK_ROUTER
Select frameworks by applicability; do not execute every framework mechanically.
F01 THEORY_OF_CHANGE / LOGIC_MODEL:
Inputs→Activities→Outputs→Intermediate_Outcomes→Business_Outcomes→Value.
For each edge: {Mechanism,Assumption,Evidence,Confounders,Validation}.
TRIGGER=ALL.
F02 CRISP_DM:
Business_Understanding→Data_Understanding→Preparation→Modelling/Analysis→Evaluation→Deployment.
Iterative; adapt modelling stage to the assessed AI system.
TRIGGER=Data-intensive or repeatable analytical workflows.
F03 LEAN/SIPOC/VSM:
Map Intake→Processing→AI→Review→Approval→Correction→Delivery.
Distinguish {Touch_Time,Wait_Time,Rework,Fallback,Throughput,Bottlenecks}.
TRIGGER=Productivity/automation.
F04 DMAIC/FMEA:
Define→Measure→Analyse→Improve→Control.
Evaluate defect severity, variation, control failure, detection and consequences.
TRIGGER=Quality/operational risk.
F05 RE_AIM:
Reach,Effectiveness,Adoption,Implementation,Maintenance.
Adapt to enterprise AI without asserting independent validation.
TRIGGER=Deployment/adoption/scale.
F06 UTAUT:
Performance_Expectancy,Effort_Expectancy,Social_Influence,Facilitating_Conditions.
Extend contextually with trust, training, incentives, workflow fit and governance.
TRIGGER=Adoption barriers.
F07 CAUSAL_INFERENCE:
Potential_Outcomes,DAGs,Confounding,Selection_Bias,Interference.
Select RCT/Matched/DiD/Interrupted_Time_Series/Observational according to design and assumptions.
TRIGGER=Attribution/comparative-effect claims.
F08 STATISTICAL_EVALUATION:
Sampling,Wilson_Intervals,Paired_Comparison,Cluster_Aware_Analysis,Sensitivity,Missingness.
Bayesian methods only with declared prior, likelihood and justified model.
TRIGGER=Uncertainty/heterogeneous effects.
F09 FINANCIAL_APPRAISAL:
Incremental_Cost_Benefit,TCO,Contribution,ROI,Payback,NPV,Scenario,Sensitivity.
TRIGGER=Financial decisions.
F10 NIST_AI_RMF:
GOVERN→MAP→MEASURE→MANAGE.
Consider NIST_AI_600-1 for generative AI.
TRIGGER=Trustworthiness/governance.
F11 ISO_IEC_42001:
Management-system principles: policy, ownership, lifecycle controls, monitoring, improvement.
TRIGGER=Enterprise AI governance.
PROHIBIT=Unjustified certification/compliance claims.
F12 DECISION_THEORY:
Expected_Value,Opportunity_Cost,Reversibility,Value_of_Information,Cost_of_Delay.
Quantify only with supported assumptions.
TRIGGER=Material uncertainty/investment alternatives.
F13 BENEFITS_REALISATION:
Benefit_Owner,Mechanism,Measurement,Attribution,Realisation_Date,Evidence.
TRIGGER=ALL financial and nonfinancial value claims.
F14 PORTFOLIO_ANALYSIS:
Strategic_Fit,Dependencies,Resource_Constraints,Cost_Overlap,Risk_Concentration.
TRIGGER=Multiple competing AI initiatives.
FRAMEWORK_SELECTION_RULE=Minimum necessary combination capable of answering the actual decision question.
## 3::EVIDENCE_MODEL
CLAIM={id,type,source,date,period,scope,method,limitations,decision_impact}.
TYPE∈{OBSERVED_VERIFIED,OBSERVED_UNVERIFIED,USER_ESTIMATE,DERIVED,INFERRED,HYPOTHESIS,FORECAST,ASSUMPTION,UNKNOWN,CONFLICTING,N/A}.
PREFERENCE:
Auditable_Primary_Records > Reconciled_Exports > Contractual/Financial_Evidence > Timestamped_Operational_Notes > User_Estimates > Contextual_External_Benchmarks > Unsupported_Assumptions.
Evaluate actual source quality; hierarchy is not absolute.
Apply W3C_PROV concepts: ENTITY/ACTIVITY/AGENT + temporal provenance.
VALIDATE:
- Units, currency, period and comparator.
- Identity and duplicate records.
- Eligible vs observed population.
- Missingness and possible missing-not-at-random mechanisms.
- Task and reviewer selection bias.
- Changes in workflow, staffing, AI version and task complexity.
- Financial attribution and overlapping benefits.
- Evidence contemporaneity.
- Outcome definitions and thresholds.
DO_NOT_IMPUTE missing commercial benefit.
DO_NOT_CONVERT historical association into causal proof.
DO_NOT_REPLACE absent facts with vendor statistics.
If external research available: prefer dated primary sources, validate population/methodology and distinguish externally observed effects from this organisation's expected outcomes.
## 4::EVALUATION_SCHEMA
ENTITY AI_INITIATIVE={objective,technology,stage,workflow,scope,owner,comparator,period}.
ENTITY TASK={id,eligibility,complexity,assignment,ai_use,outcome,effort,rework,review}.
ENTITY OUTPUT={task_id,submission,accepted,first_pass,defects,severity}.
ENTITY USER={anonymous_id,role,eligible_opportunities,usage_events,training}.
ENTITY COST={category,period,actual_or_forecast,amount,comparator,evidence}.
ENTITY BENEFIT={type,amount,mechanism,attribution,realisation,evidence,overlap}.
ENTITY INCIDENT={type,severity,consequence,authority,mitigation,status}.
ENTITY DECISION={gate_states,thresholds,scope,owner,review_trigger}.
DATA_MINIMUM:
Workflow boundary, baseline/comparator, eligible population, acceptance criteria, evaluation horizon, intervention scope, accountable decision-maker.
Separate task-level, user-level, department-level and organisation-level effects. Do not infer population-wide improvement from a self-selected group.
## 5::CAUSAL_DESIGN
DEFINE:
T=AI intervention.
C=Comparator.
Y=Primary outcome.
U=Unit of analysis.
P=Target population.
H=Observation horizon.
E=Effect/estimand of interest.
DESIGN_PRIORITY:
1 Randomised comparison where feasible.
2 Appropriate matched contemporaneous comparison.
3 Difference-in-differences where parallel-trends assumption is plausible.
4 Interrupted time series with adequate observations.
5 Historical before-after.
6 Post-only observation.
Design choice must fit data and constraints; hierarchy is not universally prescriptive.
CHECK={Selection,Confounding,Seasonality,Task_Mix,Concurrent_Changes,Learning_Effects,Interference,Model_Version,Missingness,Attrition}.
Distinguish ITT vs treatment-on-treated/per-protocol when assignment and usage differ.
Avoid adjustment for post-treatment mediators when estimating total intervention effects.
Use uncertainty analysis appropriate to data structure. No invented intervals, p-values or effect sizes.
If causality is unresolved: report association + competing explanations + test design.
## 6::FIVE_GATE_ENGINE
GATE_STATUS∈{PASS,FAIL,INSUFFICIENT_EVIDENCE,NOT_YET_APPLICABLE}.
Every gate returns {Status,Evidence,Measurements,Limitations,Required_Action,Decision_Impact}.
G1 EVIDENCE_RELIABILITY:
Comparator validity, population, sampling, outcome definitions, task-mix, temporal alignment, completeness, provenance, attribution limitations.
PASS only if evidence suffices for decision scope.
G2 QUALITY_CONTROL:
Acceptance, defects, first-pass quality, correction burden, material incidents, human approval, privacy, security, authority limits and fallback.
Critical unresolved control failures block unrestricted expansion.
G3 OPERATING_PERFORMANCE:
Accepted output, total workflow effort, rework, throughput, turnaround, fallback, task complexity and outcome heterogeneity.
Isolated step-speed improvements do not establish end-to-end gain.
G4 ADOPTION_COVERAGE:
Eligible-work coverage, repeat use, appropriate use, implementation fidelity, role heterogeneity, sustainment, training and non-use.
High usage alone does not pass.
G5 ECONOMIC_VALUE:
Supported benefits, complete incremental costs, ROI where valid, contribution, cash flow, nonfinancial value, uncertainty and opportunity cost.
Explicitly approved nonfinancial objectives can justify investment without positive monetised ROI; label separately.
Do not average gates. A failure in a non-compensatory control is not offset by gains elsewhere.
## 7::METRICS_ENGINE
All metrics require defined numerator, denominator, period, cohort, population and source.
M01 ACCEPTED_COMPLETIONS=count(distinct accepted task_id).
M02 FIRST_PASS_ACCEPTANCE=tasks accepted on first review without substantive revision / tasks first submitted for review.
M03 TOTAL_EFFORT=Σ(preparation+AI_interaction+human_review+approval_labour+rework+fallback+relevant_coordination).
M04 EFFORT_PER_ACCEPTED=TOTAL_EFFORT/ACCEPTED_COMPLETIONS.
M05 RELATIVE_EFFORT_REDUCTION=(BASELINE_EFFORT_PER_ACCEPTED−AI_EFFORT_PER_ACCEPTED)/BASELINE_EFFORT_PER_ACCEPTED.
M06 THROUGHPUT=accepted completions / defined time period.
M07 TURNAROUND=accepted completion timestamp−intake timestamp; report appropriate distribution.
M08 REWORK_SHARE=rework hours/total workflow hours.
M09 WORKFLOW_COVERAGE=AI-assisted eligible tasks/all eligible tasks.
M10 REPEAT_USE=initially active users using AI again/initially active users with another eligible opportunity.
M11 USER_ADOPTION=eligible users using AI appropriately/users with eligible opportunities.
M12 FALLBACK_RATE=eligible tasks requiring fallback/all eligible tasks.
M13 INCIDENT_PROFILE={frequency,denominator,severity,exposure,unavailable_data}.
M14 BENEFIT_REALISATION={claimed,substantiated,unresolved,nonmonetised}.
RULES:
- Denominator zero/unavailable -> NOT_CALCULABLE.
- Failed/abandoned tasks remain visible in appropriate eligibility, effort and completion measures.
- Review, correction and fallback must not disappear.
- Deduplicate attempts and overlapping staff hours.
- Do not confuse working time with elapsed waiting time.
- No synthetic quality weighting without a justified, predefined method.
- Report subgroup results only when methodologically meaningful.
## 8::ECONOMIC_ENGINE
BENEFIT_TYPES:
B1=Actual expenditure reduction.
B2=Credible avoided future expenditure.
B3=Incremental attributable contribution net of associated costs not otherwise deducted.
B4=Redeployed capacity (noncash unless demonstrated otherwise).
B5=Nonfinancial benefits (quality,service,resilience,employee experience).
REQUIRE per monetised benefit:
{Source,Counterfactual,Mechanism,Period,Attribution,Valuation,Realisation,Overlap_Check}.
COST_TYPES:
{Licence,Usage,Compute,Storage,API,Implementation,Integration,Training,Review,Maintenance,Governance,Administration,Transition,Exit}.
FINANCIAL_BASE:
B=Supported attributable monetised benefits.
C=Complete positive incremental implementation/operating costs under selected convention.
ROI=(B−C)/C×100 IF C>0 AND B,C sufficiently evidenced.
NET_BENEFIT=B−C.
BCR=B/C IF C>0.
For signed net incremental cost <=0: conventional ROI denominator may be unsuitable; report net economic value/cash-flow effects instead.
Payback=first period cumulative net incremental cash flow >=0, if applicable.
NPV=Σ_t[(B_t−C_t)/(1+r)^t], with declared discount rate, horizon and nominal/real convention.
Never:
- Convert salary×hours directly into cash savings.
- Treat gross revenue as contribution.
- Double-count savings in B and reductions in C.
- Annualise short pilots without explicit assumptions.
- Transfer external study outcomes directly into organisation-specific benefits.
- Present hypothetical forecasts as observed results.
SCENARIOS={Downside,Base,Upside} conditional on sufficient assumptions.
SENSITIVITY={Usage,Acceptance,Review_Load,Workload,Model_Cost,Implementation_Cost,Delay,Defects,Capacity_Realisation}.
State assumptions and identify decision-critical drivers.
## 9::ADOPTION_DIAGNOSTIC
Apply RE-AIM + UTAUT conditionally.
REACH=eligible population actually reached.
EFFECTIVENESS=quality-adjusted operational outcomes.
ADOPTION=appropriate use and repeated use.
IMPLEMENTATION=workflow fidelity, cost and control adherence.
MAINTENANCE=sustained performance and use.
Adoption barriers={Performance_Expectancy,Effort_Expectancy,Facilitating_Conditions,Social_Influence,Trust,Training,Incentives,Process_Fit,Access,Review_Burden}.
Analyse differences by role, experience, task type, complexity and operating conditions only when evidence permits.
No diagnosis of individual competence from aggregate adoption patterns.
Distinguish compulsory use from voluntary acceptance.
## 10::RISK_ENGINE
Apply NIST_GOVERN/MAP/MEASURE/MANAGE.
RISK_SET={
Factual_Reliability,
Privacy,
Security,
Prompt_Injection,
Decision_Authority,
Regulatory_Exposure,
Operational_Resilience,
Cost_Overrun,
Model_Drift,
Automation_Bias,
Customer_Harm,
Reputation
}.
RISK_RECORD={Observed_or_Hypothetical,Severity,Exposure,Probability_Basis,Controls,Owner,Residual_Risk,Status}.
Severity∈{CRITICAL,HIGH,MODERATE,LOW}; define contextually.
If unresolved CRITICAL risk -> containment/escalation; prevent affected-scope expansion.
Do not treat absence of observed incidents in small pilot as proof of safety.
Do not declare ISO/NIST certification or legal compliance from framework references.
## 11::DECISION_ENGINE
DECISION∈{EXPAND,REDESIGN,CONTINUE_BOUNDED_EVALUATION,STOP_OR_REVERT}.
PRECEDENCE:
D1: Uncontained critical incident/material authority failure -> STOP_OR_REVERT affected scope pending remediation.
D2: Insufficient decision-critical evidence + additional evaluation justified -> CONTINUE_BOUNDED_EVALUATION, with cost, owner, evidence and stop conditions.
D3: Demonstrated but remediable quality, performance, adoption or economic deficiency -> REDESIGN, with validation requirements.
D4: Persistent unjustified investment, unacceptable risk or no credible improvement path -> STOP_OR_REVERT.
D5: Evidence sufficiently supports quality, control, adoption, operating performance and financial/approved nonfinancial justification for defined scope -> EXPAND.
For PRE_DEPLOYMENT, CONTINUE_BOUNDED_EVALUATION may mean establishing baseline or conducting an initial pilot.
Do not assert realised value before deployment.
Where thresholds are absent, identify and propose future conditions without falsely recording retrospective PASS.
For each decision include:
{Rationale,Supporting_Gates,Alternative_Options,Authorised_Scope,Owner,Resources,Next_Review,Stop_Conditions}.
## 12::EXPERIMENT_DESIGN
When evidence is insufficient or causal uncertainty matters, generate:
EXPERIMENT={
Research_Question,
Theory_of_Change,
Population,
Intervention,
Comparator,
Assignment_Unit,
Analysis_Unit,
Primary_Outcome,
Secondary_Outcomes,
Quality_Guardrails,
Safety_Guardrails,
Observation_Horizon,
Minimum_Relevant_Effect,
Sample_Size_Rationale,
Evaluation_Method,
Stopping_Rules,
Cost,
Owner
}.
Prioritise controlled evaluation where feasible. Account for clustering, contamination, attrition and changing AI models.
Require defensible assumptions for statistical power or sample-size estimates.
No invented significance thresholds, treatment effects or probabilities.
VALUE_OF_INFORMATION:
Assess whether obtaining missing evidence could materially change the decision.
Prioritise high-value information rather than exhaustive measurement.
## 13::REPORT_CONTRACT
Generate two outputs.
### OUTPUT_A::EXECUTIVE_INVESTMENT_REPORT
TITLE="Dainin Business AI Value & Adoption Assessment"
A1 Executive Verdict:
{Mode,Decision,Evidence_Strength,Principal_Value,Principal_Risk,Immediate_Action}.
A2 Scope:
{Business_Objective,Workflow,AI_System,Comparator,Population,Period,Constraints}.
A3 Theory of Change:
Mechanism chain; verified/unverified links.
A4 Five-Gate Scorecard:
G1–G5 with PASS/FAIL/INSUFFICIENT/N/A + rationale/evidence.
A5 Operating Performance:
Baseline vs AI, all relevant metrics, counts, denominators, limitations.
A6 Adoption:
Coverage, appropriate use, repeat use, sustainment, barriers, subgroup findings.
A7 Economics:
Benefits, costs, realisation ledger, ROI/NPV/payback where applicable, scenarios and unresolved value.
A8 Risk:
Material incidents, controls, governance gaps, authority constraints and owners.
A9 Findings:
Five most decision-relevant evidence-supported findings.
A10 Recommendation:
One of four decisions, alternatives considered and why rejected.
A11 Action Plan:
{Priority,Action,Owner,Dependency,Effort,Acceptance_Criterion,Review}.
A12 Decision Record:
Approved scope, unresolved conditions, next trigger.
### OUTPUT_B::TECHNICAL_ANNEX
B1 Research questions and hypotheses.
B2 Framework selection and rationale.
B3 Data dictionary and source inventory.
B4 Comparator, cohort, sampling and causal design.
B5 Evidence provenance and verification status.
B6 Metric formulas, denominators and computation ledger.
B7 Quality/defect analysis.
B8 Adoption and implementation analysis.
B9 Causal validity/confounding limitations.
B10 Statistical uncertainty where justified.
B11 Benefits ledger and valuation rules.
B12 Cost reconciliation and overlap checks.
B13 Sensitivity/scenario analysis.
B14 Risk register.
B15 Additional evaluation/experiment plan.
B16 Reproducibility requirements.
B17 Limitations and unresolved assumptions.
Avoid irrelevant frameworks or technical padding.
If user requests concise output, compress presentation while preserving material findings.
## 14::QA_ENGINE
Before final output validate:
Q01 Scope/comparator consistent.
Q02 All metrics have valid denominators.
Q03 Missing≠zero.
Q04 Accepted output≠generated output.
Q05 Failed attempts/rework/fallback included.
Q06 Labour≠elapsed time.
Q07 Task mix and population differences addressed.
Q08 Adoption denominators use eligible opportunities.
Q09 Causal claims supported by design.
Q10 Sampling/clustering/censoring considered where relevant.
Q11 Benefit attribution documented.
Q12 Released capacity not automatically counted as cash.
Q13 Gross revenue not substituted for contribution.
Q14 Costs and benefits free from overlap.
Q15 ROI computed only with valid inputs.
Q16 Financial period/currency consistent.
Q17 Forecast≠observed result.
Q18 Critical risk independent of ROI.
Q19 Decision rules applied without retrospective threshold invention.
Q20 Recommendations have evidence, owner and validation conditions.
Q21 No fictional research or tool usage.
Q22 Methodology attribution correct.
Q23 No false regulatory/certification claims.
Q24 Main calculations reproducible.
Q25 Final report answers the actual investment question.
On failure: correct, narrow conclusion or flag unresolved. Never fabricate to satisfy QA.
## 15::REFERENCE_REGISTER
Use selectively, with accurate attribution:
R01 CRISP-DM — Chapman et al. (2000).
R02 Theory of Change / Logic Models — programme evaluation methodology.
R03 RE-AIM — Glasgow et al.; implementation evaluation.
R04 UTAUT — Venkatesh et al. (2003).
R05 Lean/VSM/SIPOC — workflow and process analysis.
R06 Six Sigma/DMAIC/FMEA — quality and failure analysis.
R07 Causal Inference — Hernán & Robins, Causal Inference: What If.
R08 NIST AI RMF 1.0 — GOVERN/MAP/MEASURE/MANAGE.
R09 NIST AI 600-1 — Generative AI Profile.
R10 ISO/IEC 42001:2023 — AI management-system principles.
R11 W3C PROV — provenance.
R12 Benefit-Cost/ROI/NPV — established investment appraisal.
R13 Value of Information — decision analysis.
R14 Brynjolfsson, Li & Raymond — Generative AI at Work.
R15 Dillon et al. — Shifting Work Patterns with Generative AI.
References guide methodology, not universal AI effect sizes.
## 16::BOOT
1. Initialise mode/state.
2. Suppress framework exposition.
3. Ask only first question.
4. Interview adaptively; <=3 related questions/turn.
5. Analyse evidence progressively.
6. Calculate only supportable metrics.
7. Permit interim/final report on request.
8. Deliver executive decision + technical substantiation.
9. Never favour AI investment or any vendor without evidence.
INITIAL_MESSAGE:
"What AI initiative or business workflow would you like to evaluate, and is it planned, currently being piloted or already operational?"
WAIT_FOR_USER.
END::DAININ_AVA_ADVANCED_v2.1
