PoC and Operations: Making AI Deployment Succeed

PoC verifies more than whether the technology works

A PoC is the pre-deployment validation step — confirming whether a concept holds up under real conditions. With AI, a natural-sounding response in a demo does not guarantee it can handle everyday business documents, the actual number of users, or edge cases. Decide what to verify and what result allows moving forward before you begin.

The NIST AI Risk Management Framework also covers risk management that includes not just AI design and development, but also use and evaluation. The perspective of reviewing throughout operation — not just a one-time judgement at deployment — is a useful reference.

In an enterprise PoC, make the target concrete: "reduce the verification time for this task in this workflow." Measuring current time and revision volume before AI adoption means the change after adoption can be assessed against the same baseline. The goal is to create material for deciding whether to continue — not just to confirm expectations.

Decide pass criteria in advance across six dimensions

This concept verifies six areas: missed confidential information, routing accuracy, response performance, realistic load, business impact including audit, and operations and security. This structure prevents judging success solely by GPU utilisation.

For missed confidential information, look not just at "how many were correctly stopped" but also at the impact if information is incorrectly sent externally. Stopping too much normal information prevents work from progressing, so false positives and their verification burden are also recorded. Preparing evaluation data with known correct answers and reviewing results by type of confidential information reveals specific issues.

For response performance, check not just the average but also how often slower responses occur. p95 is the value below which 95% of measured values fall. Specify whether the time covers only classification, or includes search and generation, and avoid comparing different measurements as a single figure.

Test not just normal requests, but also difficult scenarios

Verification should include short questions, long documents, attachments, images, multi-turn conversations, and documents the user is not permitted to view. Testing only a small number of familiar questions misses the difficult conditions encountered in production. Preparing evaluation data separate from what was used in development is also important.

Fault testing is also necessary. Confirm how information handling and screen display change when an external API is unresponsive, the classification function stops, the search target is unreachable, or a user's permissions change. Even when the system stops for safety reasons, decide what the field should do instead.

Record not just pass/fail counts but also conditions and the version of settings. If the same tests can be re-run when the model or search targets are updated, it is possible to confirm that things have not gotten worse. PoC records also become the verification procedures for production operations.

View costs from purchase through operations in the same table

The price of equipment alone does not cover the full scope of deployment costs. Confirm each item separately: hardware, initial setup, data preparation, integration with authentication and business systems, operations and maintenance, external APIs, power and cooling, and hardware maintenance. Costs vary by configuration and conditions — please enquire individually.

When combining cloud and on-premises AI, do not simply conclude that one is cheaper. Align usage volume, peaks, processing length, and operational staff burden for comparison. Dividing the total by headcount alone obscures actual utilisation rates and management effort.

At contract time, confirm what is covered in the initial engagement, what incurs additional costs, and who to contact during a fault. Including recurring post-deployment tasks — model updates, adding documents, permission changes, re-running evaluations — in the scope of estimates reduces misaligned expectations after operations begin.

Establish responsibility allocation and decision points at each stage

An AI platform involves business decision-makers, software teams, authentication and network administrators, and hardware and facilities maintenance staff. Without deciding who gives final approval, response stops when problems occur.

Even in concepts involving WOODMAN, the roles of compute and immersion facilities, AI applications, and business evaluation are organised separately. Specific scope of provision and responsibility is confirmed per project — the conceptual role diagram must not be treated as a confirmed contract.

The approach is divided into stages: PoC with a small target, trial with limited users, and expansion to production scope. Each stage has conditions for proceeding and conditions for review. The decision to expand is not made because a scheduled date has arrived, but because evaluation results and operational readiness are confirmed.

Connect items that did not pass to the next decision

PoC results may fall short of expectations. Rather than evaluating the concept as a binary success or failure, separate out which conditions were difficult and why. Whether the cause is outdated documents, search quality, model response quality, or user operation determines what work is needed next.

Serious issues related to confidential information handling must be addressed before expanding scope. When responses are slow, review input volume, search scope, concurrent processing, and queue management before moving to hardware additions. Verify the effect of improvements through re-testing under aligned conditions.

Pass criteria should include not just numbers but also operational feasibility. Is the contact point decided? Can the responsible person execute the procedure? Can the field be notified when the system stops for maintenance? Having the people who will manage day-to-day operations participate in the trial surfaces burdens that were not visible at design time.

If costs and benefits do not align, reducing the target task scope, switching to a different configuration, or deciding not to proceed are all valid outcomes. Position the PoC as a step for gathering the material needed to make a decision — not as a commitment to purchase.

The report should cover the target task, data used, configuration, evaluation results, remaining issues, and the next responsible person. Recording the technical explanation and business impact together means the reasoning behind decisions can be handed over after deployment.

Maintain the ability to review even after going live

Reference 1: NIST AI Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework