Writing success measures before you build

Most prototypes end in one of two ways. Either the team agrees it worked and moves forward, or a long, uncomfortable debate begins about whether the results are good enough. The second outcome is far more common, and it is almost always avoidable.

The cause is simple: nobody agreed beforehand how the prototype would be judged. Once results arrive, enthusiasts see promise and sceptics see shortfalls, and both are arguing from the same numbers. This guide sets out how to write success measures before any build starts, so the decision at the end is quick and fair.

Why measures must come first

Measures written after the results are known tend to be shaped by them, consciously or not. A team that achieves a 20% time saving will argue that 20% was always the goal. A sponsor who wanted 50% will say so. Neither can prove their position.

Writing measures first also sharpens the prototype itself. When the team knows it will be judged on cycle time, it designs the test to capture cycle time cleanly. When it knows errors matter, it builds a way to count them. The measure shapes the work in a useful way.

There is a further benefit. Agreed measures give the delivery team permission to say no to scope creep. When a stakeholder asks for an extra feature halfway through, the team can ask a simple question: will this help us move the primary measure within the test period? If not, it waits. That discipline keeps prototypes short, which is the whole point of prototyping.

Step 1: Pick the metric

Choose one primary measure that reflects the business value you expect. It should be something the sponsor already cares about and can explain to their own leadership without a glossary.

• Hours saved: time spent by staff on a task before and after, per case or per week.
• Errors avoided: the rate of mistakes, rework or exceptions in the output.
• Cycle time: elapsed time from request to completion, such as quote turnaround.
• Throughput: volume handled per person or per day without extra staff.
• Quality score: the proportion of outputs a reviewer accepts without changes.

Add one or two secondary measures to catch unintended effects. If the primary measure is hours saved, a sensible secondary measure is error rate, so that speed is not bought at the cost of accuracy. Keep the list short. A prototype judged on eight measures will produce a mixed picture that invites the very debate you are trying to avoid.

Be precise about how each measure will be captured. “Hours saved” sounds clear until you ask whether it means time logged in a system, time estimated by staff at the end of the week or time observed by someone with a stopwatch. Each method has strengths. System logs are objective but miss work done outside the system. Staff estimates are easy to collect but tend to drift. Direct observation is accurate but expensive. Choose one, document it and use the same method for the baseline and the prototype.

Step 2: Set the threshold

Decide what result would justify going further. The threshold should be grounded in economics, not optimism. Work backwards from the investment the next stage would need.

Suppose a team of twelve spends an average of six hours a week each preparing supplier reports. Moving to a production-ready tool would cost a known amount in build, licences and support. Calculate the saving needed to recover that within an acceptable period, perhaps eighteen months. That figure, expressed as hours saved per person per week, becomes your threshold.

It is often useful to define three bands rather than a single line:

1. Proceed: the result clearly meets or beats the threshold. Move to MVP planning.
2. Refine: the result is promising but below threshold, with a clear, specific reason to believe a further short iteration will close the gap.
3. Stop: the result is well below threshold, or the refine band has already been used once.

Step 3: Decide the stop rule

This is the hardest part and the most valuable. Say in advance what result would end the project. Write it down in plain terms: “If the prototype saves less than two hours per person per week after four weeks of use, we will stop.”

Stop rules feel negative, but they protect everyone. They protect the sponsor from sinking more money into a weak idea. They protect the team from endless extensions that drain morale. And they protect the organisation’s appetite for experimentation, because a project stopped cleanly is a lesson learned rather than a failure to explain.

A stop rule should also cover time and money, not just results. Set a maximum duration and a maximum spend for the prototype phase. If either is reached before the decision meeting, the project pauses for a deliberate decision rather than continuing by default.

The stop rule was the most useful line in the brief. When the first idea fell short we closed it in a week and moved the team to the next one.

Head of operations, insurance client

Step 4: Write it on one page

Bring the measures together into a single page that the sponsor signs before work begins. It should include:

• The problem being tested, in one or two sentences.
• The primary measure and how it will be captured.
• One or two secondary measures and their acceptable limits.
• The baseline figure and the date it was measured.
• The proceed, refine and stop thresholds.
• The test period, the users involved and the date of the decision meeting.

Share the page with everyone involved. When results arrive, the decision meeting should take less than an hour, because the criteria are already agreed.

Common pitfalls

Choosing a measure that the prototype cannot influence within the test period is a frequent mistake. Revenue growth, for example, often takes too long to show. Pick something closer to the process being changed.

Testing with unrepresentative users is another. If only the most enthusiastic staff try the prototype, results will flatter it. Include a mix of experience levels and at least one sceptic. Finally, avoid moving the thresholds once results arrive. If new information genuinely changes the economics, record why and get the sponsor to approve the change in writing.

A subtler pitfall is measuring the prototype in conditions that will not exist in production. A prototype that relies on a team member manually cleaning input data each morning may perform well, but that hidden effort will not scale. Record any manual workarounds used during the test and factor their cost into the decision. The question is not whether the prototype worked, but whether a production version could deliver the same result at a sustainable cost.

Getting started

For your next prototype, hold a 45-minute session with the sponsor before any build begins. Agree the problem, pick one primary measure, set the three bands and write the stop rule. Start measuring the baseline the same week. It is a small investment that turns the end of every prototype into a clear decision.

Source: Intelliversal analysis.

Frequently asked questions

What if we do not know what threshold to set?

Work backwards from the cost of the next stage. Estimate what building and running a production version would cost, then calculate the improvement needed to recover that in an acceptable period.

Can we change the success measures during the prototype?

Only if new information genuinely changes the economics, and only with the sponsor's written approval and a recorded reason. Changing them because results are disappointing defeats the purpose.

How long should a prototype test period be?

Long enough to capture a representative sample of the work, usually two to six weeks. The period should be agreed alongside the measures.

Explore AI Prototyping and MVP Development