Is the model ready to help inspect production?
The fictional Alder Paper Works pilots AI inspection on printed instruction cards.
The question
The pilot must detect at least 95% of known defects in each design. Each flag needs review, and the team can review 30 flags per 500 units in a production-length trial. The existing quality check stays in place. The quality owner makes release decisions.
| Known defects | Normal units | |
|---|---|---|
| Units in the labelled set | 20 | 980 |
| Units flagged by the model | 15 | 10 |
This trial uses 1,000 labelled units.
“Evaluate the pilot and decide what the quality team needs to see before it advances.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What does the model detect?
What would you do here? Open the move you’d make.
Use correctly classified units to measure defect detection
Tempting, but it misses something. That measures overall accuracy. Detection uses only the known defects in its denominator.
Divide known defects flagged by all known defects
The strong move. It found 15 of 20, which is 75% detection.
Count all 25 flags as defects found
Tempting, but it misses something. Ten flagged units were normal according to the labelled set.
See how a strong candidate says it
“It finds 15 of 20 known defects, or 75%. It misses five. It correctly leaves 970 normal units unflagged, which gives 98.5% accuracy across all units.”
Why it works: With many normal units, the overall accuracy can stay high even when the model misses defects.
Step 2: What do you recommend after this pilot?
What would you do here? Open the move you’d make.
Advance based on the 98.5% overall accuracy
Tempting, but it misses something. The stated detection requirement is 95% of known defects per design.
Keep the existing check and test both criteria
The strong move. The quality owner retains the decision and gets measurements on detection and workload.
Reduce flags first and ignore missed defects until launch
Tempting, but it misses something. Fewer flags can reduce workload while leaving more defects undetected.
See how a strong candidate says it
“The model needs to find at least 19 of these 20 defects to meet the detection target. I would keep the current check, run the model alongside it and record detection and review volume by design.”
Why it works: The opening sample misses the detection target. The team needs evidence on both criteria.
Step 3: What does the new design show?
A separate 500-unit labelled trial uses a new design. It has 10 known defects. The model finds 5 and flags 8 normal units.
What would you do here? Open the move you’d make.
Keep the result for the new design separate
The strong move. Five of 10 known defects were found. The decision uses that design’s result.
Pool the two samples and report only overall accuracy
Tempting, but it misses something. The larger sample can hide the miss rate on the new design.
Advance because the team can review all 13 flags
Tempting, but it misses something. This meets the workload limit, but misses half the known defects.
See how a strong candidate says it
“The new design has 50% detection, 13 flags and 97.4% overall accuracy. I would keep that result separate because it shows how often this design’s defects are missed.”
Why it works: The new design is below target even though the trial generates fewer than 30 flags.
Step 4: Does the revised threshold pass?
A separate 500-unit test on the new design contains 20 known defects. At a revised threshold, the model flags 19 of them and 24 normal units.
What would you do here? Open the move you’d make.
Advance because the detection target is met
Tempting, but it misses something. There are 13 more flags than the team can review in the stated window.
Review just the 19 known defects and discard the other flags
Tempting, but it misses something. The trial labels identify the normal units after the fact. In use, each flag needs a review.
Keep the existing check and retest detection and review volume together
The strong move. This keeps the release decision with the quality owner until both criteria are met.
See how a strong candidate says it
“It finds 19 of 20 defects, or 95%, in this trial. It sends 43 units for review, 13 above capacity. I would keep the existing check and ask for a trial that meets both requirements.”
Why it works: The revised threshold meets detection in this sample. It needs more review time than the team has.
The answer, in short
- Decision
- Require both detection and review capacity in the quality decision.
- Because
- The threshold changes how many defects the model finds and how much work its flags create.
- What would change it
- A representative test that meets both criteria would support the next quality review.
Common traps
- Use all known defects to calculate detection.
- Count every flagged unit in review workload.
Check your answer
- I separated accuracy and detection.
- I kept design results separate.
- I counted all flags.
- I explained what evidence is needed next.
Last reviewed Nov 11, 2026