Pattern 02 · During
The Glass Box
The machine shows its reasoning, not just its output, so operators can trust or override it with context.
The mechanism
inputs → decision logged
- 01InputsSensors, history, thresholds
- 02InferenceModel weighs the evidence
- 03 · Key momentRecommend + why“X, because of A, B and C”
- 04Operator judgesAgree, question or override
- 05Decision loggedWith the reasons behind it
At 03: Surface the drivers: sensor confidence, historical match, and which threshold was breached.
At 05: Overrides feed back, so disagreement becomes signal instead of noise.
Without / with the pattern
Without
Black-box model, then “Do X”, then Blind accept or ignore
Operators either rubber-stamp or distrust everything. Neither is calibrated trust.
With the pattern
Model + evidence, then “X, because…”, then Informed decision
Operators check the reasoning against what they see, and override when the machine is missing something.
Related patterns
- Complements · 08Provenance LogThe Glass Box logs the operator's decision and the reasons for it. This pattern covers every other state change, including the ones no operator made.
- Complements · 09Learning LoopLogged overrides are the richest input this loop has: repeated disagreement in the same situation is the clearest sign the machine is wrong.
Sources
An explanation only helps if the operator can check it against something they can see for themselves. Dumping the whole model into the interface produces the same rubber-stamping as showing nothing, so the reasons have to be few and specific to the decision in front of them. Recording what operators override matters as much as showing the reasons, because repeated disagreement in the same situation is the clearest sign the model is wrong.