Act · component measurement

Execution integrity

The other stages of this program ask whether a fact survived being communicated, stored and resolved. This one asks the question that only arises once an agent can do things: whether a claim is permitted to become an action.

Yao Tsakpo · Independent research

The gap

A pipeline that ends in an action has a seam nobody owns. A model produces a judgement; something downstream carries it out. In between, nothing asks whether the judgement earned the right to be carried out. The judgement is the authority, and a wrong one becomes a wrong act with no record that anything was ever uncertain.

This is the same shape as the earlier measurements in this program. A flat memory store treats recency as authority; a provenance store makes authority explicit. Here a classifier’s output is treated as authority; the measurement asks what changes when it is not.

The setting

Industrial visual inspection, which has the virtue of being a problem the field already admits it has. A camera sees a part, an anomaly model scores it, a robot arm places it in a good or reject bin. Any threshold dividing the two creates a documented tradeoff: tighten it and good parts are scrapped, loosen it and defects reach the customer, which is usually the more expensive error because everything later built on the part inherits the fault.

The tension is unavoidable as posed, because the score distributions for good and defective parts genuinely overlap. No tuning recovers information that is not there. It is also easy to miss: on the standard benchmark, pixel-level AUROC can exceed 0.99 by predicting that nothing is defective at all, so a headline number can look excellent while defects pass.

The intervention

The premise under the tradeoff is that every part must go to one of two bins, so the overlap has to be guessed at. The intervention is a third outcome. A boundary sits between the model and the arm and asks whether this judgement has standing to move this part. Where it does not, the placement is not performed: the part is returned rather than binned, and the reason is recorded.

Standing is derived at that boundary from evidence rather than read off the claim: how far the score sits from the decision boundary, whether the scene matches the conditions the model was trained under, whether the part resembles anything in the training set. A confident score means little if the lighting has changed.

What it measures

A gate cannot make a model more accurate, so it cannot reduce both kinds of error for free. What it does is convert some of them into deferrals, and the honest way to present that is the whole curve rather than one flattering point. Sweeping the floor over a score distribution with a realistic overlap:

Read it as a tradeoff rather than a win. The first row is the pipeline as it works today. A modest floor halves the defects reaching the customer at the cost of deferring a tenth of the parts; a strict one stops them entirely and defers nearly half. No setting is simply best, because the right floor depends on what a shipped defect costs relative to a person’s time. The contribution is that the choice becomes available at all.

What this is not

Abstaining when uncertain is well established: it is the reject option, or selective prediction, with a 2024 survey in Machine Learning, and the two checks used here are its textbook ambiguity and novelty categories. Nothing about deferring to a human is new.

What differs is where the decision lives and what it governs. A reject option sits inside a classifier and decides whether to answer. This sits at the boundary before the actuators and decides whether an answer becomes a motion, across a pipeline of several models none of which owns that seam. It also applies to a model you did not train and cannot modify, which is the normal case once agents compose tools they did not build.

The limit is worth stating plainly: a boundary of this kind cannot catch a confidently wrong model. It governs authority, not truth. What it catches is uncertainty, conditions outside training, and inputs unlike anything seen before, which are the realistic failure modes rather than the adversarial worst case.

Why it belongs to this program

The frametreats each stage of an agent’s context as separately testable. The communicate and store/resolve stages were measured on information. This one is measured on a physical action, in a domain with no email in it, which is the point: if the property only held where it was first noticed, it would be a feature rather than a property. The reference implementation is Inbin, where the same boundary governs whether an email may be acted on.