How it was measured
The two figures we publish, and the six things they do not cover.
We say PAIR is 2.6 times faster at filling a form and gets 95 % of the fields right. This page says what that was measured on, with what method, and which questions we still cannot answer.
What we publish#
2.6×
Faster at filling the form, with the time spent correcting what the AI got wrong already deducted. Between 2.4 and 2.75 depending on the end of the interval.
95%
Of the fillable fields, correct against the same form filled by hand by an inspector. 95 % confidence interval: 91.9 % to 96.9 %.
The measurement, as it happened#
Three hundred forms, created from three hundred photographs. The comparison standard was those same forms filled by hand by a real inspector.
The conditions: in the morning, in good light, on a Samsung S24, in Spanish, over industrial structures and buildings. Mostly conveyor belt inspection. The sample came from two customers and two form types: we took the ones that were there.
The unit of error is the fillable field. A field counts as an error only if the photograph carried enough information and the real inspector was able to fill it. If the information was insufficient for both, the field is excluded from the calculation.
Why the wide interval and not the narrow one#
At 95 % accuracy, the Wilson interval depends on what you count as an observation. If each form is one observation, with n=300, the interval runs from 91.9 % to 96.9 %. If you count each field separately, at around 4,500, it narrows to 94.3 % – 95.6 %.
We publish the wide one. The narrow one is four times more flattering and is the one that would be wrong: it assumes those 4,500 fields are independent observations, and they are not. They come from three hundred photographs. A backlit photo ruins ten fields at once and a sharp one gets all thirty: multiplying probabilities as if they were coin tosses assumes independence where there is a common cause, and that cause has a name.
Counting each form as a single observation means assuming maximum correlation within the form. It is the conservative end, and it is the one that survives a question.
What that means on a real form#
A percentage says nothing on a site. What says something is how many fields you have to fix on a form with fifteen fillable fields.
| Accuracy | Fields to fix | Time advantage |
|---|---|---|
| 91.9 % · low end | 1.21 | 2.4 × |
| 95.0 % · centre | 0.75 | 2.6 × |
| 96.9 % · high end | 0.46 | 2.75 × |
[ the table continues to the right ]
The time advantage already deducts the correction: if a form arrives with 0.75 fields to fix, fixing them costs time, and that time is part of the cycle. The undeducted 3 exists; the 2.6 is the one that survives somebody doing the subtraction in front of you.
The six limits#
None of these six invalidates the measurement. All six limit what it can be extrapolated to, which is why they are written here and not in a footnote.
- Convenience sample. Two customers and two form types: we took the ones that were there. It is not a random sample of industrial inspection forms, and the interval holds for what was measured, not for any form from any customer.
- A single reference rater. The standard was one inspector. With a single rater you cannot separate model error from disagreement between people: some of that 5 % may be a field where two experts would have written different things.
- Favourable lighting conditions. Morning and good light. It does not cover night shift, rain, airborne dust or backlighting — and backlighting at midday in northern Chile is the norm, not the exception.
- A high-end phone. Samsung S24. There is no measurement on a mid-range device, which is what a whole crew usually carries.
- One language and one domain. Spanish, and industrial structures and buildings, mostly conveyor belts. No measurement on gauges and digital readouts, handwriting, serial number plates or already-filled paper forms.
- The field test record was lost. The test at two northern mine sites happened and has no documentary backup, so no figure from it is published: no hours, no shifts, no battery life. The offline mode is argued from what is written in the source code.
A correction we made ourselves#
The first version of this analysis calculated an arithmetic ceiling of 1.67× for the time advantage, assuming the 3× was for the whole cycle: going out, filling, transcribing, drafting and delivering the report.
It was not. The 3× is fill against fill, over the same three hundred forms, with the app working on a poor connection. The 1.67× ceiling does not apply to this comparison and was withdrawn.
We leave it written down because a page that only publishes the calculations that suit it is not a method page.
What we will measure next, by return on effort#
- A second inspector over the same three hundred forms. It is the cheapest thing there is and it closes the second limit: it separates model error from human disagreement.
- Storing the per-field count. With per-field data you can calculate the real design effect, and the interval no longer has to pick the worse of two.
- A hundred forms backlit and a hundred at night. It closes the third limit, which is the one that most resembles the actual site.
- Measuring transcription and reporting. Filling contributes around 6 % of the whole-cycle saving; transcription and reporting contribute close to 80 %, and they are not measured. Arguing about the 2.6× is arguing about the 6 %.
Of the six competitors we studied, none cites a source for its benefit figures. Not one. Publishing the method with its six limits is not decorative honesty: it is the only thing on this page that cannot be copied in an afternoon, because copying it requires having measured.
Demo
Measure it yourself, with your forms
The way to know whether these figures apply to your case is to measure it with your forms and your conditions. Bring two or three to the demo and we will do it with a stopwatch.
- Would you rather look at the plans first? See pricing.
- Still have questions? Check the FAQ.