We gave TypeSafe’s Jev 419 scanned pages, a two-line definition of a signature and no training. It said “signed” on every signed page and boxed most lone signatures within 5% of the page. It missed all three bars we set before the runs, and it couldn’t box several signatures on one page. However, in some use-cases, image quality, prompt specificity, and page simplicity could drastically increase detection.
Checking a document for a signature comes down to two small questions: is the page signed, and where is the signature? There are three common ways to answer them today. A detector trained for the job needs labelled data and a training run. A document API with a signature feature is ready to call, but Amazon Textract and Azure Document Intelligence publish no accuracy figures for it to plan around. A vision-capable LLM can be prompted to look, which means paying for open-ended text generation to get back a yes, a no and four coordinates.
Decision models offer a fourth way. They take a structured state and a question you define, and return a probability or a choice instead of generated text. In October, TypeSafe added image input and a Box question, which returns a bounding box, to Jev 1.14, its small, fast “System One” decision model. That turned signature detection into something we could ask Jev directly.
So we ran a lab: Jev as it ships, prompted only, with no signature-specific training and no fine-tuning, on 419 scanned pages. The results suggest it could take on part of the job on single-signature pages. It didn’t clear the bars we set, though, and on pages with several signatures it fell well short.
Context
Each Jev request carries a JSON state, here the page image, and one or more typed questions. We used three types. A Noul returns the probability that a statement is true. A Choice picks one option from a short list and gives a probability for each. A Box returns one bounding box with a confidence. Jev bills input tokens only, at $0.042 per million in TypeSafe’s published price when we ran.
Image input is still experimental: TypeSafe puts it behind an opt-in header and says so. Before using real pages we checked how the API handles images, using synthetic pages with scribbles. Jev accepted images between 280 and 1,568 pixels on each side and rejected anything outside that range. From the smallest to the largest accepted size, image tokens grew 27 times while pixels grew 31 times, so there was no sign of the image being shrunk to a fixed size before it was tokenised. We couldn’t see what happens inside the model after that.
The same check found something that shaped everything after it. A Box question returns exactly one box, every time. All 15 Box answers held one box, including on a synthetic page with three scribbled signatures where we asked for “all the handwritten signatures”, and on a page with none. There is no “not found” answer, so counting signatures, and deciding when there’s nothing to box, have to come from other questions.
What we tested
The pages. We used a 419-page test split of a public signature-detection dataset. 383 pages carry at least one labelled signature, 467 signatures in all, and 60 of those pages carry two or more. The other 36 have no labelled signature, and we checked all 36 by hand and found none signed. Every page is a 640×640 image as published, and we sent it at that size.
The questions. We sent one request per page with two questions. We fixed the wording before Jev saw any real page and never changed it.
Presence, a Noul: “Does
pagecontain a handwritten signature?” Short criteria said what counts (pen strokes forming a signature, usually cursive) and what doesn’t (printed or typed names, ‘/s/‘ marks, stamps, logos, cursive letterheads). A score of 0.5 or above counts as “signed”.Location, a Box: “a handwritten signature in
page“. A box is right if each of its four edges is within 5% of the page’s width or height of the matching edge of any labelled signature.
The bars. Before the runs, we set three pass thresholds for Jev on its own: balanced accuracy on presence of at least 0.95, a false-yes rate on unsigned pages of at most 5%, and at least 90% of boxes right. We also planned to run Amazon Textract’s signature feature on the same pages for comparison. A 40-page pilot came first, with a rule to stop if Jev was clearly failing. It didn’t trigger, and the remaining 379 pages ran with nothing changed.
After seeing those results, we added one exploratory run for pages with several signatures. It’s reported separately below and doesn’t change the scored result.
Results
Presence held up best. Jev said “signed” on all 383 signed pages. It also said “signed” on 6 of the 36 unsigned pages, a false-yes rate of 16.7%, which put balanced accuracy at 0.917.
Location was weaker. 284 of the 383 boxes (74.2%) were within 5% on every edge. On pages with one signature that rose to 255 of 323 (78.9%). On pages with two or more it fell to 29 of 60 (48.3%).
All 419 pages | Jev | Bar set before the runs | 95% interval |
Signed pages called signed | 383 / 383 (100%) | at least 95% | 100% to 100% |
Unsigned pages called signed | 6 / 36 (16.7%) | at most 5% | 5.6% to 30.6% |
Presence balanced accuracy | 0.917 | at least 0.95 | 0.847 to 0.972 |
Boxes right, all signed pages | 284 / 383 (74.2%) | at least 90% | 69.7% to 78.6% |
Boxes right, one signature | 255 / 323 (78.9%) | at least 90% | 74.4% to 83.4% |
Boxes right, two or more | 29 / 60 (48.3%) | no hypothesis | 35.6% to 61.4% |
Intervals are 95% bootstrap intervals over pages, resampling signed and unsigned pages separately.
Jev missed all three bars. The lab passes only if every bar holds.
The misses are uneven. On the 353 signed pages of the full run, the median page’s worst edge was off by 3.0% of the page, inside the 5% tolerance. Of the 90 boxes that missed, 22 were off by between 5% and 6%, and 53 by more than 10%.
The false yeses follow a pattern. On all six unsigned pages Jev flagged, its box sat on or near handwriting that isn’t a signing mark: a filing name, routing notes on two pages, a short flourish by a stamp, a faint scribble by a document number, and header initials. Our criteria said a signature could be “anywhere on the page” and didn’t mention notes or initials, so some of this comes from our wording. Two of the six looked borderline on review. Leaving those two out, a check we’d committed to before the full run, lifts balanced accuracy to 0.941 and lowers the false-yes rate to 4 of 34 (11.8%). Both are still short of their bars, and the check decides nothing.
Several signatures on one page. On the 60 multi-signature pages, Jev’s single box was right on 29. Of the other 31, it overlapped one signature but missed the tolerance on 22, spanned two or more signatures on 5, and missed every signature on 4. Because Box gives one box per question, we tried the approach TypeSafe’s documentation suggests for several objects. A count question (none, one or several) comes first, and on pages counted as several, the page is cut into four overlapping tiles and Jev is asked for a box in each.
The count question did well. It matched the label on 372 of 419 pages (88.8%) and called 53 of the 60 multi-signature pages “several”. The tiles didn’t help. Across all 467 labelled signatures, count-then-tile found 283 (60.6%), against 284 (60.8%) for the single page box. On the multi-signature pages it found 29 of 144 signatures, the same number as the single box. The tiles also returned more boxes, so precision fell from 284 of 389 (73.0%) to 287 of 575 (49.9%). It found every signature on 4 of the 60 multi-signature pages, where the single box never did, but found at least one on only 25, against 29. We haven’t examined why the tile boxes miss. Tiles cutting a signature at the edge is one possibility we didn’t check.
Cost and speed. The 379-page full run used 551,345 input tokens, about 1,450 per page, and cost $0.023 at the published price, or about six cents per thousand pages. Every run in the lab, including the image check and the exploratory run, came to $0.066 in total. On the full run the median response time was 0.35 seconds per page and the 95th percentile 0.89 seconds, measured from our machine including the network. Two caveats apply. We computed cost from the tokens the API reported rather than from a bill, and TypeSafe’s price page didn’t say whether image tokens are billed at the text rate.
Confidence. On the full run, Box confidence separated right boxes from wrong ones: a median of 0.85 on the 263 right boxes against 0.385 on the 90 wrong ones. We recorded it but didn’t use it, so we don’t yet know how well a confidence cut-off would work in practice. The presence score is less useful as a filter. The six false yeses scored between 0.66 and 0.96, inside the range of the signed pages (0.69 to 0.99), so no single threshold removes all six without losing signed pages too.
Limitations
We didn’t run an LLM. The comparison the “replace the LLM” claim needs — a vision-capable LLM on the same 419 pages under the same scoring — isn’t in this lab.
We didn’t measure scan quality or signature position. Every page was 640×640, we never varied resolution, and we didn’t tag pages as clean or noisy, or by where the signature sits.
The unsigned sample is small. With 36 unsigned pages, the false-yes interval runs from 5.6% to 30.6%. One person hand-checked them. The dataset’s labels aren’t perfect either: we found a page with two signatures on signature lines and only one labelled.
We didn’t tune the prompt. The wording was fixed before Jev saw a real page. That keeps the result clean, but we don’t know how much better wording would recover, especially on the false yeses.
Image input is experimental. It needs an opt-in header, and when we ran, TypeSafe said it couldn’t yet vouch for it in production.
Verification wasn’t tested. The lab had a second part, telling genuine signatures from forgeries given one reference. We didn’t run it, and this lab makes no conclusions about verification.
Where might Jev be useful for signatures?
These uses fit what we measured, but we haven’t tested any of them end to end.
A first-pass “is anything signed here?” gate, where missing a signed page costs more than a false flag. Jev missed none of the 383 signed pages, and its 6 false flags fell on pages with other handwriting, which a person can clear quickly.
Picking out pages that need another path. The count question called 53 of the 60 multi-signature pages “several”, even though Jev couldn’t box every signature on them. Those pages could go to a person or another tool.
A location hint on single-signature pages, for a reviewer to confirm. 79% of those boxes were within 5%, and Box confidence may show which ones to check first.
Alternatives, and how we’d choose
A vision LLM. Its answer is free text, so it isn’t limited to one box per question.
A detector trained for signatures. The dataset’s publisher also publishes a trained detector with results on a 419-image test split, under a different metric (mAP at IoU 0.5). We didn’t run it. The trade-off is the one Jev avoids: labelled data, a training run, and retraining when your documents change.
A document API with a signature feature, such as AWS Textract or Azure Document Intelligence.
Fine-tuning a vision LLM, or training a custom model with reinforcement learning. Not attempted. The lab set out to see how far prompting alone gets.
How we’d choose:
If you need every signature boxed on multi-signature pages, Jev as we used it isn’t the answer. Test a trained detector or a vision LLM first.
If you need a fast, cheap signed/unsigned flag with a person reviewing what it flags, Jev is worth testing on a labelled sample of your own pages. It’s one request per page, with no training step.
Either way, test on your own documents. Our numbers come from one public dataset at one resolution with a generic prompt.
Where this leaves us
With prompting alone, Jev called every signed page we gave it signed, and placed a single signature within 5% on about four pages in five. It did that for about six cents per thousand pages, with a median response of 0.35 seconds. It didn’t meet the bars we set. Its false yeses came from handwriting that isn’t a signature, and it couldn’t box several signatures on one page, even with tiles.
That makes it plausible that a decision model could replace a vision LLM as the signature detector, for a narrower job than we started with: single-signature pages, presence first, with a person or a confidence cut-off catching the misses. We have no LLM figures on these pages, and nothing in our data isolates scan quality or signature position. “It works when the scan is good and the signature is where you’d expect” is a hypothesis for the next lab, not a finding of this one.
What we’d test next:
A vision LLM on the same 419 pages with the same scoring, recording cost and latency, so the comparison is head to head.
Pages tagged by scan quality and signature position, scored per group, to test that condition directly.
Larger images, up to 1,568 pixels a side, to see whether resolution moves the box score.
Presence criteria narrowed to the signing mark, and Box confidence as a cut-off.
Sources
Tell us what you're working on.
