---
title: "TypeSafe's Jev Can't See. I Made It Guess What I Drew Anyway"
description: "I turned 400 hand-drawn sketches into text and asked Jev, TypeSafe's text-only System One model, to name them. It beat chance by a wide margin and lost to Claude Sonnet 5. It also answered airplane for more than half of them."
author: "Bartosz Mikulski"
author_bio: "AI & MLOps Engineer. I bridge the gap between \"it works in a notebook\" and \"it works for 200 million users.\""
author_url: https://mikulskibartosz.name
author_linkedin: https://www.linkedin.com/in/mikulskibartosz/
author_github: https://github.com/mikulskibartosz
canonical_url: https://mikulskibartosz.name/typesafe-jev-guess-what-i-drew
---

I tried to draw a bicycle using a mouse, with photos of bicycles open on a second screen. After the third attempt, I deleted *bicycle* from the game. I changed the game's list of objects three times, always for the same reason. I can't draw.

The other player had a bigger problem. It can't see.

The other player was Jev, a model made by TypeSafe. Jev reads text and nothing else, at least for now. [TypeSafe's documentation](https://docs.typesafe.ai/concepts/system-one) says images, audio, and video are not supported yet. So I turned every drawing into text and made Jev guess what I drew anyway.

## Why I made a text model look at pictures

I keep seeing posts about Jev on X. Tech bros describe it as a cheaper, faster classifier and show off ideas they came up with. My reaction: "Dude, I could have done that five years ago with XGBoost and BERT."

Their ideas undersell the model. With XGBoost or a fine-tuned BERT, every new question meant a new labeled dataset and a new training run. With Jev, I describe the classifier I want in plain English and get an answer from a single API call. I don't collect labels, and I don't train anything. Neither XGBoost nor BERT gave me that.

TypeSafe calls Jev a System One model, after the fast, intuitive System 1 that Daniel Kahneman popularized. Jev doesn't write text. You send it some text and a question, and it answers in a fixed format. For a multiple-choice question like mine, it returns a probability for every answer.

Guessing a doodle is the most System 1 game I know. Nobody at a party reasons their way to *cat*. You squint at the doodle through the beer and shout. Preferably without spilling it all over yourself.

TypeSafe also publishes a list of [Jev's known weak spots](https://docs.typesafe.ai/model-jaggedness/jev-1.13). One of them says Jev handles words better than numbers and can't reliably tell whether two hex colors are close to each other. A drawing turned into text is a pile of numbers. I was about to test Jev, on purpose, on the kind of input its makers had warned me about.

## What Jev sees

The game is a small web app. I draw on a canvas with the mouse, and the app squeezes my strokes into a 100 by 100 box. It drops the points that barely change the shape (the Ramer-Douglas-Peucker algorithm does that part). Then it writes the result as an SVG, one `<path>` per stroke. That string of coordinates is everything Jev gets from the drawing.

Next to the drawing, Jev gets the names of ten objects and one line describing each. For *airplane*, the line says: a long thin body with two swept-back wings and a small upright tail fin. Those ten sentences and a one-line question are the whole classifier.

![Manual play with an early version of the object list. My drawing at the top, the simplified SVG Jev reads below it, and Jev's probability for every object on the right.](/images/2026-09-19-typesafe-jev-guess-what-i-drew/manual_flower.png)

In the game, Jev ranked my flower, my cat, and my tree first.

![Manual play, the same early object list. A cat drawn with a mouse by someone who can't draw.](/images/2026-09-19-typesafe-jev-guess-what-i-drew/manual_cat.png)

![Manual play, a later version of the game with 40 objects to choose from. The chart shows only some of them.](/images/2026-09-19-typesafe-jev-guess-what-i-drew/manual_tree.png)

None of it proved anything.

I made somewhere between 20 and 30 drawings. The flower and the cat come from a different object list than the tree, and neither list is the one I tested later. A game where I delete every object I fail to draw makes a nice demo. For numbers, I needed drawings I didn't make and rules I couldn't change halfway through.

## The test: 400 drawings by strangers

For the test, I used the [Quick, Draw!](https://github.com/googlecreativelab/quickdraw-dataset) dataset from Google: doodles made by players of the Quick, Draw! game, each drawn in under 20 seconds. I picked ten categories of objects because I can draw them (more or less).

From each category, I sampled 40 drawings with a fixed random seed, 400 in total. I only sampled drawings that Google's own recognizer had classified correctly. Every number below describes sketches that were already recognizable.

The drawings went through the same code as the game, so the test measures exactly what the game sends to Jev.

Four setups saw all 400 drawings:

- Jev reading the SVG, exactly as in the game.
- Jev reading the PNG as base64 text, which is the picture itself spelled out in characters. The median drawing takes 336 characters as SVG and 3,018 as base64.
- Claude Sonnet 5 looking at the actual image. A model that can see sets the ceiling.
- Claude Sonnet 5 reading the same SVG as Jev. The fair fight.

## The rules I wrote before the run

I wrote the analysis plan before the full run. I didn't want to see the results first and then pick the test that flattered them.

Every accuracy comes with a range, the 95% [Wilson](https://mikulskibartosz.name/wilson-score-in-python-example) confidence interval. The range shows which true accuracies are consistent with what I measured on 400 drawings.

First, each setup had to beat guessing. With ten objects, a random guess is right 10% of the time, so I checked how often pure guessing would score at least as well as each setup. That's an exact binomial test against 10%.

The main question was whether Jev is as good as Sonnet when both read the same SVG, and I defined *as good* in advance. Jev was allowed to trail Sonnet by up to 5 percentage points. If even the worst end of the range for the difference stayed within those 5 points, I would call Jev non-inferior, meaning no more than 5 points worse.

I also compared all six pairs of setups. They all saw the same 400 drawings, so the comparison goes drawing by drawing. Drawings both setups got right, or both got wrong, say nothing about which one is better. Only the disagreements count, and McNemar's exact test looks only at those. Six comparisons give luck enough chances to make one of them look impressive, so I adjusted the p-values with the Holm method.

I only scored right or wrong. Both models report a confidence number, but Sonnet's is self-reported, and an LLM can write down any number. Confidence stayed out of every test.

## The results

![Accuracy of each setup on the same 400 drawings. Error bars show the 95% confidence interval. The dashed line is chance.](/images/2026-09-19-typesafe-jev-guess-what-i-drew/accuracy_by_variant.png)

- Sonnet looking at the image: about 91% (88% to 93%).
- Sonnet reading the SVG: about 57% (52% to 62%).
- Jev reading the SVG: about 35% (31% to 40%).
- Jev reading base64: about 9% (6% to 12%).

Three setups beat guessing by a wide margin (p < 0.001 for each). Jev reading base64 did not.

Reading the same SVG, Jev is 21.8 points behind Sonnet, and the plausible range runs from 16 to 28 points behind. My margin was 5, so the answer to the main question is a clear no. Jev doesn't compete with Sonnet on accuracy here.

All six pairwise differences held up after the correction.

![Paired differences in accuracy between every two setups, with 95% confidence intervals. No interval crosses zero.](/images/2026-09-19-typesafe-jev-guess-what-i-drew/paired_differences.png)

## Half of everything is an airplane

I opened the [confusion matrix](https://mikulskibartosz.name/how-to-read-a-confusion-matrix), and the 35% stopped meaning what I thought it meant.

![Jev reading the SVG. Rows show the real object, columns show Jev's answer. Look at the first column.](/images/2026-09-19-typesafe-jev-guess-what-i-drew/confusion_jev_svg.png)

Jev answered *airplane* for 209 of the 400 drawings. More than half.

Its accuracy per object:

| Object | Correct |
|---|---|
| airplane | 34/40 |
| skull | 28/40 |
| sailboat | 27/40 |
| tree | 25/40 |
| fish | 13/40 |
| sun | 8/40 |
| house | 3/40 |
| flower | 3/40 |
| clock | 1/40 |
| cat | 0/40 |

Jev ranked my hand-drawn cat first in the game. In the test, it got none of the 40 cats. It called 23 of them airplanes and 15 of them skulls. My cat never had to compete with an airplane or a skull. The object list in that version of the game had neither.

The signal is still real. Answering *airplane* every time would score exactly 10%, because airplanes are 40 of the 400 drawings. Jev got about 35% right, so it does read something from the coordinates. But the 35% is an average of 85% on airplanes, zero on cats, and everything in between.

Why airplane? I don't know. I can't ask Jev either. System One models don't explain their reasoning.

## Base64 lands on chance

Jev reading the base64 PNG scored about 9% (6% to 12%). The range includes 10%, so the result is indistinguishable from chance. It's not evidence that Jev did worse than guessing, or that it did exactly as well.

Jev wasn't picking at random, though. It answered *airplane* 199 times and *house* 197 times. Whatever the object, 396 of its 400 answers were one of those two. With ten balanced categories, any strategy that doesn't use the shape scores around 10%.

I think the file format explains it. PNG compression loses nothing, so the whole shape is still somewhere in the text. But pixels that sit next to each other in the picture end up nowhere near each other in the base64 string, and no readable spatial structure survives. So Jev's 35% on SVG comes from the coordinates, which still describe lines.

## What these numbers don't cover

- Only drawings that Google's recognizer had already classified correctly.
- Ten objects I chose because I can draw them. A different ten would give different numbers.
- One version of each model and one attempt per drawing. I measured the difference between models, not the difference between two runs of the same model.
- 400 drawings. Enough for the large differences above, too few for claims about any single object.

## Better than a ten-sided die

Sonnet does far better on the same SVG, and better still when it can see the picture. For practical sketch recognition, Jev isn't useful.

TypeSafe sells Jev as a text model and lists numbers among its weak spots. I fed it pictures written as numbers. It still found enough shape in the coordinates to beat chance, even if much of what it found was airplanes.

I pushed Jev somewhere it shouldn't go, and it wasn't that bad.

---

*Quick, Draw! Dataset by Google, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). [https://github.com/googlecreativelab/quickdraw-dataset](https://github.com/googlecreativelab/quickdraw-dataset)*