I Couldn't Read the Tiles, So I Taught My Phone

The quiz, the walkthrough and both tests show boxes the shipped model drew on this photo, run on a computer instead of the phone, with the app’s own rules: boxes under 0.30 are dropped, overlapping boxes are merged whatever tile they name, and anything at 0.72 or above is accepted without review. The page runs no model.

The photo is one frame, IMG_0157.HEIC, which I took at night under room light, standing, with the whole floor in view. The close-up is a crop of the same pixels, not a second photo. In the full frame the model found 7 boxes, none at or above 0.72, and one of them is on a book. In the crop it found 44, all at or above 0.72, and I checked every name by eye. Before merging, those tiles had collected 17 to 39 overlapping guesses each, 1,308 in all.

The light test multiplies every pixel of the crop by a factor, the same arithmetic as the page’s brightness filter, so what you see is what the model saw. All 44 tiles survive down to 10% and none at 2.5%. The size test pastes the crop, shrunk, onto a canvas of the floor’s average colour. Tile heights are the median box height in the model’s input.

This is one photo, not a benchmark. Darkening in software adds no sensor noise, motion blur or colour shift, so it tests brightness and nothing else. A real dim room could still do more damage than this test shows.

One frame only suggests. The Tracker names a tile after two strong reads of the same face, each scoring at least 0.80, at least half a second apart. If two later strong reads agree on a different face, it clears the name and asks the user rather than switching on its own. A correction by the user holds for that tile until it leaves the table.

The Winter and Autumn captures in the story came from the Lens view, which reads one tile at a time, a minute apart. The first was the first version of the detector, picked from the model menu that development builds have; the second was Darknet Mobile M, the model that ships. An earlier version of this page said both came from the shipped model. They didn’t.

Counts are training boxes per tile face, recounted from the label files. The labels also have a class for face-down tiles, and V1 held only six of those. An earlier version of this page compared 5,097 with those six and called the data 850-to-one out of balance. That was true of tile backs, which the app never has to name. Between faces it was 5.7-to-one.

I removed exact duplicates by file hash and near-duplicates by a perceptual hash, which catches the same photo resized or recompressed. One source almost disappeared: its 4,893 training images fell to 16, because nearly all of them were already in another download. I left a seventh download out because its labels were poor. The final set holds 28,369 images and 226,923 labelled boxes across 43 classes: 24,275 for training, 3,467 for validation and 627 for the test. The training part is 18,239 real photos, 3,198 composites and 2,838 repeated copies of rare tiles.

Validation and test changed once, when V3 brought in sources that came with their own splits. From then on they were frozen. The composites and copies went into training only, and I opened the test once, after I had already picked the winner on validation.

The seasons still score lowest on that test: Spring 0.59, Summer 0.61, Autumn 0.58, Winter 0.64, against an average of 0.67 across all 43 classes. I never trained the same model without the composites, so this can’t tell you what they added.

Detection accuracy is scored by how much a predicted box overlaps the true one. At the looser setting, a box counts if it covers at least half the tile, which in practice means “did you find the right tile”. At the stricter setting it has to fit much more tightly, so it measures how neatly the box was drawn. (Researchers write these as mAP@0.50 and mAP@0.75.) The app only uses which tile and roughly where. Nothing in it reads a box edge, so the stricter score buys the product nothing.

The chart has three models rather than four because RF-DETR’s tight-box score was never recorded. Between the shipped model and D-FINE alone the gaps are 0.0037 and 0.1241. Every row comes from the same scoring run. Mixing runs would compare a model against itself under a different scorer.

The smaller-input probe. Feeding the shipped model 512-pixel frames instead of 640 sped it up from 18 to 29 frames a second (54.7 to 34.1 ms a frame) and dropped the looser score from 0.9605 to 0.9458, more than the 0.01 I had allowed. 36 of 43 classes got worse, and split by tile size, the smallest tiles lost the most.

Each number counts something different. Considered is designs I assessed at any depth, from reading the paper to training it: the ten in my original plan, plus Darknet, built on YOLOv7-tiny, and RTMDet, which came later. Trained is the four I trained all the way; RTMDet got a training setup and a budget but never ran. On a phone is the four that produced an iPhone model file that ran. Shipped is what an App Store build can reach; the Release build leaves the rest out.

71 is the number of different detector model files on my disk, after removing identical copies: 93 folders, 3.4 GB. It is a floor, not a total, because exports I overwrote leave no trace.

The build correction, on a Mac over the 627 test images, August 12. The answers didn’t change between builds, only the time, and unevenly. So every comparison was redone with Release timings.

The licence figure is the commercial route for YOLO26 (AGPL-3.0) as I found it at the time, about $5,000 a year. It is not a current quote.

I ran both models on the same 37-tile image for five minutes each, on the evening of August 20, about ninety minutes apart. The D-FINE row is a mixed-precision 512-pixel version, not the full-size model scored above. The phone never got warm enough for iOS to flag it, and the battery reading didn’t move in either run, so nothing here is a throttling result. The only record of either run is the results panel the bench copies out.

These runs came from a development build (-Onone). That slows every model, and D-FINE’s kind far more than the shipped one (see the table in the search disclosure). So D-FINE’s speeds here are a floor: in the build that ships it can only be faster. Tile counts and where the operations run don’t depend on the build and can be read as they are.

The ship decision rests on a Release sweep instead: the median of 50 timed runs per setting, each started with the phone already hot, so every time is a worst case. Darknet Mobile M ran at about 33 frames a second on the Neural Engine (30.3 ms a frame). D-FINE’s fastest were about 24 (41.6 ms, 512) and 17 (60.5 ms, full size) frames a second, both on the CPU. On the graphics chip, D-FINE returned 5 and 9 tiles of 37, with no crash, no error and nothing in the log. That is why the app won’t run that model on the graphics chip.

CPU time per frame is a rough stand-in for cost to the phone, not energy: iOS has no public way to measure power, or how busy the graphics chip and Neural Engine are. Operation counts are where the model’s plan puts each step before it runs, not where it ran. Peak memory is what iOS charges the app for.

Every time a model changes format it can quietly change its answers, so I checked each step against the one before. First the original training output against the ONNX export, then the export against the iPhone’s Core ML model, on 24 fixed test inputs, with and without the Neural Engine. Then I froze the model that won on validation and opened the 627-image test once. It scored 0.967 on the release measure, the looser accuracy score above with a box counting once it covers 30% of the tile, against a bar of 0.77 I set before looking.

The check of the finished app and the later phone run are separate records, not one automated pipeline. The first makes sure the exact App Store build contains the real model rather than a demo fallback. The second shows how it behaves over time, on one phone and one scene.