Hello
A month ago I posted v1.0 here (12k images, one 7–12 letter word each). A commenter pointed out that real queries are much shorter, and the production numbers agreed: the median query has 5 distinct characters and a quarter have 3 or fewer. So v2.0 matches that.
WhatFontIs-Bench v2.0:
- 60,000 images, 3,000 fonts (1,059 sans, 1,000 serif, 500 slab, 300 script, 141 mono), same three types and difficulty levels as v1
- the text length, case and digit/& mix are sampled from production query statistics; the strings themselves are COCO-Text v2 strings, not user queries
- labels: font, text, word quad, per-letter quads, camera transform; JSONL + COCO, CC BY 4.0
- a 1,200-image subset that scores within about a point of the full set, for quick checks
Baseline (our own API, open catalogue of 1.2M fonts, top-k by font family), top-1 by number of different letters, pro mode / default mode:
- 1 letter: 48.2% / 43.2%
- 2 letters: 73.8% / 64.7%
- 3 letters: 86.2% / 77.3%
- 4 letters: 89.6% / 82.6%
- 5–6 letters: 92.0% / 86.8%
- 7+ letters: 93.5% / 89.8%
- all 60,000: 88.1% / 82.2%
Above five letters it is as easy as v1; below three, there often isn't enough of the font in the image to separate it from its near-twins. The controlled-glyph diagnostic set (same font and background, only the word varies) is the next one.
Hugging Face (viewer, config v2): https://huggingface.co/datasets/whatfontis/WhatFontIs-Bench
GitHub: https://github.com/whatfontis/WhatFontIs-Bench
Disclosure: I run WhatFontIs, and the baseline is our own system. If you run a model on it, I'd like to see the numbers.