Open benchmark

Photo to calories, measured in the open

Version 1 · rules frozen 2026-10-19

Many apps say they can count calories from a photo. Few say how well, on what photos, or how they checked. This page sets out a fixed test that anyone can run: the same photos, the same scoring and the same rules for every reader, Marble's included.

Two tracks

Public track: run it yourself

The test dishes of Nutrition5k (Google, CC BY 4.0) that have an overhead photo: the 507 dish ids in the dataset's own depth_test_ids.txt. Each dish was weighed ingredient by ingredient, so the truth is measured, not guessed. The truth file is built from the dataset's release files by a short script, so you can rebuild it and check it byte for byte.

Because the photos and truth are public, a public-track score is only as honest as whoever ran it. We list them as reported, with who ran them.

Hidden track: we run it for you

Weighed meal photos that no reader, ours included, has been tuned on. They are the held-out half of the datasets in our study, split by participant: a leaderboard set (about 60%) for repeated checks and a vault (about 40%) read once per reader. The split is made by a fixed hash of each participant's id and frozen before either half is read. Only the vault score is quoted as a result.

The hidden track opens once our pre-registered study has finished with these photos. Until then they stay sealed, and that includes being kept from us.

Rules

1. One read per photo. No human in the loop, no second try because an answer looked wrong. A retry after a network or server error is fine. A combined reader, such as two reads averaged, counts as one reader with its total cost and time.
2. The photo only. No meal names, notes, weights or metadata, unless your app always asks the user for them, and if it does, say so.
3. Every photo counts. A missing, empty or refused answer scores as 0 kcal, as it would in an app.
4. Three numbers, always. Accuracy, cost per photo and seconds per photo. A reader that is 5 kcal better but ten times the price is a different product, and the table shows it.
5. Declare training data. Say whether your model or prompt was trained or tuned on Nutrition5k. Tuning on it is allowed, but the score is labelled.

Scoring

One file, Python 3.8 or later, no dependencies: score.py. It takes a truth file (id,kcal[,cluster]) and your predictions (id,kcal[,cost_usd,seconds]), and prints:

• MAE: the mean absolute error in kcal per photo, with a 95% interval by bootstrap over clusters (participants, where a set has them).
• MdAPE: the median absolute percentage error, so a 40 kcal miss on a 50 kcal snack and a 400 kcal miss on a 500 kcal dinner are both visible.
• Bias: the mean of predicted minus true. A reader that always reads low can look precise and still undercount every day.
• The share of photos within 25% of the truth, and mean cost and seconds as you measured them.

python3 build_public.py path/to/nutrition5k > truth.csv
python3 score.py truth.csv your_predictions.csv

build_public.py builds the public truth file from Nutrition5k's own metadata.

How to submit

Email support@getmarble.app with:

• Public track: your predictions CSV, how you produced it (model, prompt or app version, photo size), cost and seconds per photo, and the answer to rule 5.
• Hidden track: a way for us to run your reader: a model and prompt we can call through the model's own API, or an HTTPS endpoint that takes a JPEG and returns calories. The vault photos are sent only to the model provider, or to your endpoint, and an endpoint sees the photos. So each vault run uses photos no endpoint has seen before, and after a run they move to the leaderboard set. We publish your score, your cost and seconds, and the date. You can withdraw before we publish.

Results

Public track, Nutrition5k overhead test dishes. Rows are never edited; a new reader is a new row.

* Read on 2026-10-13, before any tuning on this set. That harness enlarged Nutrition5k's 640×480 photos to 1,280 px, which the app doesn't do. The next row will use the app's own photo path.

Published numbers on overlapping Nutrition5k test photos exist, for example GPT-4.1 at 34.6% MAE as a share of the mean (Mu et al., 2025, 505 overhead images, which 505 not stated). Those come from a different harness, so they are context, not rows in this table.

Limits

• Nutrition5k is cafeteria food on a fixed overhead camera, and its portions are small (the median dish is under 200 kcal). It is a clean test of portion reading, not of messy home photos.
• Its labels can miss visible ingredients (Sterling et al., 2026), so its truth may run low, and a reader that sees everything can be scored as reading high.
• The hidden track's sets have licences that limit how they may be used and shown. A set's scores are published only where its owners allow it.

How we use this benchmark to improve our own reader, and what it found: the research post.