Research

How accurate is a calorie count from a photo?

The Marble team · 20 October 2026

Every calorie app with a camera says it's accurate. Almost none say how they checked. We wanted a number we could stand behind, so we tested Marble's photo reader on thousands of meals that were actually weighed, tried every improvement we could think of, and wrote down the rules before looking at the answers. This post covers what we found, including the parts that don't flatter us.

Why most accuracy numbers can't be trusted

Three things make a calorie-accuracy claim easy to inflate:

• Tuning on the test. If you try fifty prompts and report the one that scored best on your test photos, the score is partly luck you selected for. Do it on fresh photos and the score drops.
• Guessed "truth". Many food-photo datasets label meals with someone's estimate, or a model's. Matching another estimate isn't accuracy. Only weighed food is.
• Easy photos. A reader can look great on clean, single-item, top-down photos and fall apart on a real dinner.

What we measured on

Public datasets, so anyone can check our work:

Only the weighed sets can support an accuracy claim. The others catch regressions and show how the reader copes with messy photos. ACETADA's owners haven't yet said whether we may publish scores on their data, so we use it to make decisions but don't print its numbers here.

The method: practice, leaderboard, vault

A score is only as honest as how independent your choices were from it. So every dataset is split by person (or by dish), and each part has one job:

1. Practice (half of each dataset). Try as many ideas as the budget allows, each checked many times. Every idea gets three numbers: accuracy, cost per photo and seconds per photo.
2. Leaderboard (about 60% of the other half). Checked about once per big round, 5–10 times in total, so we can't hill-climb on it.
3. Vault (the remaining 40%). Read once, by the final reader. That is the only number we will quote in marketing.

Alongside this, we ran a formal study with the analysis plan fixed in advance: it's registered on OSF (18 October 2026, under embargo until publication). It tests whether a small set of pre-declared changes beats our production reader on participants it never saw. That study is paused after its exam and will be written up later. Everything below is practice data, and we label it that way.

Where Marble stands today

Our production reader (Claude Haiku 5.5 with Marble's prompt), read once per photo on the full public sets, before any of this round's tuning. (Update, 19 October 2026: on the practice results below, Marble now reads every photo with Claude Sonnet 5.5 at medium effort. These tables are the Haiku reader it replaced.)

MAE is the mean absolute error per photo. Median error is the median of |predicted − true| ÷ true. Bias is the mean of predicted − true: positive reads high. Read on 13 October 2026 at about $0.0006 and 4.9 seconds per photo (median). That harness enlarged Nutrition5k's and SNAPMe's small photos to 1,280 px, which the app doesn't do. In NutritionVerse-Real, one food ("meatloaf") is labelled at 701 g per piece, a whole loaf, while the photos show slices. Without its 56 photos the MAE is 325 kcal.

Two things stand out. On Nutrition5k's small cafeteria portions (the median dish is under 200 kcal) the reader reads high, by about 120 kcal a plate. On NutritionVerse-Real's large plates it reads low. Percentages flatter big plates and punish small ones, which is why we always show kcal and percent side by side.

What we tried

Round 2 tested every idea on the practice halves only, with each comparison read side by side in the same session, so drift in the model's behaviour falls on both arms alike. Seventeen readers and ideas:

• Prompt changes (six): count oil and sauce only when visible, judge size against the plate and cutlery, account for hidden food, ignore text in the photo, everyday portion references, and the study's two pre-registered favourites combined.
• Effort: low, medium and high (production).
• Thinking: letting the model reason before it answers.
• Two reads averaged, and a second look only when the first read's own calorie range is wide.
• Pricing the items: summing the model's listed foods and grams from the USDA table instead of trusting its total.
• A calibration factor: scaling every answer down a fixed amount.
• Bigger models: Claude Sonnet 5.5 (at three effort levels) and Opus 5.5.

Intervals are 95% bootstrap intervals on the paired difference. Seconds are the median time to a complete answer. Prices are the API's list prices for the tokens each read used.

What we learned

1. Prompt tweaks are mostly noise. None of the six prompt changes moved accuracy by more than about 10 kcal on any set, and their rankings didn't agree between datasets. That includes the two changes our own pre-registered study had picked as winners in its tuning phase. When we repeated them on the same practice photos, in a fresh session, the gain we had seen there all but disappeared. This is exactly the trap that a held-out vault is for.

2. How you score failures can manufacture a win. On large, many-item meals, the reader sometimes runs out of room before it finishes its full answer. The calorie number comes first, so the app still logs it, but a strict scoring rule counts the whole answer as a miss. Under that rule, a setting that writes shorter answers looked like a big improvement. Scored the way the app works, most of the gain was gone: it had mostly fixed the cut-off answers, not the reading. We now score every reader the way the app uses it.

3. Model size is the one lever that clearly worked. Sonnet cut the error by about 40 kcal (a quarter) on Nutrition5k, and it also clearly won on a second weighed set of real phone photos. We'll publish that set's numbers if its owners agree. Opus was no better than Sonnet at three times the price and twice the wait. On about 190 of SNAPMe's self-reported home meals, Sonnet mostly removed the over-read (bias +40 → +13 kcal) without a clear change in MAE.

4. The calibration trick doesn't travel. Scaling every Nutrition5k answer down by 20% looked like a 50 kcal win, because Nutrition5k's labels run low. The same factor made the second weighed set worse. A fix that wins on one dataset only is fitting that dataset's quirks, not reading plates better.

5. Two looks aren't better than one. Averaging two reads, or re-reading only the plates the model is unsure of, gained at most 3–5 kcal for up to twice the cost.

What we'll do with it. Sonnet costs about 20 times more per photo than Haiku and takes about 1.7 seconds longer. On the weighed sets, that buys a clearly smaller error. Whether that's worth it is a product decision, and it waits for the vault. If Sonnet wins there, we'll say so here with that number.

Limits

• These are practice numbers. They come from the half of each dataset we tuned on. The leaderboard and vault numbers come in a later round, and they will be lower-variance and probably a little worse.
• Nutrition5k's labels run low. An independent check found its original labels missed visible ingredients on many of the images it re-examined (Sterling et al., 2026). A reader that sees the butter can be scored as reading high.
• SNAPMe's truth is self-reported. People under-record oils, dressings and seconds, so a reader that under-reads can look better there than it is. We use it to catch regressions, not to claim accuracy.
• A single photo can't see everything. Oil soaked into rice, sugar in a drink and food under other food are invisible to any camera. In the app you can tell the coach what it missed, and that is often worth more than any prompt change.
• The error that matters most is the one that repeats. Marble corrects your calorie target from your weigh-ins, so a reader that is consistently a little high or low is absorbed by the correction. Random error is what costs you.
• We built the reader and we graded it. The study's design was reviewed by AI reviewers, not an independent statistician or dietitian. The open benchmark is our answer: anyone can rerun the public track, and we'll run anyone's reader on the hidden one.

Compete with us

If you make a calorie app or a food model, run it on the Marble open benchmark. The public track has the scoring code and the photo list. For the hidden track, send us your reader and we'll run it on photos no one has tuned on, ours included, and publish the result next to ours, with cost and speed.

Sources

• Thames, Karpur, Norris, Xia, Panait, Weyand, Sim. Nutrition5k: towards automatic nutritional understanding of generic food. CVPR 2021. Dataset CC BY 4.0.
• Coburn, He, Rollo, Dhaliwal, Kerr, Zhu. Evaluating large multimodal models for nutrition analysis: a new benchmark enriched with contextual metadata (ACETADA). IEEE EMBS BHI 2025; arXiv:2507.07048. Dataset CC BY-NC 4.0.
• Larke, Chin, Bouzid, Nguyen, Vainberg, Lee, Pirsiavash, Smilowitz, Lemay. Surveying Nutrient Assessment with Photographs of Meals (SNAPMe). Nutrients 15(23):4972, 2023. Dataset (Ag Data Commons, 2022) CC BY-SA 4.0.
• Hosseinian, Dehghani Zahedani, Mansoor, Hashemi, Woodward. January Food Benchmark (JFB). arXiv:2508.09966, 2025. CC BY 4.0.
• Tai, Nair, Markham, Keller, Wu, Chen, Wong. NutritionVerse-Real. arXiv:2401.08598. Used for evaluation and publication with the authors' written permission.
• Sterling, Berube, Glenn, Shaukat, Barua, Grams, Tsirigos. Vision-language models for image-based dietary assessment. bioRxiv 2026, doi:10.64898/2026.07.26.740845. (On 440 reviewed images where original and AI labels disagreed, Nutrition5k's labels listed 54.7% of the visible items.)
• Analysis plan: OSF registration osf.io/8s2wp, 18 October 2026 (embargoed until publication).