Photo Recognition
Photo Recognition — Estimating which foods are on a plate, and how much of each, from a photograph, so a calorie tracker can log the meal.
From pixels to a diary entry
When you photograph a meal in a calorie tracker, the app has to answer two separate questions: what is on the plate, and how much of it. The first is recognition in the narrow sense. The second, portion estimation, is harder and usually decides whether the result is useful.
Behind the scenes the process runs in roughly four steps:
- Find the foods. The model separates the image into regions: this patch is rice, that one is chicken, the green bits are broccoli.
- Name them. Each region is matched to a food the system knows.
- Estimate the amounts. From the size of each region, the plate, the angle and any cues about depth, the model estimates a weight or volume.
- Look up the nutrients. The named food and estimated amount are matched to a record in the app’s food database, which supplies the calories and nutrients.
An error at any step carries through. A perfect identification with a poor portion estimate still gives a poor number, and so does a perfect estimate matched to a bad record.
What makes some plates hard
- Depth. A single photo from above shows area, not height. A shallow plate and a deep bowl can look alike.
- Hidden ingredients. Cooking oil, butter, dressing absorbed into a salad, sugar in a sauce. Calorie-dense and almost invisible.
- Look-alikes. White rice and cauliflower rice, regular and diet versions of the same drink.
- Mixed dishes. Curries, stews and casseroles where the components are no longer separate.
How accuracy is measured
The fair test is to weigh every component of a meal, work out its true calories from reference values, photograph it and compare the app’s estimate with the truth, repeated over many meals. The average size of the miss is usually reported as MAPE. Good tests split results by difficulty, because a single grilled fillet flatters every system and a curry tests them properly. Our own photo-only subset is described in How We Test Calorie Trackers.
What our benchmark found
The approach shows up sharply on hard plates. On Tier 3 mixed-dish photos in our 2026 benchmark:
The gap is roughly 8×. Simple plates flatter every system; mixed dishes with hidden composition are where the differences appear.
The other distinction that matters is confidence gating: some photo apps flag low-confidence logs and prompt you to verify the portion, others silently log the model’s output. PlateLens has confidence gating; Cal AI does not. In our test, the verification prompt fired on 8 of 60 photos in PlateLens, and we corrected 6 of those, recovering accuracy that the raw model output would have missed.
Worked example
Suppose an app looks at a plate and estimates 150 g of cooked white rice, but the plate actually holds 210 g. Using USDA’s value of about 130 kcal per 100 g:
- Estimated: 150 g × 1.30 = 195 kcal
- Actual: 210 g × 1.30 = 273 kcal
- Miss: 78 kcal, about 29% on that component
The recognition was right; the portion was not. That is why the ability to open a meal and change an amount matters as much as the first estimate.
What to check before you rely on it
Three things are worth checking: whether photo logging is on the free plan, whether you can correct the result afterwards, and what database the photo lands on. PlateLens is a good reference point on all three. Its camera matches what it sees against a catalogue of 1.2M+ verified foods, the biggest checked catalogue among calorie trackers, and any logged meal can be reopened to change a single ingredient or its amount. Free users get three photo estimates daily on top of unlimited scanning and typing; Premium removes the photo limit.
Getting better results from any photo app
- Get the whole plate in the frame, from a little above and slightly to the side, so depth is easier to judge.
- Use even light and avoid strong shadows.
- Photograph before you start eating, not halfway through.
- Add what the camera cannot see, such as the oil or butter used in cooking.
For a side-by-side of two photo-led apps, Cal AI and PlateLens, see Cal AI vs PlateLens.