{"id":178,"date":"2026-10-14T12:16:48","date_gmt":"2026-10-14T12:16:48","guid":{"rendered":"https:\/\/aunhumano.com\/?p=178"},"modified":"2026-10-14T13:59:06","modified_gmt":"2026-10-14T13:59:06","slug":"four-ai-models-walked-into-a-supermarket-none-could-replace-the-person-who-audits-it","status":"publish","type":"post","link":"https:\/\/aunhumano.com\/index.php\/2026\/10\/14\/four-ai-models-walked-into-a-supermarket-none-could-replace-the-person-who-audits-it\/","title":{"rendered":"Can LLMs audit a supermarket shelf yet?"},"content":{"rendered":"\n

A few years before the advent of gpt-3, a colleague introduced me to an entrepreneur who had a problem he wanted to solve. He ran a “human supermarket shelf supervisors” business where each supervisor visited many supermarkets q day in order to check whether some products of a target brand were being “optimally presented” on the shelves.<\/p>\n\n\n\n

Their job was to go to each supermarket, find the right shelves, take one or more pictures to each shelf and then upload the pictures to an app along with some data: how many spaces does the product occupied on a shelf and whether there were empty spaces at the front and how many. This process presented two issues, the time it took and human errors like bad pictures or wrong annotations.<\/p>\n\n\n\n

The entrepreneur’s idea was to partially solve the time issue and the annotations issue. He knew I was working in “AI” at the moment so we talked about the problem, I was able to get access to some pictures and the data extracted of them in order to get an idea of the difficulty of this problem. The next time we talked I gave him my conclusion: “this is not possible unless you deploy many monies to hire a team to do it”. That was the end of the conversation, in Argentina the salaries in that year were very low so it was cheaper to still pay humans instead of building that kind of technology.<\/p>\n\n\n\n

And that brings us to today, I’m working on several products where I need to check whether the newest LLMs finally crack some challenges so, with the help of my friend Claude, I built tinybenchmarks<\/a>, a project to quickly run a one off test of a given task and check if some LLMs can solve it correctly.<\/p>\n\n\n\n

This problem came to mind while I was thinking about interesting problems outside my current focus (I’ll share them soon) to start testing the tinybenchmarks project, in the next section I’ll share the results.<\/p>\n\n\n\n

The supermarket shelf tiny benchmark<\/h3>\n\n\n\n

I used tinybenchmarks to check how good are some current LLM models (OpenAI family for now) when they need to do shelf supervision work, the chosen models were: gpt-6-astra, gpt-6.1-sol, gpt-6-luna and gpt-4.1.<\/p>\n\n\n\n

I’ve picked 3 photos from argentinian supermarkets available online and created five tasks for the models to accomplish.<\/p>\n\n\n\n

Let’s see them case by case:<\/p>\n\n\n\n

Case 1: Facings of Ala sachets<\/h4>\n\n\n\n
\"Full
The detergent aisle. The yellow box marks the zoomed area.<\/figcaption><\/figure>\n\n\n\n
\"Six
Zoom: second shelf from the top.<\/figcaption><\/figure>\n\n\n\n

How many facings of the yellow “Ala concentrado” sachets are on the second shelf from the top?<\/em><\/p>\n\n\n\n

Expected: 6<\/strong><\/p>\n\n\n\n

Model<\/th>Run 1<\/th>Run 2<\/th>Run 3<\/th><\/tr><\/thead>
gpt-6-astra<\/code><\/td>\u2705 6<\/td>\u2705 6<\/td>\u2705 6<\/td><\/tr>
gpt-6.1-sol<\/code><\/td>\u2705 6<\/td>\u2705 6<\/td>\u2705 6<\/td><\/tr>
gpt-6-luna<\/code><\/td>\u274c 7<\/td>\u2705 6<\/td>\u274c 7<\/td><\/tr>
gpt-4.1<\/code><\/td>\u274c 5<\/td>\u2705 6<\/td>\u2705 6<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Case 2: Gaps on a detergent shelf<\/h4>\n\n\n\n
\"Full
The detergent aisle. The yellow box marks the zoomed area.<\/figcaption><\/figure>\n\n\n\n
\"Detergent
Zoom: the shelf with the large yellow Limzul bottles at the left end and the blue Magistral bottles. One Limzul and one Magistral spot are empty.<\/figcaption><\/figure>\n\n\n\n

On the shelf with the large yellow “Limzul” bottles at the left end and the blue “Magistral” bottles, count the empty facings, split into yellow Limzul, blue Magistral and anything else.<\/em><\/p>\n\n\n\n

Expected: 2<\/strong> (1 Limzul \u00b7 1 Magistral \u00b7 0 other)<\/p>\n\n\n\n

Model<\/th>Run 1<\/th>Run 2<\/th>Run 3<\/th><\/tr><\/thead>
gpt-6-astra<\/code><\/td>\u2705 2 (1 Lim \u00b7 1 Mag)<\/td>\u274c 3 (1 Lim \u00b7 2 Mag)<\/td>\u2705 2 (1 Lim \u00b7 1 Mag)<\/td><\/tr>
gpt-6.1-sol<\/code><\/td>\u2705 2 (1 Lim \u00b7 1 Mag)<\/td>\u2705 2 (1 Lim \u00b7 1 Mag)<\/td>\u2705 2 (1 Lim \u00b7 1 Mag)<\/td><\/tr>
gpt-6-luna<\/code><\/td>\u274c 1 (1 Mag)<\/td>\u274c 0<\/td>\u274c 0<\/td><\/tr>
gpt-4.1<\/code><\/td>\u274c 1 (1 Mag)<\/td>\u2705 2 (1 Lim \u00b7 1 Mag)<\/td>\u274c 1 (1 Mag)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Case 3: Facings of Gallo Largo rice<\/h4>\n\n\n\n
\"Full
The rice aisle. The yellow box marks the zoomed area.<\/figcaption><\/figure>\n\n\n\n
\"Eight
Zoom: right-hand unit, third shelf. A ninth bag is cut off by the edge of the photo.<\/figcaption><\/figure>\n\n\n\n

How many facings of “Gallo Largo” rice are there? Count only bags whose logo is visible; ignore the bag cut off at the right edge.<\/em><\/p>\n\n\n\n

Expected: 8<\/strong><\/p>\n\n\n\n

Model<\/th>Run 1<\/th>Run 2<\/th>Run 3<\/th><\/tr><\/thead>
gpt-6-astra<\/code><\/td>\u2705 8<\/td>\u2705 8<\/td>\u2705 8<\/td><\/tr>
gpt-6.1-sol<\/code><\/td>\u2705 8<\/td>\u2705 8<\/td>\u2705 8<\/td><\/tr>
gpt-6-luna<\/code><\/td>\u2705 8<\/td>\u2705 8<\/td>\u274c 9<\/td><\/tr>
gpt-4.1<\/code><\/td>\u274c 5<\/td>\u274c 5<\/td>\u274c 6<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Case 4: Bags placed backwards<\/h4>\n\n\n\n
\"Full
The rice aisle. The yellow box marks the zoomed area.<\/figcaption><\/figure>\n\n\n\n
\"Bottom
Zoom: right-hand unit, bottom shelf. The two pale yellow packs are Don Bernardo bags turned around.<\/figcaption><\/figure>\n\n\n\n

Count the facings that need restocking. A bag placed backwards also counts as a missing facing. Classify each gap by the product that should be there.<\/em><\/p>\n\n\n\n

Expected: 2<\/strong> (2 Don Bernardo \u00b7 0 other)<\/p>\n\n\n\n

Model<\/th>Run 1<\/th>Run 2<\/th>Run 3<\/th><\/tr><\/thead>
gpt-6-astra<\/code><\/td>\u2705 2 (2 DB)<\/td>\u2705 2 (2 DB)<\/td>\u2705 2 (2 DB)<\/td><\/tr>
gpt-6.1-sol<\/code><\/td>\u2705 2 (2 DB)<\/td>\u2705 2 (2 DB)<\/td>\u2705 2 (2 DB)<\/td><\/tr>
gpt-6-luna<\/code><\/td>\u274c 3 (3 DB)<\/td>\u274c 3 (3 DB)<\/td>\u274c 3 (3 DB)<\/td><\/tr>
gpt-4.1<\/code><\/td>\u274c 5 (4 DB \u00b7 1 other)<\/td>\u274c 3 (2 DB \u00b7 1 other)<\/td>\u274c 5 (3 DB \u00b7 2 other)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Case 5: The missing top layer<\/h4>\n\n\n\n
\"Full
The dairy fridge. The yellow box marks the zoomed area.<\/figcaption><\/figure>\n\n\n\n
\"Yogur\u00edsimo
Zoom: third shelf. The cups at the back are stacked two high; six cups at the front are missing their top cup.<\/figcaption><\/figure>\n\n\n\n

The strawberry “Yogur\u00edsimo” cups are stacked two high. How many cups need restocking in the top layer?<\/em><\/p>\n\n\n\n

Expected: 6<\/strong><\/p>\n\n\n\n

Model<\/th>Run 1<\/th>Run 2<\/th>Run 3<\/th><\/tr><\/thead>
gpt-6-astra<\/code><\/td>\u274c 7<\/td>\u274c 7<\/td>\u274c 7<\/td><\/tr>
gpt-6.1-sol<\/code><\/td>\u274c 7<\/td>\u2705 6<\/td>\u274c 8<\/td><\/tr>
gpt-6-luna<\/code><\/td>\u274c 3<\/td>\u274c 3<\/td>\u274c 4<\/td><\/tr>
gpt-4.1<\/code><\/td>\u274c 4<\/td>\u274c 2<\/td>\u274c 3<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

The results<\/h3>\n\n\n\n
Model<\/th>Facings error \u2193<\/th>Facings exact<\/th>Gap count error \u2193<\/th>Gap count exact<\/th>Gaps by product<\/th>Cost per run<\/th><\/tr><\/thead>
gpt-6-astra<\/code><\/td>0.00<\/td>100%<\/td>0.44<\/td>56%<\/td>80%<\/td>$1.15<\/td><\/tr>
gpt-6.1-sol<\/code><\/td>0.00<\/td>100%<\/td>0.33<\/td>78%<\/td>89%<\/td>$0.24<\/td><\/tr>
gpt-6-luna<\/code><\/td>0.50<\/td>50%<\/td>1.78<\/td>0%<\/td>48%<\/td>$0.02<\/td><\/tr>
gpt-4.1<\/code><\/td>1.50<\/td>33%<\/td>2.00<\/td>11%<\/td>48%<\/td>\u2014<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Errors are the mean absolute difference from the right count. “Gaps by product” is the share of products whose gap count was right. Means over 3 runs of each case. The three GPT-6 models ran with reasoning_effort: high<\/code>; gpt-4.1 doesn’t reason. Cost is the whole run per model (15 calls) at list prices; gpt-4.1’s price isn’t included.<\/p>\n\n\n \n

Conclusion<\/h3>\n \n\n \n

Back then the problem was the cost: building this would have meant hiring a team, and humans were cheaper. Today the cost is partly\n gone. Running all five tasks with gpt-6.1-sol costs about eight cents so I don’t think that would be a big issue to take it into consideration.<\/p>\n \n\n \n

What’s still missing is reliability. The two best models count facings perfectly, but that’s the easy half of the job. The\n point of auditing a shelf is finding what needs restocking, and there even the best models get the gap count wrong in one or two\n answers out of every four or five. Nobody reliably counted the missing layer of yogurts. A supervisor who missed one gap out of\n every four or five wouldn’t keep the job.<\/p>\n \n\n \n

So, not yet. The models could already help, for example by counting facings and flagging photos for a human to check, but\n they can’t replace the person walking the aisles. This is five cases on three photos, so take it as a first look, not a final\n answer. I’ll keep adding harder cases and re-run it when new models come out.<\/p>\n \n\n\n

Run it yourself<\/h3>\n\n\n\n

The benchmark is already available in tinybenchmarks<\/a> if you’d like to take a look.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

A few years before the advent of gpt-3, a colleague introduced me to an entrepreneur who had a problem he wanted to solve. He ran a “human supermarket shelf supervisors” business where each supervisor visited many supermarkets q day in order to check whether some products of a target brand were being “optimally presented” on […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-178","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/posts\/178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/comments?post=178"}],"version-history":[{"count":11,"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/posts\/178\/revisions"}],"predecessor-version":[{"id":203,"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/posts\/178\/revisions\/203"}],"wp:attachment":[{"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/media?parent=178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/categories?post=178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aunhumano.com\/index.php\/wp-json\/wp\/v2\/tags?post=178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}