Avnzor — Ask & Ask Live
Voice, camera, and text shopping on a 30,000-SKU health & beauty store — built so the language model talks, and smaller classifiers decide what is true.
I built Ask (search and product-page Q&A) and Ask Live (two-way voice and rear camera) inside Avnzor’s Hydrogen storefront, in English and Saudi Arabic. Tested against 302 real shopper questions pulled from a month of store logs, the setup scores 40 candidate products in 133 ms, drops items carrying ingredients the shopper asked to avoid, and caught 35 of 36 planted product-page errors before they reached the screen — for about two cents per thousand checks.
A chat box over vector search is a weekend build. Putting it on a pharmacy and skincare store in Riyadh is how you end up pitching niacinamide to someone who just asked for no niacinamide — or inventing sun protection a lotion does not have.
The Constraint
Search embeddings do not understand ‘without.’ Type “moisturizer, no niacinamide, under 100 SAR” or “غسول بدون عطور” (fragrance-free cleanser) into a standard vector index and it hands back niacinamide serums and scented washes, because the word you wanted to avoid is the loudest word in the query.
Product pages fail for a different reason: missing fields. Top sellers have clean ingredient and pack data; the long tail arrives from distributor spreadsheets where bottle size or UV filters sit on the carton photo and nowhere else. Ask a language model whether CeraVe Moisturising Lotion has SPF when CeraVe AM does, and it will happily fill in the gap. On a fashion store a wrong detail is a return. On a store selling supplements, retinoids, and over-the-counter medicine, guessing is a safety problem.
There was also a plain physical problem on mobile. Shoppers standing in a bathroom holding an empty bottle do not want to type Latin ingredient names onto glass — and I did not want a ‘chat on WhatsApp’ button kicking people out of the browser right before checkout.
What I Killed Early
One prompt doing retrieval, filtering, and writing. Killed: two to three seconds of dead air before a word shows up, and still unreliable on hard price limits or excluded ingredients.
Telling the writer ‘only use the provided product JSON.’ Killed: models still leak bottle sizes from neighbouring products or assume a whole brand line shares SPF.
Bouncing hard questions to human support on WhatsApp. Killed: pulls buyers out of the store mid-order and buries support under stock and ingredient questions the catalog already has answers for.
Where naive chat broke on real store traffic
Status quo
Vector search reads “no niacinamide” or “بدون سلفات” as a strong match for niacinamide and sulfate products. Asking a large language model to re-read 40 ingredient lists one by one adds seconds of wait time.
Shop Canvas
Pull excluded ingredient terms out of the question with Levanto Sage while Jev scores all 40 shortlisted products in one 133 ms request against the full bottle ingredient list and price ceiling.
The System I Introduced
One rule holds the architecture together: language models speak and format, small classifiers decide.
Instead of sending every message through one heavy model, incoming questions from the search bar, product pages, Ask Live, and WhatsApp split by what kind of check they actually need.
Predictable questions never touch a model. ‘Where is my order?’ checks Shopify Admin and courier tracking (SMSA and Aramex) behind a phone code in 0 ms. Standard product-page chips (‘What does it do?’, ‘What’s in it?’) read from a cache keyed to a hash of that product’s verified facts — instant until the underlying product data changes.
When a shopper searches by need, budget, or skin concern, Jev routes the question (77 ms), matches store-policy topics like returns or COD fees (84 ms across 19 topics), and scores 40 candidate products in one request (133 ms). If the shopper excluded an ingredient — ‘without fragrance,’ ‘no retinol,’ ‘بدون نياسيناميد’ — Levanto Sage runs alongside it to pull out the exact forbidden words so products hiding them mid-label get dropped.
Neither the text writer nor Gemini Live gets free rein over catalog data. Both receive only the surviving product rows and verified bottle facts, capped at four short sentences.
Three real questions from store logs
Click through to see what gets kept, dropped, or handed to a pharmacist before the language model answers.
“Moisturizer, no niacinamide, under 100 SAR”
Classifier output
What happens to the shortlist (4 of 40)
CeraVe Moisturising Lotion · 473 ml
97.00 SAR · In stock
Ceramides and hyaluronic acid. Ingredient list checked — no niacinamide, under 100 SAR.
Bioderma Atoderm Crème Ultra · 200 ml
59.00 SAR · In stock
Fits the budget, and niacinamide isn’t in the title — it sits halfway down the bottle’s ingredient list. Caught and removed.
The Ordinary The Balance Set
155.00 SAR · In stock
Includes Niacinamide 10% + Zinc 1% and costs 55 SAR over the limit.
Neutrogena Hydro Boost Water Gel · 50 ml
26.00 SAR · In stock
Hyaluronic acid gel, no niacinamide in the formula, well under budget.
Search embeddings pull niacinamide products whenever the word appears in the query. Pulling excluded terms out in parallel fixes the shortlist before the writer ever sees it.
Order & chip lookups
0 ms
Where’s-my-order checks and common product-page chips read straight from Shopify, couriers, or a fact-hashed cache. No model call.
40-product shortlist
133 ms
Jev scores 40 candidate products in one request (roughly 9.5× faster than running them through Sage’s batch endpoint).
Product-page fact check
35 / 36
Sage caught 35 of 36 planted errors on product pages — including all 12 wrong numbers and all 12 cross-product leaks.
Classifier cost
~2¢ / 1k
With reasoning turned off (input tokens only), a thousand routing or fact-check calls cost about 2.2 cents.
Voice Without Hanging Up to Buy
Most voice demos look good for ten seconds and fall apart the moment you try to buy something. Cards jump around while the model is still talking, switching from English to Arabic leaves buttons facing the wrong way, and tapping a product kills the audio session.
Ask Live keeps the audio stream, rear camera, product rail, and Hydrogen cart on one state machine. You can interrupt mid-sentence, point the camera at a carton to read the label, stage items into the bag, and switch between English and Saudi Arabic without reconnecting.
Key Decisions
Keep two classifier engines instead of standardising on one
Context
I wanted one provider for routing, ranking, and fact checking so there was only one client and one set of thresholds to maintain. Before deleting the second integration, I ran both against 302 real evaluations from store logs.
What I chose
Kept both, split by job, with each acting as fallback if the other times out. Jev runs the 40-product shortlist because it finishes in 133 ms versus 1,267 ms on Sage. Sage runs ingredient extraction and the product-page fact stream because it caught 35 of 36 planted errors versus 33 on Jev.
Tradeoff
Two schemas and two test suites to keep in sync. Worth it for sub-150 ms search and a second engine ready when an API has a bad afternoon.
Keep the raw probability when a classifier says ‘unsure’
Context
During testing, fallback recall on mixed Arabic questions (someone describing a sore throat while also asking to buy lozenges) quietly dropped from 95% to 85%.
What I chose
Found the culprit in my own wrapper. When Sage is uncertain (scores between 0.35 and 0.65), it abstains with a null answer while still returning the probability (say, 0.506). My code was turning null into 0 and throwing that signal away. Passing the raw number through brought recall back to 100%.
Tradeoff
Callers cannot check a simple true/false flag anymore — every route has to declare a number it cares about (0.50 for medical handoff, 0.70 for line support). More explicit to wire up, much harder to break silently.
Cut unbacked product-page sentences even when they look harmless
Context
On product pages, I planted true sentences about the wrong product — for instance, ‘Inglot Duraline comes in a 9 ml bottle’ on a CeraVe lotion page. Three out of twelve slipped past my first gate because they didn’t mention CeraVe’s own fields.
What I chose
Added a blunt rule: if a line scores ≥ 0.85 as unbacked by the current product’s record, cut it regardless. When the catalog genuinely lacks a bottle detail, a background job fetches the official brand page, checks that it is the exact variant rather than a sister product, and queues verbatim quotes for review.
Tradeoff
Answers read noticeably drier — usually two or three clipped sentences, because loose filler gets trimmed too. On a pharmacy page, dry and right beats warm and made up.
Refuse symptom and dosing questions outright
Context
Shoppers regularly ask what antibiotic to take for a cough or whether a supplement clashes with prescription medication.
What I chose
Hard stop at the router, backed by a dedicated safety queue in admin. Symptom and interaction questions skip product search completely and hand off to a licensed pharmacist card.
Tradeoff
Walks away from easy add-to-carts on symptom searches. Pretending a storefront assistant is a clinician was never an option.
What the Bench Showed
Method note: timings and accuracy come from a 302-case bench built out of 2,395 real shopper turns over thirty days (19 store-policy topics, 200 query-to-product pairs, and 36 planted product-page errors). Isolating conversion lift between Ask users and regular searchers is still noisy, so I am not quoting a conversion claim here — only what the bench measures directly.
- 40-product shortlist (Jev median)
- 133 ms
- Product-page errors caught
- 35 of 36
- Store-policy routing (19 topics)
- 100% · 84 ms
- Wrong numbers & cross-SKU leaks cut
- 24 of 24
- Order status & cached product chips
- 0 ms
- Classifier cost (input-only mode)
- ~$0.022 / 1k
Honest Limits
I turned on open-ended product-page questions before backfilling bottle facts across the long tail of the catalog. Best-sellers had full ingredient lists; tail items imported from supplier sheets often didn’t, so early shoppers hit ‘I don’t have that detail on the label yet’ more often than they should have. If I ran this sequence again, the nightly fact-enrichment job would run two weeks before the input box shipped.
Camera reading in Ask Live also has a very ordinary enemy: bathroom glare on curved plastic tubes and small ampoules. When Arabic label text washes out under a spotlight, the app asks you to tilt the bottle rather than guessing from a bad frame — honest, though still a beat of friction.
What’s Next
Wiring Ask Live’s camera straight into My Stack — so a shopper can pan across the bottles they already own at home, check whether two night steps double up on retinoids or acids, and drop everything into morning and evening slots without typing a single product name.
Let's talk about your project.
ARES — Snap Inc.