This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
We had a similar problem where you can literally millions of email that we were pretty sure came from only a limited set of bad actors.
We first started classifying emails into buckets by From, mailserver relay chains etc as that's all we had to to go on.
Over time, those buckets got linked to spammer signatures and then we narrowed down from there.
Fascinating to see this happening nowadays with LLMs.
But honestly, it only works for common knowledge that's already in the LLM. If the target document contains very niche or private information, then the hallucinated answer's embedding can be even farther away than the query's.
Even better is to search the corpus first with like naive BM25 / embedding search, aggregate over top N to get most representative categories, then have the LLM categorize in that set.
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
I've found, though, getting it in the language of the vocabulary has generally improved performance.
Further, when searching for "blue shoes" you want to separate the color from the item type. So its useful to have a dumb LLM do this for you. And with the LLM in the loop, its further useful to get it into the language of the taxonomy to improve embedding retrieval accuracy.
There are of course many ways to skin the cat here :)
But no classification is perfect. In search in particular, you will also want to have places for manual intervention for high priority queries.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
Putting all the labels into the LLM is super expensive per call when you have millions of items to classify.
You can't reduce the number of labels becasue they are correctly organizes/structured. This class of problem exists in many different domains.
{ rationale, categories }
Luna is surprising capable and cheap, and I haven’t done this type of thing since before GPT 5 so might not be such a useful trick now
It didn't end up being very useful - I ran a comparison where I just had a bigger agent do the organization in a more straightforward way, and that had better results.
I did find that Flash 3.6 High was >9x faster than Luna xhigh for this task, and got very similar results, though.
But if accuracy matters, you can't rely on embedding sort to get a closet match. With a real test set they usually don't hold up under scrutiny.
Everything in AI is like this. You get an idea, try it once or twice, "LGTM" and you ship. Then it never survives contact reality.
Embedding sort gives you a better shortlist than the whole list, but you will probably want a heavier model to vet candidates.
https://github.com/aurelio-labs/semantic-router
I guess it is based on the same fundamentals as well.
You'd think they would have solved it by now.
Asking an LLM for a list of every county in the USA for example, or every county with a population of more than 100,000 people.
Even if those county names and their populations are mixed up in their weights, the nature of next-token-prediction does not lend them to effectively answering comprehensive, detailed questions like that.
An agent system build on top of an LLM can do it, if it has access to tools which can help access eg a table of counties and then filter them with SQL or Pandas or similar.
(Lets ignore for now that no one seems to agree to what should be the spec sheets)
The idea is to make structured queries using these protocols which can be used to fetch top products matching the user needs instead of just relying on semantic search.
https://developers.openai.com/commerce/specs/file-upload/pro...
But you'll be amazed by the abundance.
Instead: use an LLM to build a large (old-school) database of products with all their specifications. The LLM can also build the schema for that database as it finds more data.
Then use an LLM to query that database based on the user's specifications (+ add some intelligence to find nice suggestions for a birthday if wanted, but I'd consider that an extra).
I noticed yesterday when browsing on mobile that there used to be a box where I could search reviews and it got swapped with a Rufus box. I guess somebody needs to juice their engagement numbers for an investor briefing.
honestly, Amazon doesnt even need AI it just needs a better UI, more metadata for its products and to make reviews less scammy.
He clearly didn't know enough about vector embeddings.
It's the same for so many things:
- reading documentation (what do I expect this function to be called?)
- finding clothes in a shop (something long-sleeved and light)
- picking the fridge for dinner
- finding a book in the library...
so many analogues where I'm not coming cold to a choice.
Worse is when you don't know whether the answers you have are totally wrong.
The entire problem of search is that the user has the wrong data and wants to use it to receive the correct data. That was the start, not the state we’ve ended up at - it is unironically how we got to LLMs.