How to improve product search with AI
TL;DR
As part of a new quoting tool we are building, we want to develop the best text-based "in-CRM" product search you have ever used. We were sure the answer would be an LLM with a vector database or ElasticSearch. Due to the challenges of 100% accuracy and keeping the LLM from making things up. It turned out that building a custom product database combined with an LLM was the best answer.
Warning - You will need to be deeply interested in product data to get through this article.
World's Best Text-based in-CRM Product Search.
Our quoting tool targets organisations with lots of products/SKUs where remembering the product name is problematic (a wholesaler may have 100,000+ SKUs) or where there are lots of similar products. For example, there are 1,442 possible combinations of the new Apple MacBook Pro. Each one of these combinations needs to be able to be represented as a single Product Line Item if you want to add them to your CRM.
In addition, we want our search to support bundling. For example, once you have selected your Product, you may need a warranty or a cover/case. We want to provide this data immediately without any further searching required.
We also wanted to be able to provide answers to product questions. Either you have selected a product, and you need to check something like "What is the maximum memory this device supports?" Or before you start searching, you might ask, "Which MacBook supports the most memory?"
Standard text-based product search requires you to know what you are searching for to use the right words for the search. In addition, after a search that returns more than one result, you often need to intuit how to filter the results further to get down to a single result. While this is not overly complex, it slows the quoting process and can cause problems when the incorrect product is selected.
We wanted to build a search that helps you find the product quickly, even if you mistype or guess incorrectly. We also help refine your search by prompting you with the correct terms. You could search easily with zero product knowledge.
Lastly, we want the search to run on your current product data stored in CRM or your product spreadsheet. No additional work, just an improved experience.
Before I explain why we chose to build a custom database, it is essential to understand how products are represented in a CRM. If you know this, skip to 'Data Challenges created by Product Line Items'.
What is a Product Line Item (PLI)?
A PLI is, at a minimum, a Product Description (1), Product Name, quantity (1, if not specified) and SKU or Product Code. Pricing is attached to a PLI but not necessarily part of it (2). Ultimately, it's a single-line representation of the Product that can be readily displayed in a table format in the opportunity record in CRM, in a quote or order form and on an invoice. It's also how you show a product in a tabular price list.
(1) Completely different to an e-commerce product description - see below Apple example.(2) Simplification for brevity.The Product Description aims to capture the key features and should clearly explain what the Product is so it's distinct from other similar products. Here is an example from an Apple price list where the description includes the product name:
14-inch MacBook Pro: Apple M2 Max chip with 12-core CPU and 30-core GPU, 1TB SSD - Silver
The Apple price list is here if you are interested.You can see that this description is hierarchal. It starts with the product name, 14-inch MacBook Pro, then follows with the Chip Type Apple M2 Max chip, then the System on a Chip (SoC) specs 12-core CPU and 30-core GPU, then Storage, 1TB SSD and finally the colour, Silver. The colour is at the end in Apple's Product Description when it's a PLI, which makes sense for readability in a list (price list or CRM).
Hopefully, you are picking up that writing the description is a bit like the Guess Who board game. The value that rules out the biggest or equal biggest number of options should follow the name, then the next biggest and so on down the line. This way, when you organise them in a list, they are much easier to read and understand. This is important to know when searching through Product Line Items because if you can weight the search towards the essential data in the Description, we have found this improves the accuracy of results.
Data Challenges Created by Product Line Items
Now that you understand how Product Descriptions work, you can see why text matching is essential when searching PLIs - users are obviously going to search for product names and the specifications they need. Text translation is required to make searching much easier as many of these products have multiple names for their features, but in a Product Description, you must use one. For example, Storage is analogous to a Hard Disk Drive (HDD) or Solid State Drive (SSD). An LLM has proven the best way to solve translation so far - ChatGPT 3.5 turbo, for performance and cost, is what we are currently using.
The last key ingredient is understanding and converting units of measurement. Conversion is required to allow products to have values such as 500 GB (Gigabytes) and 1 TB (Terabyte) and for the system to understand that 1TB (or 1000 GB) is twice as big as 500 GB. The database needs to understand units of measure inherently.
The Solution - Where we started
We first thought that an LLM with a vector DB would handle the conversion, text translation and text matching, and it could if you built a custom fine-tuned model and could afford the training costs (we can't). Training a custom model is problematic not just because of the upfront time and cost. If you wanted to add an entirely new type of Product, it may require more training (more time and cost).
After much testing of numerous LLMs, vector databases, document databases, in-memory databases and ElasticSearch, we realised that our use case required an accuracy we could not deliver using off-the-shelf LLMs and databases. So, we decided to build a custom database tuned specifically for returning products very accurately combined with an LLM as a final pass to increase accuracy further and deal with any translation requirements.
What did we try?
We considered ElasticSearch as it's a powerful option. However, it requires a lot of configuration and setup to get it right. In testing, we failed to get it working to the level of accuracy we wanted and decided the configuration here would take too long. ElasticSearch is better suited where accuracy is not critical. Think of an e-commerce recommendation engine. If it has a few misses, it does not matter.
We tried Vectra (a fast and free local vector database for JavaScript/TypeScript) and Redis (with a vector DB plug-in), and while they have many great use cases, we could not get the initial retrieval of Products to be consistent or accurate enough for our needs.
We have experience with MongoDB, but the retrieval queries had to be very specific to make it work. We worried that the complexity could be error-prone.
A NoSQL database seemed like a great option as it has the flexibility to have differing structures based on product variations. The only issue we saw was that the queries for an off-the-shelf NoSQL database are still pretty structured, which could be limiting. We needed a No-SQL database where the text string (the search) is the query. Let's build it!
The Solution - What we built
The Product Database (PDB) is a text queryable, NoSQL database using the Levenshtein Algorithm for text matching combined with fuzzy search logic to end with a database that performs like a cross between a Vector DB and ElasticSearch.
The PDB converts input text into a hierarchy of standardised product-specific tokens. For searching, it scores tokens based on exact match, partial match, occurrence and fuzzy text match (various partial matches) using the Levenshtein distance algorithm.
When data is ingested or when you are searching, the text string is broken into Tokens, as an example.
The Product Description: 14-inch MacBook Pro: Apple M3 Pro chip with 12-core CPU and 18-core GPU, 4TB SSD, 36GB Memory - Space Black would be tokenised into the following key-value pairs.
Product Name: MacBook Pro
Screen Size: 14-inch
Chip: Apple M3 Pro chip
CPU: 12-core
GPU: 18-core
Storage: 4TB SSD
Memory: 36GB
Colour: Space Black
We are using ChatGPT to classify the tokens. It works very well when we use it to call our product function. This allows us to get ChatGPT to only look for the right specs and, importantly, tell it, "If you can't find the parameter, give me none", as opposed to making shit up, AKA hallucinating.
OpenAI describes their Function Calling:
'Developers can now describe functions and have the model intelligently choose to output a JSON object containing arguments to call those functions. This is a new way to more reliably connect GPT's capabilities with external tools and APIs.'
How does it work?
The data is ingested from a CSV file or via API from your CRM. On ingestion, it is tokenised. Then, when you search, the text search string is tokenised to retrieve the closest matches. ChatGPT then runs over this list of closest matches and selects the answer most proximate to the original search string.
If your search string does not result in a single correct match, the system will return specifications that will help you refine your search.
If you search iMac, the list of products that match is too extensive to present; instead, the specifications of iMacs are provided to help you filter your search. It enables you to select the right one, even if you don't know the available variables.
Example
Searching 'iMac 24' returns the variables [Chip], [Memory], [Storage], [Colour]; pick the variable/s you want, and you are returned the Product you wanted.
Of course, if you know what you want, you can go straight to it.
Example
If you search for 'iMac 24, 8 core CPU, 8 Core GPU, 8GB memory, 256GB storage, Blue', the exact product you want is returned.
You can also ask product-related questions in the search, like "What is the weight of a 14-inch MacBook Pro with an M3 chip? "And you would get an answer like this:
"According to the Apple website, the weight of a 14-inch MacBook Pro with an M3 chip is 3.4 pounds (1.55 kg). This is the lightest of the three M3 chip options, with the M3 Pro and M3 Max models weighing 3.5 pounds and 3.6 pounds, respectively."
If you understand the issues with searching Product Line Items, we welcome any insights you might have or information on solutions you have found.