Understanding Ecommerce Search
E-commerce is a complex and ever-evolving field, where businesses constantly face challenges in managing effective search functionalities within their platforms.
Introduction to Ecommerce Search
Our e-commerce solution enhances search capabilities by transforming search phrases through advanced text analysis, and then performs searches on data from both products and content. Ecommerce Search provides suggestions, facets, and search results that are ranked based on commercial values, with the option to incorporate user affinities for personalized results if needed. Additionally, it supports authenticated searches, enabling businesses to control which data is returned, such as additional fields or special prices for a specific set of customers. This optimization of search results helps businesses improve overall conversion rates and customer satisfaction.
Info
For an explanation of terms and acronyms, please refer to our Definitions. You can also use the search bar in the top right corner. Additionally, you can find extensive information about the various features of Ecommerce Search in the menu under "User Guides", "Concepts", and "For Developers". We encourage you to explore this content after reading this article.
Ecommerce Search is a cloud-based product that provides businesses with a comprehensive user interface for managing all aspects of the search flow. In addition, Ecommerce Search offers three standard REST APIs that enable businesses to manage administrative features through code and facilitate the actual search process. These APIs allow for seamless integration with existing systems and enable developers to easily build custom features to meet the unique needs of their business.
In this article, we will dive into the features and functionalities of Ecommerce Search, which will give you a comprehensive understanding of how to leverage this powerful tool to enhance your business.
Segments
Segments are a crucial aspect of Ecommerce Search, allowing businesses to tailor search engine behavior to meet the needs of different customer segments. Each segment is based on culture and restricted to a single language, as the stemming algorithm varies by language.
For businesses operating in diverse markets, such as B2B and B2C, different sets of dictionaries may be required. In such cases, businesses can create two segments with separate dictionaries: one for B2B and one for B2C. This is particularly useful when the vocabulary of B2B customers differs from that of B2C customers. For example, while B2C customers might search for a hammer or saw, B2B customers might search for specific tools by name or model number.
When a segment is created, the following features are automatically established and connected to the new segment:
- Dictionaries: A collection of settings to fine-tune the text analysis of the search engine.
- Scopes: Defines what is returned from a request, including which fields should be included in a search result and how many products, SKUs, contents, suggestions, etc.
- Contexts: Pertains to commercial sorting.
- Allowed filters: Controls which filters can be used in search requests.
- Searchable fields: Determines which fields are utilized during a search.
Segments can be created using the Bizzkit Ecommerce Search UI or API.
Ingesting data
At the heart of Ecommerce Search is data. Effective search functionality relies on accurate and well-structured data, all of which is organized within segments. A segment acts as the top-level container for the data and ensures that it is processed correctly according to its specific requirements. When ingesting data, a segment ID is necessary to associate the data with the appropriate segment. Data is ingested through the API and can be updated (patched) as frequently as needed, ensuring that search results remain relevant and up-to-date.
Data can come from various sources:
- Products and SKUs: Product data follows a predefined format but allows for additional user defined attributes to be included. This flexibility ensures that businesses can customise their product data to meet specific requirements.
- Content: Typically sourced from a CMS, content data also follows a predefined format, allowing for consistency and easy integration.
Products and SKUs
In Ecommerce Search, product data is divided into two levels: products and their underlying SKUs. This structure ensures comprehensive organisation and detailed information for each item.
A product with a single SKU might be a shop selling a toy truck that has one variation (one size, one colour, etc.).
On the other hand, a product with multiple SKUs could be a shop selling a black t-shirt available in different sizes. In this system, the SKU represents the "selling" product in the shop, identified by an EAN number.
A product includes essential metadata such as:
- Product ID and Product Number
- Name, Short Description, and Long Description
- Rating and active periods (from and to dates)
- Alternative search words and related products
- Metadata, including top parent ID and associated images
- Categories that classify the product
- Media, such as URLs for product images
- etc.
A SKU includes key details like:
- Product ID (reference to product)
- SKU ID
- SKU number
- EAN number
- Name
- user defined text, number, and interval attributes, such as brand and color, which are used in searching, creating facets, etc.
- Stock quantity
- Multiple pricing information based on a date interval or an authenticated search (such as a request from VIP customers, etc.)
- Metadata
- Commercial parameters (used for commercial sorting)
- Commercial affinities (used for personalized commercial sorting)
- etc.
Content
When ingesting content like articles, blog posts, etc. that should be part of the search result, you would include data like:
- Name
- Description
- Active from and to
- Alternative IDs
- user defined text attributes
- Media
- Content like headers, paragraphs, etc.
Data examples
Just to get an idea on how ingested data could look like, here is an example of the JSON structure used by the API. Please note, that the format itself is not important to understand for a non-developer, but it gives you an idea of what data Ecommerce Search knows about a product, SKUs, and content.
Example of JSON structures
Product
Here is an example of the product "Bizzkit Unisex BAT t-shirt Black":
SKUs
Here is an example of the products two SKUs (the t-shirt in small and medium).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
|
Content
When ingesting content you would include data like headers, text, etc. Here is an example of data related to an article:
Scopes
Scopes in Ecommerce Search define what data is included in a search result. Configurable settings include:
- Fields to return
- Number of products and SKUs to return
- Number of suggestions to return
- Number of contents to return
- etc.
When creating a segment, several default scopes are generated, such as full search, quick search, browse, and details scopes. These predefined scopes are editable, but by default, they specify what data should be returned when a request is made. They are based on types which allow tracking user behaviour and intent - typically viewable on the dashboard in the UI.
As an example, the quick-search scope (typically used in a quick search bar on the website) will return 5 products, 5 SKUs (1 SKU per product), 5 contents, and 8 suggestions, and include fields like ProductNumber, ProductName, SkuName, Price, Stock, and Media. In contrast, the details scope (usually used on a product details page, or PDP) will return only 1 SKU and 1 content but includes more fields such as short and long descriptions, additional attributes, and metadata.
It is possible to create user defined scopes including authenticated scopes, which are only accessible with the correct authentication token included in the search. This allows for fields to be hidden in a public search but returned in an authenticated search, providing an additional layer of security and customisation.
Having scopes is crucial for security and performance. Scopes ensure that only the necessary data is returned, reducing the load on the system and protecting sensitive information.
Info
When performing a search, two arguments are required: the segment to search in and the scope to use when determining what to return.
Searchable fields
Searchable fields in Ecommerce Search must be designated to be included in search queries. This ensures that the search tool recognises and considers the information within those fields.
Searchable fields can be configured for partial matching, allowing users to type only part of the information to generate relevant results, or full matching, requiring precise entries for exact matches.
By default, some fields are added as searchable fields, such as:
Name | Partial match |
---|---|
ProductId | true |
ProductName | true |
ProductNumber | false |
ShortDescription | true |
SkuEan | true |
SkuId | false |
However, other fields can be added, including ingested user defined attributes.
Contexts
Contexts are related to commercial sorting, filtering, pinning of results, controlling facets, and more. In a solution, you will typically find contexts related to specific parts of the website, such as PLPs on the front page, product bands, quick and full search, category pages, etc.
A context always plays a part when Ecommerce Search returns data, determining both what products to return and in what order. If no user defined contexts are created, a fallback context will be used, and each segment must have exactly one fallback context. While we will delve deeper into commercial sorting, facets, etc., later, it is important to understand the concept of a context now.
A context has a set of conditions that Ecommerce Search uses to determine which context to apply when returning data. These conditions can be based on a combination of:
- A specific scope
- A specific category
- Stock information
- User defined attributes (like color or brand)
- etc.
The more context conditions that match the data in the search request, the more relevant the context is. The most relevant context, which has all context conditions satisfied, is used. Ecommerce Search selects the appropriate context for a search request based on the API call.
Temporary summary
With the concepts of segments, data, scopes, searchable fields, and contexts introduced, it's now possible to get a feel for how Ecommerce Search performs the actual searching. This initial setup enables a client to make a request to Ecommerce Search and receive a response, though it does not yet include features like creating suggestions, text analysis, facets, or commercial sorting - more on that later.
Typically, from a solution perspective, a client would be a browser sending a request to the API when a user types into a quick search box, browses a category of products, or views a product details page (PDP).
Imagine a user typing a search phrase like "red" in a quick search text box. This would make the browser create a request to the public Ecommerce Search API (called the Search host) with:
- A phrase (like
red
) - A segment (like
segment1
) - A scope (like
quick-search
)
Based on this information, Ecommerce Search will know what segment with data to use, what fields and number of results to return based on the scope, and based on context conditions, find the correct context. This context will determine the sorting and boosting to use, what products to return, and what facets to return. If no context conditions match a specific user-defined context, the fallback context will be used.
The response from the API call could include:
- A number of products with various fields and underlying SKUs, all determined by the specific scope
- Products filtered and ordered by the context Ecommerce Search has found to be the correct context
- A collection of suggestions for alternative search phrases determined by the specific scope
- A collection of related tags for alternative search phrases determined by the specific scope
- A collection of "did you mean" phrases for alternative search phrases determined by the specific scope
- A collection of content (like articles and blog posts) that matches the search phrase determined by the specific scope
- A collection of facets determined by the specific context
- Etc.
This response can then be used by the client (like a browser) to create a user interface. Here is an example of a quick search from a simple website we use in Bizzkit Academy to teach developers how to work with Ecommerce Search:
Both the list of products and various suggestions are created based on the API response.
HTTP example
Even if you don't have developer experience, looking at the HTTP request (API request from the browser) and the HTTP response from Ecommerce Search will give you an idea of how an e-commerce solution could be created. Please note that the request can contain other information like filters, authentication tokens, forced sorting, etc., and the response is influenced by both scope and context and only showing one product.
HTTP example of request and response
Request
ResponseSuggestions
Suggestions in Ecommerce Search are phrases that can be proposed to users as they, for instance, type in the search box of a web shop. The aim of these suggestions is to direct users towards phrases that yield the most relevant results. The implementation of these suggestions from the Ecommerce Search API varies, as there are three types to choose from:
- Suggestions are phrases that might interest the user. For example, searching for
hat
could suggestgrey hat
,green hat
,red hat
, etc. Similarly, a search forposter
might suggestred moon poster
. - Related tags expand a search phrase by adding a tag, thus creating a new search possibility. For instance, if a user searches for
red
, the related tags could behoodie
,t-shirt
, andhat
, leading to potential new searches likered hoodie
,red t-shirt
, andred hat
. - Did you mean suggestions offer users an alternative search phrase in case of minor typos or misspellings. For example, if a user searches for
grean
, the 'did you mean' suggestion might begreen
.
Managing suggestions and related tags is done through the UI. There's also a straightforward method to automate the creation of suggestions using a suggestion generator. This tool automatically generates suggestions based on fields like product name, colour, brand, etc.
Suggestion candidates can originate from various sources, such as product names from ERP/PIM systems, brands, or top searches from engines like Google Analytics. To prevent unwanted suggestions from external sources, taboos are words or phrases that should not appear in search suggestions. These might include products not suitable for children or adolescents, which should only be found via direct product search.
As you saw under HTTP example, the response from an API call to the Search host will include different suggestions (determined by the chosen scope).
Text adjustment and analysis
A major feature of Ecommerce Search is in text analysis of product data and search queries. One of the main benefits is that search administrators can customize the search experience without any technical expertise, simply by understanding the language. To provide administrators with a high degree of customizability, Ecommerce Search offers several text analysis tools. These tools enable administrators to modify how product information is matched with search queries, enhancing the relevance and accuracy of search results.
Text analyses are based on dictionaries, and each dictionary has a language associated with it. A segment can utilize multiple dictionaries of the same type, enabling businesses to attach a general dictionary with linguistic data that can be reused across all segments, and a specialized dictionary with linguistic data specific to that segment. It's also possible to share dictionaries between segments. By utilizing this feature, businesses can for example choose to share a common set of synonyms between their B2B and B2C solution.
There exists a dictionary type for each of the following general linguistic data:
-
Hypernyms are words that are superset of other words. For example,
pasta
is a general description, withlasagna
,fettuccine
, andtortelli
being subsets, also known as hyponyms, within the general category of pasta -
Synonyms are words that have the same meaning in the given context. E.g.,
pants
andtrousers
,mug
andcup
, orspace
anduniverse
-
Misspellings are misspellings of other words
-
Irregular words are words that do not follow the normal grammatical rules for conjugation. A common case in English is nouns whose plural form does not end in s or es (e.g.,
woman
,women
) -
Ignored words are words that will be ignored if given as part of a search query. It is typically common words that have little meaning on their own like
the
,is
,of
, orbe
More advanced text analysis includes Stemming overrides that are used to override the stemming of a word. Stemming overrides should only be used to prevent words with a different meaning from being stemmed to the same word. E.g., preventing Training
being stemmed to Train
and Universe
being stemmed to Univers
.
Publications
The management of dictionaries etc. is based on publications, allowing for testing, and reverting of settings before they impact end-users. When a search is conducted, the published settings for the segment will be used, rather than the unpublished settings in the admin interface. This approach enables businesses to test the effects of settings before implementing them and to revert to old settings if new ones cause unexpected issues.
By utilizing publications, businesses can ensure a smooth and reliable search experience for their customers.
Commercial sorting
Ecommerce Search offers a valuable feature that allows businesses to sort product results based on commercial properties, product and user affinities. This is particularly useful when there are multiple products with equal search relevance, as commercial sorting can be used as a tiebreaker to prioritize certain products. Unlike traditional search relevance, which is based solely on the search phrase, commercial sorting considers a product's commercial relevance.
Using commercial sorting, businesses can ensure that the products most likely to be purchased are prominently displayed at the top of the search results. This is especially beneficial for products with a higher profit margin or significant commercial value, such as those that help build brand awareness. By prioritizing these products, businesses can increase their chances of converting customers and boosting sales.
Commercial parameters
A product's commercial score is determined by the commercial parameters of the product, depending on the current context. The context determines the weight of the commercial parameters of the product while the user might have an affinity towards certain affinities of the product. These two contributions can be combined to calculate the product's commercial score, and use this for sorting and ranking.
Examples of commercial parameters could include:
- number of clicks
- profit margin
- a trend index
- items in stock
- index representing when a product was added
- a calculated index based on a weather forecast
Following types of commercial parameters are supported:
-
Rank is a simple index. Examples could be number of clicks or contribution margin.
-
Cluster rank is similar to a normal rank normalization, but calculates rank based on a sample of values. Cluster rank normalization is significantly faster than normal rank normalization.
-
Range is a range of values. An example is a range indication when a product was added to the shop (like 7 days with values 0-7, 14 days with values 8-15 and 21 days with values from 15-22).
-
Term will look at the input value and match the input against the terms defined on partitions to find a fitting partition. An example is terms like "few in stock", "sold out" and "plenty in stock".
-
Proxy is a custom value between 0 and 1 so it fits the normalized values generated for other parameters.
Each parameter type is used to determine how ingested values should be normalized.
Parameters are created through the UI or API but will only be functional after commercial parameter values has been ingested to SKUs. As an example see the example SKU data mentioned previously.
Controlling the sorting and ranking
Using the UI, a search manager can control the sorting and ranking of search results for a particular context by assigning weights to various parameters. When assigning weight to a single parameter, the value itself is not as important. However, when adding weight to multiple parameters, the weights can have an influence on one another.
Through the UI, it is possible to view the commercial index calculation for each product
and compare the calculated index in different ways. This includes the ability to compare a couple of SKUs as well as compare different contexts.
Search relevance
To further control sorting a search relevance can be used to configure how important a phrase match is compared to other contributions. Often, a match in the name or short description of a product is more important than a match in long description. Also, a match on the product number is often the most important. A context is created with a set of recommended settings.
Filtering
A context can include filters to restrict products returned from a search based on specific categories, prices, stock availability, and user-defined attributes. This functionality can be applied in various areas of an online store, such as a specific PLP on a page, a product band, or category pages.
Facets
Facets are user-facing filters that allow shoppers to refine their search results interactively. These attributes can dynamically narrow down search results and are typically displayed on the search results page. Users can filter results based on categories such as color, size, brand, and more.
Facets are managed in the UI when editing a context and are included in the response from a search.
Pinning
The concept of 'Pinning' serves as a tool for businesses aiming to spotlight specific products within search results. This functionality enables the ability to 'pin' selected products, identified by their unique IDs, to the uppermost part of a search result, giving them immediate visibility. The feature is not designed to replace well-balanced commercial parameters or search relevance boosts. Rather, it's a tool of precision, intended to be used sparingly and strategically to highlight select products in specific circumstances.
Pinned products come with distinct features that make them stand out. Firstly, the order of these pinned products remains fixed; the product placed first in the list will invariably appear first in the search results, followed by the rest in the order they were pinned. This allows businesses to strategically feature products based on factors like popularity, new arrivals, merchandising, or promotional campaigns.
Secondly, a pinned product will be displayed only if it's relevant to the search query. If a product doesn't match the user's search, it won't be unnecessarily thrust upon the user, ensuring that the search results remain pertinent and user-friendly.
Lastly, to maintain the integrity of the search experience, a pinned product appears only once. It doesn't reappear in the general list of sorted products, avoiding repetition and potential clutter in the search results.
The user interface (UI) is designed for the pinning process to be intuitive and user-friendly, simplifying the task of pinning products and rearranging their order as needed.
Product and user affinities
Just as commercial values for each SKU can be uploaded to Ecommerce Search, it's equally important to understand both product and user affinities. A product affinity might represent a product's vibrant color (e.g., pink or neon) or a numerical score reflecting its "organic" quality. In parallel, user affinities, such as those for "vibrant colors" and "organic products," allow Ecommerce Sort to produce tailored sorting. These user affinities are typically inferred from a range of data, including product purchases, site analytics, and newsletter preferences, among others.
Ecommerce Search generates this personalization based on a user ID provided within the search request. The degree of personalization, or the "personalization weight," is adjustable by the search manager via the UI. Additionally, this interface allows for the testing of sort order by simulating specific user interactions.
Insights
Ecommerce Search includes several features to gain information about search history and tools for diagnostics.
Dashboards
Dashboard is a tool in the UI that can be used to start improving search results for end users by showing tables containing key metrics for the search experience of customers.
Ecommerce Search will regularly perform searches based on the ingested metrics to analyze the results and present them as tables. Several widgets are available:
- 0 - Product results (30 days): Displays the most popular searches in the last 30 days with no matching products.
- Popular searches last week: Displays the most popular searches in the last 7 days, along with the number of searches performed.
- Number of products in popular searches last week: Displays the most popular searches in the last 7 days and the number of products that match the search phrase.
- Top phrases to improve: Displays metrics for enhancing search experience. Findability indicates how effectively a phrase yields desired results. Satisfaction combines findability and search frequency to identify key phrases to optimize.
- Tracked phrases since last week: Displays the tracked phrases the last 7 days.
These widgets provide valuable insights into search behavior and can help businesses optimize their search engine performance to better meet their customers' needs.
From a dashboard a search manager can easily create and test misspellings, synonyms, hypernyms, etc.
Info
Typical workflows includes starting on the dashboard, investigating further in the diagnostics, fixing the problem, and repeating the process until the desired results are achieved. Alternative starting points may include Google Analytics or receiving a request for assistance from a customer or colleague via email, phone, or Teams, in which case the diagnostics feature would also be a natural next step.
Diagnostics
Diagnostics is a useful tool for businesses looking to diagnose search engine behavior and test new configurations. It allows users to mimic search behavior on a live site using unpublished data, which means that businesses can test changes before implementing them on the live site. Additionally, pipeline analysis is a feature of diagnostics that provides a deeper understanding of how the search engine interprets search phrases and how different rules influence search results. The pipeline analysis tool allows businesses to see the effects of text analysis rules for the current segment, making it easier to identify modifications needed to improve search results.
Search results and missing products are two additional features of diagnostics. Search results display all products found by the search phrase and provide insight into why specific products were matched by the search engine. Missing products, on the other hand, allow businesses to compare specific products against search phrases and determine why those products are not appearing in search results. By combining missing products with the pipeline analysis tool, businesses can identify which modifications are needed in the text analysis rules to improve the accuracy of their search results. Overall, diagnostics is a powerful tool that can help businesses optimize their search engine performance and improve their customers' search experience.
Bizzkit AI assistant
The Bizzkit AI Assistant (BAIA) is an advanced optional feature within the Bizzkit platform, designed to enhance user experience through automation and intelligent interaction. It serves several purposes:
BAIA is based on the latest AI models, allowing users to interact with BAIA using natural language. BAIA possesses comprehensive and updated knowledge of Bizzkit Documentation and can, therefore, provide users with answers to questions about all products and features. BAIA can access the Bizzkit APIs, enabling users to communicate with BAIA in natural language to both query and modify data across different products. Leveraging its language model foundation, BAIA can also enrich queries with additional data.
In Ecommerce Search, using BAIA makes it extremely easy to access documentation in a natural way. Here is an example of how to communicate with BAIA to get information about the different dictionaries:
Using BAIA to access the API can also be extremely effective and time-saving. Here is a video showing how to use a recipe (predefined prompt) for validating synonyms.