Traffic filtering can be used to avoid crawlers, specific user agents, and specific IP addresses or CIDR ranges to be included in Search tracking. The filtering is applied by Bizzkit Event Tracking, which collects the tracked search events.
Using Traffic Filtering
The search API uses the following headers to find the user agent and IP when a search request is received.
Crawlers and user agents: X-Forwarded-User-Agent
IP addresses: X-Forwarded-For
When using the Search SDK the following can be used to forward headers from an incoming request to the search method
Crawlers are automatically filtered out of search tracking based on a list maintained here: monperrus/crawler-user-agents
IP filtering
A request is filtered if its IP address matches a configured IP address or falls within a configured CIDR range.
User agent filtering
A request is filtered if its user agent contains a configured value. The comparison ignores letter case. Wildcards and regular expressions are not supported, so use a distinctive token such as the name of the product or tool, for example UptimeMonitor.
Managing traffic filters
IP, CIDR, and user agent traffic filters can be managed in the Ecommerce Search admin UI or through the Admin API. Bot filtering needs no configuration.
In the admin UI
Go to Settings > Tracking and select the Traffic filters tab. The tab lists the configured traffic filters. Adding, editing, and deleting traffic filters requires write permission.
To add a traffic filter:
Select Add traffic filter.
Enter a Name that describes the filter, for example Office network.
Select the Type: IP, CIDR, or UserAgent.
Enter the Value to filter, for example 203.0.113.10, 203.0.113.0/24, or UptimeMonitor.
Select Add.
Use the edit and delete actions on a row in the list to change or remove a traffic filter.
Through the Admin API
Traffic filters can also be managed through the Admin API, for example to automate the configuration across environments. Use the type IP, CIDR, or UserAgent.