Skip to content

Traffic Filtering

Traffic filtering can be used to avoid crawlers, specific user agents, and specific IP addresses or CIDR ranges to be included in Search tracking. The filtering is applied by Bizzkit Event Tracking, which collects the tracked search events.

Using Traffic Filtering

The search API uses the following headers to find the user agent and IP when a search request is received.

  • Crawlers and user agents: X-Forwarded-User-Agent
  • IP addresses: X-Forwarded-For

When using the Search SDK the following can be used to forward headers from an incoming request to the search method

Traffic Filtering Sample

Full tutorial sample.

using Bizzkit.Sdk.Search;

const string segmentId = "merch-b2c-da";
const string scopeId = "full-search";
const string searchHost = "https://ecs-search.bizzkit.it/";

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<
    ISearchClientFactory>(
    new SearchClientFactory(
        new SearchConnectionOptions { BaseUrl = searchHost },
        new HttpClient()));

var app = builder.Build();

app.MapGet(
        "/search/{phrase?}",
        async (string? phrase, ISearchClientFactory searchClientFactory, HttpContext httpContext) =>
        {
            var searchClient = await searchClientFactory.CreateUnauthenticatedClientAsync();
            httpContext.Request.Headers.TryGetValue("User-Agent", out var userAgent);
            var remoteIpAddress = httpContext.Connection.RemoteIpAddress;

            var result = await searchClient.SearchAsync(
                new UnifiedSearchRequestModel { SegmentId = segmentId, ScopeId = scopeId, Phrase = phrase },
                userAgent,
                remoteIpAddress,
                CancellationToken.None);

            return Results.Ok(result);
        })
    .WithName("Search");

app.Run();

Configuration

Bot filtering

Crawlers are automatically filtered out of search tracking based on a list maintained here: monperrus/crawler-user-agents

IP filtering

A request is filtered if its IP address matches a configured IP address or falls within a configured CIDR range.

User agent filtering

A request is filtered if its user agent contains a configured value. The comparison ignores letter case. Wildcards and regular expressions are not supported, so use a distinctive token such as the name of the product or tool, for example UptimeMonitor.

Managing traffic filters

IP, CIDR, and user agent traffic filters can be managed in the Ecommerce Search admin UI or through the Admin API. Bot filtering needs no configuration.

In the admin UI

Go to Settings > Tracking and select the Traffic filters tab. The tab lists the configured traffic filters. Adding, editing, and deleting traffic filters requires write permission.

To add a traffic filter:

  1. Select Add traffic filter.
  2. Enter a Name that describes the filter, for example Office network.
  3. Select the Type: IP, CIDR, or UserAgent.
  4. Enter the Value to filter, for example 203.0.113.10, 203.0.113.0/24, or UptimeMonitor.
  5. Select Add.

Use the edit and delete actions on a row in the list to change or remove a traffic filter.

Through the Admin API

Traffic filters can also be managed through the Admin API, for example to automate the configuration across environments. Use the type IP, CIDR, or UserAgent.

Configure Filtering Sample
using Bizzkit.Sdk.EcommerceSearch.Preview;

namespace ecs_traffic_filter_sample.TrafficFiltering;

public class Configuration
{
    public static async Task ConfigureTrafficFilteringAsync()
    {
        var factory = new SearchAdministrationClientFactory(
            new SearchAdministrationConnectionOptions
            {
                BaseUrl = "esc-admin-host",
                Authority = "auth",
                ClientId = "client-id",
                ClientSecret = "client-secret",
                Scope = "searchapi/"
            });

        var client = await factory.CreateAuthenticatedClientAsync();

        await client.TrafficFiltersCreatePreviewAsync(
            new TrafficFilterModel
            {
                Name = "Block Bizzkit",
                TrafficFilterId = "block-bizzkit",
                Type = TrafficFilterType.CIDR,
                Value = "10.10.0.1/24"
            });

        await client.TrafficFiltersCreatePreviewAsync(
            new TrafficFilterModel
            {
                Name = "Block Localhost",
                TrafficFilterId = "block-localhost",
                Type = TrafficFilterType.IP,
                Value = "127.0.0.1"
            });

        await client.TrafficFiltersCreatePreviewAsync(
            new TrafficFilterModel
            {
                Name = "Block uptime monitor",
                TrafficFilterId = "block-uptime-monitor",
                Type = TrafficFilterType.UserAgent,
                Value = "UptimeMonitor"
            });
    }
}