Traffic Filtering

Traffic filtering can be used to avoid crawlers, specific user agents, and specific IP addresses or CIDR ranges to be included in Search tracking. The filtering is applied by Bizzkit Event Tracking, which collects the tracked search events.

Using Traffic Filtering

The search API uses the following headers to find the user agent and IP when a search request is received.

Crawlers and user agents: X-Forwarded-User-Agent
IP addresses: X-Forwarded-For

When using the Search SDK the following can be used to forward headers from an incoming request to the search method

Traffic Filtering Sample

Full tutorial sample.

using Bizzkit.Sdk.Search;

const string segmentId = "merch-b2c-da";
const string scopeId = "full-search";
const string searchHost = "https://ecs-search.bizzkit.it/";

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<
    ISearchClientFactory>(
    new SearchClientFactory(
        new SearchConnectionOptions { BaseUrl = searchHost },
        new HttpClient()));

var app = builder.Build();

app.MapGet(
        "/search/{phrase?}",
        async (string? phrase, ISearchClientFactory searchClientFactory, HttpContext httpContext) =>
        {
            var searchClient = await searchClientFactory.CreateUnauthenticatedClientAsync();
            httpContext.Request.Headers.TryGetValue("User-Agent", out var userAgent);
            var remoteIpAddress = httpContext.Connection.RemoteIpAddress;

            var result = await searchClient.SearchAsync(
                new UnifiedSearchRequestModel { SegmentId = segmentId, ScopeId = scopeId, Phrase = phrase },
                userAgent,
                remoteIpAddress,
                CancellationToken.None);

            return Results.Ok(result);
        })
    .WithName("Search");

app.Run();

Configuration

Bot filtering

Crawlers are automatically filtered out of search tracking based on a list maintained here: monperrus/crawler-user-agents

IP filtering

IP addresses will be filtered if they matches a configured IP or CIDR

IP addresses and CIDR traffic filters can be configured using the Admin Api

User agent filtering

A request is filtered if its user agent contains a configured value. The comparison ignores letter case. Wildcards and regular expressions are not supported, so use a distinctive token such as the name of the product or tool, for example UptimeMonitor.

User agent traffic filters can be configured using the Admin API with the type UserAgent.

Configure Filtering Sample

using Bizzkit.Sdk.EcommerceSearch.Preview;

namespace ecs_traffic_filter_sample.TrafficFiltering;

public class Configuration
{
    public static async Task ConfigureTrafficFilteringAsync()
    {
        var factory = new SearchAdministrationClientFactory(
            new SearchAdministrationConnectionOptions
            {
                BaseUrl = "esc-admin-host",
                Authority = "auth",
                ClientId = "client-id",
                ClientSecret = "client-secret",
                Scope = "searchapi/"
            });

        var client = await factory.CreateAuthenticatedClientAsync();

        await client.TrafficFiltersCreatePreviewAsync(
            new TrafficFilterModel
            {
                Name = "Block Bizzkit",
                TrafficFilterId = "block-bizzkit",
                Type = TrafficFilterType.CIDR,
                Value = "10.10.0.1/24"
            });

        await client.TrafficFiltersCreatePreviewAsync(
            new TrafficFilterModel
            {
                Name = "Block Localhost",
                TrafficFilterId = "block-localhost",
                Type = TrafficFilterType.IP,
                Value = "127.0.0.1"
            });

        await client.TrafficFiltersCreatePreviewAsync(
            new TrafficFilterModel
            {
                Name = "Block uptime monitor",
                TrafficFilterId = "block-uptime-monitor",
                Type = TrafficFilterType.UserAgent,
                Value = "UptimeMonitor"
            });
    }
}