Skip to content

Traffic Filtering

Traffic filtering can be used to avoid crawlers and specfic IP addresses or CIDR ranges to be included in Search tracking

Using Traffic Filtering

The search api uses the following headers to find the Crawler and IP when a search request is received.

  • Crawlers: X-Forwarded-User-Agent
  • IP addresses: X-Forwarded-For

When using the Search SDK the following can be used to forward headers from an incoming request to the search method

Traffic Filtering Sample

Full tutorial sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
using Bizzkit.Sdk.Search;

const string segmentId = "merch-b2c-da";
const string scopeId = "full-search";
const string searchHost = "https://ecs-search.bizzkit.it/";

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<
    ISearchClientFactory>(
    new SearchClientFactory(
        new SearchConnectionOptions { BaseUrl = searchHost },
        new HttpClient()));

var app = builder.Build();

app.MapGet(
        "/search/{phrase?}",
        async (string? phrase, ISearchClientFactory searchClientFactory, HttpContext httpContext) =>
        {
            var searchClient = await searchClientFactory.CreateUnauthenticatedClientAsync();
            httpContext.Request.Headers.TryGetValue("User-Agent", out var userAgent);
            var remoteIpAddress = httpContext.Connection.RemoteIpAddress;

            var result = await searchClient.SearchAsync(
                new UnifiedSearchRequestModel { SegmentId = segmentId, ScopeId = scopeId, Phrase = phrase },
                userAgent,
                remoteIpAddress,
                CancellationToken.None);

            return Results.Ok(result);
        })
    .WithName("Search");

app.Run();

Configuration

Bot filtering

Crawlers are automatically filtered out of search tracking based on a list maintained here: monperrus/crawler-user-agents

IP filtering

IP addresses will be filtered if they matches a configured IP or CIDR

IP addresses and CIDR traffic filters can be configured using the Admin Api

Configure Filtering Sample 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
using Bizzkit.Sdk.EcommerceSearch.Preview;

namespace ecs_traffic_filter_sample.TrafficFiltering;

public class Configuration
{
    public static async Task ConfigureTrafficFilteringAsync()
    {
        var factory = new SearchAdministrationClientFactory(
            new SearchAdministrationConnectionOptions
            {
                BaseUrl = "esc-admin-host",
                Authority = "auth",
                ClientId = "client-id",
                ClientSecret = "client-secret",
                Scope = "searchapi/"
            });

        var client = await factory.CreateAuthenticatedClientAsync();

        await client.TrafficFiltersCreateAsync(
            new TrafficFilterModel
            {
                Name = "Block Bizzkit",
                TrafficFilterId = "block-bizzkit",
                Type = TrafficFilterType.CIDR,
                Value = "10.10.0.1/24"
            });

        await client.TrafficFiltersCreateAsync(
            new TrafficFilterModel
            {
                Name = "Block Localhost",
                TrafficFilterId = "block-localhost",
                Type = TrafficFilterType.IP,
                Value = "127.0.0.1"
            });
    }
}