Skip to content

Common pitfalls

This page describes common pitfalls that developers fall into. Be sure to read this page carefully. By avoiding these common pitfalls and following best practices, solution partners can ensure efficient and effective operation, minimizing the load on the environment and maintaining optimal performance.

Ingestion pitfalls

Inefficient update strategies

Each time data is ingested into Ecommerce Search the data is processed to handle changes to normalization, pricing, suggestions, and other data driven features. Poor data ingestion strategies can lead to increased system load as data is processed, which can be both time-consuming and resource-intensive.

Examples of poor data ingestion strategies include:

  • Bulk deleting & creating products instead of patching
  • Patching unchanged fields on product updates
  • Patching unchanged products
  • Ingesting products way before they become active
  • Full rebuilds of the dataset
  • Querying search/export endpoints to identify outdated data

It is more efficient to use partial updates. By sending only the changed products and only including changed fields, the system can quickly apply updates without the need to reprocess the entire dataset, thus saving time and resources. Full rebuilds are resource-intensive and should be reserved for critical situations, such as major structural changes to data model or data corruption.

Effective partial updates requires the integrating solution to keep track of data ingested into Ecommerce Search.

It is recommended to:

  • Track ingested products
    • Maintain a record of products already ingested into Ecommerce Search. This helps in identifying which products need updates and which do not.
  • Identify and track changes
    • Implement a mechanism to detect changes in product data. This may be achieved through versioning, timestamped product records, or a product hash.
  • Employ selective updates
    • Develop a system to send only the changed fields for updates. This minimizes data processing and speeds up the update process.
  • Delete unnecessary data
    • Ensure that products no longer needed are deleted from the search index to keep it clean and efficient.