TALK: BUILDING REAL-TIME SEARCH AT MAILCHIMP

To me, one of the most frustrating things is looking for something that I know is there, but not being able to find it. At MailChimp, this was a frequent pain point for our users trying to search their campaigns or contacts using the in-app search service. At the time, the underlying search service was plagued by data drift and frequently required developer support.

In order to address these user concerns, we explored alternative architectures and built out new infrastructure while trying not to knock over the existing system. This talk will discuss some of the technical decisions made, as well as break down the roles of each component in the indexing pipeline (Kafka) and new search feature (Elasticsearch). As with many distributed systems, we encountered production issues ranging from uneven data partitioning to variable indexing lag, which we tackled on both the operations and development side. Tooling, whether in-house or open-source (Grafana), is essential when debugging and gaining visibility into these types of data-intensive applications.