Member of Technical Staff (Search Crawler Analyst)

The role

You will work at the intersection of data analysis and engineering. You will design metrics, build data pipelines, and improve the quality of search and answer systems.

What you will do

• Find and diagnose quality issues in our crawling pipeline
• Train small models that optimize particular aspects of the pipeline
• Build datasets for model training, including LLM-as-a-judge labeling pipelines
• Improve page selection algorithms for indexing
• Design and analyze experiments to validate improvements

Required qualifications

• 4+ years of experience as a data analyst, ML engineer, or in a related role
• Strong coding skills to write production-grade code at the level of a mid-level backend engineer
• Experience designing metrics from scratch
• Experience training ML models that shipped to production with measurable metric improvements

Nice to have

• Direct experience working on web crawling or indexing pipelines

About the company

Perplexity has a complex crawling and storage system with multiple stages including URL discovery, crawling, parsing, and indexing.