Software Engineer, Web Crawling

The role

You will build a distributed crawler that can handle 100M+ pages per day. You will work on Google-scale crawling to build massive-scale ML systems.

What you will do

• Build a distributed crawler that can handle 100M+ pages per day
• Optimize crawl politeness and rate limiting across thousands of domains
• Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
• Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

Required qualifications

• Extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
• Experience with some high performance language (C++, Rust, etc.)
• Familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
• Comfortable optimizing a system to an exceptional degree

Benefits

• Premium healthcare benefits (medical, dental, vision)
• Fertility benefits
• 16 weeks of fully paid parental leave for all new parents
• Monthly wellness stipend

Before you apply

• We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3)

About the company

Exa is an applied AI lab building a search engine. They build massive-scale infra to crawl the entire web, train embedding models, and design vector databases.