Member of Technical Staff, Web Crawl Engineer

The role

You will build and operate large-scale web crawling systems to discover, acquire, and process content from the internet. You will own infrastructure for URL discovery, scheduling, distributed crawling, content extraction, and dataset delivery.

What you will do

• Build and operate web-scale crawling infrastructure
• Design and optimize URL discovery, prioritization, scheduling, and crawl orchestration systems
• Develop distributed crawlers that acquire content while respecting site constraints
• Build systems for content extraction, rendering, parsing, and normalization
• Improve crawl coverage, freshness, efficiency, and quality through measurement and experimentation
• Design infrastructure for large-scale recrawling, change detection, and incremental updates
• Develop specialized crawlers for high-value domains and dynamic websites
• Analyze crawl performance and web coverage to identify gaps and inefficiencies
• Build observability, monitoring, and reliability systems
• Debug production issues and improve performance, scalability, and resilience

Required qualifications

• Experience building large-scale web crawling, search indexing, content acquisition, or internet-scale data collection systems
• Strong understanding of crawling architectures, URL frontier management, scheduling, and distributed crawl coordination
• Experience with large-scale distributed systems using technologies such as Ray, Spark, Beam, Flink, or similar frameworks
• Familiarity with content extraction, HTML parsing, browser automation, rendering systems, and modern web technologies
• Experience operating systems that process petabyte-scale datasets
• Strong systems engineering skills, including reliability, observability, performance optimization, and debugging
• Experience designing experiments and using data to improve crawl quality, coverage, and efficiency
• Excellent communication skills

Nice to have

• Experience building search engines, web indexes, or internet-scale crawling platforms
• Familiarity with dynamic web content, browser automation, and large-scale extraction pipelines
• Understanding of how web data is used in training and evaluating large language models
• Experience with distributed storage systems, content deduplication, and web-scale dataset management

Benefits

• Salary and equity
• Stock options
• Comprehensive medical, dental, vision, and life
• Annual wellness allowance
• Lunch and dinner provided in the office daily
• 22 weeks paid parental leave for all new birthing and non-birthing parents
• Unlimited paid time off in the U.S.
• 30 days paid time off in the U.K.
• Visa sponsorship support
• Regular off-sites, happy hours, and team celebrations

Before you apply

• This position may require access to technology or source code subject to the U.S. Export Administration Regulations
• Any offer of employment may be conditioned on the Company's ability to provide access to technology or source code in compliance with U.S. export control laws

About the company

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. They build open models to help shape the future of AI.