Research Crawling Engineer

The role

You will design and operate large-scale web data acquisition systems for research and model development. This work spans distributed systems, scraping infrastructure, and data pipelines.

What you will do

• Build and maintain large-scale web crawlers across diverse domains
• Design high-throughput, fault-tolerant systems for data collection
• Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites
• Develop pipelines for cleaning, deduplication, filtering, and normalization
• Construct and maintain datasets for research and model training
• Monitor crawl performance, coverage, and data quality
• Collaborate with research teams to align data collection with modeling needs
• Optimize infrastructure for cost, latency, and reliability

Required qualifications

• Strong programming experience in one or more of: Go, Rust, Python, Java, or C++
• Experience building web crawlers or large-scale data pipelines
• Solid understanding of HTTP, networking, and browser behavior
• Familiarity with distributed systems and parallel processing
• Ability to debug unstable or adversarial environments

Nice to have

• Experience with large datasets (TB–PB scale preferred)
• Experience with NLP pipelines or dataset curation for ML
• Familiarity with LLM pretraining data or retrieval systems
• Experience with headless browsers (e.g., Chrome DevTools Protocol, Playwright, Puppeteer)
• Knowledge of proxy systems, IP rotation, and large-scale request orchestration
• Background in data quality evaluation or benchmarking
• Experience running workloads on cloud or bare-metal infrastructure

Benefits

• competitive salary
• benefits

About the company

wynd-labs builds infrastructure that delivers massive amounts of web data to companies training AI models. They power Grass, a bandwidth-sharing network that operates a massive distributed crawler.