Software Engineer - Web Crawling

The role

You will build distributed data acquisition systems to capture and structure the live web. You will design and implement large-scale, intelligent web acquisition pipelines to provide AI agents with real-time knowledge.

What you will do

• Build distributed data acquisition systems that capture and structure the live web
• Investigate and analyze browser internals, fingerprinting, and anti-automation systems
• Prototype and deploy intelligent automation frameworks using Playwright, Puppeteer, and CDP
• Conduct research into network flows, JavaScript-based protections, and emerging web standards
• Collaborate with AI and infrastructure teams to integrate real-time web data into retrieval pipelines
• Translate technical insights into production-grade components
• Evolve capabilities in resilience, speed, and authenticity of web interaction

Required qualifications

• 3–5 years of experience as a backend or systems engineer
• Strong programming skills in Python or Node.js
• Proven experience building and maintaining browser automation systems (Playwright, Puppeteer, or CDP) in production environments
• Solid understanding of browser internals, network protocols, and web interaction mechanisms
• Experience designing high-performance, resilient systems that handle scale, concurrency, and complex orchestration
• Strong debugging and analytical skills
• Excellent documentation and communication skills

Nice to have

• Experience in Go or C++
• Familiarity with cloud infrastructure, containerization (Docker), Kubernetes, and CI/CD
• Experience integrating AI or retrieval systems into production pipelines

Before you apply

• Requires existing legal authorization to work in Israel
• Tavily is not able to provide visa sponsorship for this role

About the company

Tavily is building the search engine for AI agents. Their Search API provides web access for AI agents, enabling real-time internet search optimized for LLMs and Retrieval-Augmented Generation (RAG).