Member of Technical Staff (Backend Software Engineer, Search Crawler)

The role

You will design, develop, and operate systems that ingest, process, and manage web-scale data. This role involves building large-scale web crawlers, ingestion pipelines, and data processing systems.

What you will do

• Lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems.
• Build, maintain, and optimize core backend and frontend components for crawler services.
• Collaborate with Search and Infrastructure teams to ensure reliable ingestion and processing of web pages.
• Architect and implement fullstack features and scalable distributed systems.
• Iterate and experiment with novel approaches to enhance system performance, usability, and reliability.

Required qualifications

• Minimum of 5 years of software development experience.
• Strong knowledge of data structures and algorithms in at least one of the following languages: Python, C++, Rust, or Go.
• Proven experience building, deploying, and optimizing high-load, distributed, and hardware-adjacent services.
• Deep understanding of cloud infrastructure.
• Hands-on experience in Kubernetes (K8s) and AWS.
• Demonstrated passion for writing clean, efficient, and scalable systems.

Nice to have

• Experience with large-scale web crawlers is highly desirable.

About the company

Perplexity is seeking an experienced Backend Software Engineer to join their Crawler team.