Member of Technical Staff (Search Core, Indexing & Data Pipeline Stream)

The role

This role involves designing and building core search engine components like indexing pipelines and data storage formats. You will develop streaming and batch data processing systems and optimize hardware performance.

What you will do

• Design and build core search engine components, especially indexing pipelines, data storage formats that operate at the scale of billions of pages
• Develop streaming and batch data processing with YT systems for search index construction in a high-load environment
• Push the limits of hardware performance through low-level optimizations and system-level tuning
• Tackle challenges in multithreading, concurrency, and system-level optimization
• Design and build core search engine components, including indexing pipelines, retrieval algorithms, and ranking systems that operate at the scale of billions of pages
• Develop streaming and batch data processing systems for search index construction in a high-load environment
• Push the limits of hardware performance through low-level optimizations and systems-level tuning
• Tackle challenges in multithreading, concurrency, and system-level optimization

Required qualifications

• 3+ years of hands-on experience in systems programming (Rust, C++, C, or similar)
• Ownership of full project lifecycle
• Knowledge of Python or other scripting languages
• Knowledge of Data processing systems (YT/Kafka/Hadoop)
• Passion for writing clean, efficient, and scalable systems-level code
• Strong knowledge of algorithms and data structures
• Deep understanding of multithreading, including various approaches, challenges, and trade-offs
• Experience building high-load, distributed, and hardware-adjacent services
• Solid understanding of Linux internals (syscalls, networking stack, memory model, kernel tuning)
• Familiarity with low-level optimization techniques (memory management, cache efficiency, SIMD, profiling)

Nice to have

• Experience developing core components of search engines, databases, or information retrieval systems
• Understanding of search fundamentals: indexing, query parsing, ranking, and relevance
• Experience with trading systems or other latency-sensitive real-time systems
• Familiarity with cloud services, Kubernetes, and AWS infrastructure

About the company

Perplexity is looking for a highly skilled Expert Systems Engineer to join our Search Core team.