Principal Production Engineer

The role

This person will design and build highly available, scalable infrastructure across AWS, GCP and bare-metal environments. They will drive an automation-first culture by writing Python or Go code to remove manual work, and will lead incident response on a 24/7 on-call rotation.

What you will do

• Design and implement highly available, scalable infrastructure across AWS, GCP and bare-metal environments
• Drive an automation-first culture by writing Python or Go code to eliminate manual toil and build self-healing systems
• Implement and maintain observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs and establish error budgets
• Act as lead Incident Commander (TDO on-call), develop response playbooks and conduct post-incident analyses
• Partner with Engineering and partner teams to conduct operability reviews

Required qualifications

• Foundational understanding of AI/ML technologies and experience leveraging, securing or positioning AI-driven solutions
• 10+ years managing reliability, scalability and availability for large-scale production services
• Deep expertise in programming (e.g., Python, Go, or C/C++)
• Strong background in networking protocols, Linux/RHEL systems and distributed architecture
• Experience in high-stakes incident management and participation in a 24/7 on-call rotation
• Proficiency in ITIL frameworks and incident data to drive service maturity through problem management and operability reviews

Nice to have

• Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency
• Extensive experience with public cloud (AWS, Azure, GCP) and Infrastructure-as-Code (Ansible, Terraform, Helm, Temporal)
• Experience with chaos engineering and disaster recovery planning at scale
• Expertise in global routing (BGP) and traffic tunneling (GRE, IPSec), L7 proxy architectures (HAProxy), DNS at scale and OS networking stack internals

Benefits

• Various health plans
• Time off plans for vacation and sick time
• Parental leave options
• Retirement options
• Education reimbursement
• In-office perks

Before you apply

• Applicants must adhere to applicable laws, regulations and Zscaler security and privacy policies

About the company

Zscaler is a security company whose Zero Trust Exchange platform protects customers from cyberattacks and data loss. Its SASE-based platform is distributed across 200+ public data centers and thousands of private edge sites.