Principal Software Development Engineer - Agentic Systems

The role

This Principal Software Development Engineer will lead the architecture and delivery of large-scale AI/ML infrastructure and production pipelines. The role also involves mentoring team members and driving strategic technical initiatives across the organization.

What you will do

• Lead the architecture and development of scalable AI/ML infrastructure, LLM serving platforms, and agentic workflows for cloud-scale cybersecurity use cases
• Architect reusable platform foundations, including networking, security, authentication, and monitoring, to standardize and secure enterprise AI/ML services
• Define and implement system observability, evaluation, and reliability frameworks for AI systems in production
• Drive cross-functional architectural initiatives with product, security, and operations teams
• Provide technical leadership and mentorship on distributed systems design
• Translate complex architectures to non-technical stakeholders

Required qualifications

• Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions
• Proven experience designing and operating large-scale, low-latency distributed systems, including availability, fault tolerance, consistency, and cloud-native microservices
• AWS preferred; GCP or Azure also accepted
• Expertise in LLMs and production AI/ML infrastructure, including orchestration frameworks, inference systems, and full-lifecycle pipeline operations
• Demonstrated track record of cross-organizational technical leadership, driving architectural decisions, mentoring teams, and shipping production-grade services
• Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field

Nice to have

• Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency
• Track record with LLM observability, explainability, and evaluation frameworks in production, plus serverless architectures for data-intensive workloads
• Deep foundation in systems, networking, and security architecture for distributed systems, including identity, encryption, access control, VPC, and network security

Benefits

• Various health plans
• Time off plans for vacation and sick time
• Parental leave options
• Retirement options
• Education reimbursement
• In-office perks

About the company

Zscaler (NASDAQ: ZS) provides a cloud security platform, the Zero Trust Exchange, that securely connects users, devices, and applications in any location. The company describes it as the world's largest in-line cloud security platform.