Principal Engineer, Compute Fleet Management

The role

The Technical Lead for Compute Fleet Management will set standards for how Databricks consumes and optimizes compute across AWS, Azure, and GCP. This person will pioneer fleet optimization, build hyper-scale resilient architecture, and lead the development of low-dependency systems to manage the compute platform.

What you will do

• Provisioning and pooling of cloud resources to achieve peak workload performance and resource isolation.
• Building architecture that guarantees horizontal scaling and resilience against zonal or cloud account-level failures.
• Leading the development of the lowest-dependency systems required to bootstrap and manage the compute platform.
• Achieving and maintaining 99.99% availability for all batch and serving workloads.
• Driving utilization to 60% or higher.
• Architecting and enforcing strong security and performance isolation across customer workloads.
• Taking ownership of complex, cross-team, cross-layer, and multi-quarter strategic engineering initiatives.
• Driving consensus and establishing technical direction across organizational boundaries.
• Planning, tracking project progress, and managing complex cross-organizational dependencies.

Required qualifications

• Successfully operated large-scale, mission-critical infrastructure systems in production.
• Leading transformative projects from concept to execution.
• Deep, hands-on experience developing and operating high-scale distributed systems on at least one major public cloud.
• Proven ability to drive consensus, establish technical direction, and lead large technical efforts across organizational boundaries.
• Exceptional strength in planning, tracking project progress, and managing complex cross-organizational dependencies.

Nice to have

• Experience managing and scaling a massive fleet of GPUs for AI/ML workloads.
• Experience with developing and operating large-scale distributed systems across all major clouds (AWS, Azure, and GCP).

Benefits

• Annual performance bonus
• Equity

About the company

Databricks is the Data and AI company. More than 20,000 organizations worldwide rely on the Databricks Data + AI Platform to build and scale data and AI apps, analytics and agents.