Senior Software Engineer, Data Platform

The role

You will build and operate infrastructure that moves data across the career marketplace and AI products. You will own systems that orchestrate batch workloads and deliver data to engineers, analysts, and product teams.

What you will do

• Lead technical direction and roadmap for Airflow and Astronomer platform.
• Own production operations including deployment, DAG packaging, scheduler, worker capacity, upgrades, observability, and incident response.
• Build paved paths for authoring, testing, deploying, and operating workflows.
• Design and operate streaming and change data capture pipelines.
• Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay.
• Improve cloud foundation with Kubernetes, Terraform, IAM, secrets, CI/CD, and cost-aware capacity management.
• Lead cross-team design decisions on data contracts, interfaces, and operational ownership.
• Mentor engineers and align partners on scalable platform approaches.
• Participate in on-call and troubleshoot production failures.
• Build data and orchestration foundations for AI workloads.

Required qualifications

• Strong software engineering skills in Python.
• Experience building and operating production data or distributed systems.
• Deep hands-on Airflow experience including scheduling, execution behavior, deployment, scaling, upgrades, monitoring, and debugging failures.
• Experience with event-driven or streaming data infrastructure, including a message broker or managed event bus and a stream processing system.
• Sound understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and recovery or replay in production pipelines.
• Experience with cloud infrastructure and infrastructure as code.
• Comfortable working with containers, Kubernetes, CI/CD, access controls, and production observability.
• A track record of leading ambiguous infrastructure initiatives.
• Clear communication and a track record of partnering across teams while owning systems through production support.

Nice to have

• GCP services such as Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage.
• Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark or Dataproc.
• Supporting both batch and low-latency consumers, including product-facing data services or ML features.
• Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access.

Benefits

• Equity in a fast-growing company
• 401(k) match
• Competitive compensation
• Financial coaching
• Paid parental leave
• Fertility benefits
• Parental coaching
• Medical, dental, & vision
• Mental health support
• $500 wellness stipend
• $2,000 learning stipend
• Internet, commuting, & free lunch/gym in our SF office

About the company

Handshake powers 25 million job seekers, 1 million+ employers, and 1,600 educational institutions. Handshake AI is an AI data business that works with frontier AI lab researchers to create evaluations and publish benchmarks.