Senior Data Engineer, Platform

The role

You will own the data infrastructure, including compute, orchestration, deployment, and connectivity to partners. You will set the technical direction for the platform, making architecture decisions and building it as software using infrastructure as code.

What you will do

• Shape the data platform roadmap and own delivery of work
• Set technical direction, design standards, and mentor engineers
• Own platform architecture on Databricks and AWS
• Own connectivity to partners and systems via SFTP, S3, and private networking
• Decide where and how jobs run to ensure they are fast and economical
• Own orchestration and deploys using Terraform and GitLab CI/CD
• Make the platform reliable and observable
• Make data usable by AI through governed metric definitions and MCP servers
• Validate data arrival against promises
• Replace manual work with automation and AI agents

Required qualifications

• Seven or more years building production software or data systems
• Several years owning data infrastructure or pipelines end to end
• Hands-on production experience with at least one distributed data processing engine at scale (Spark, Databricks, Snowflake, BigQuery, Flink, Dask or Ray)
• Strong Python
• Strong SQL
• Experience designing systems on a major cloud, AWS preferred
• Experience with automated testing, CI/CD, infrastructure as code, code review, observability and incident response
• Experience leading technical work and mentoring other engineers
• Use of AI coding agents in daily engineering work

Nice to have

• Databricks, Delta Lake, Unity Catalog and dbt
• Databricks Genie, MCP servers, or other ways of putting governed data in front of AI agents
• Healthcare data: claims (837/835), eligibility (834), FHIR, HL7 v2 and hospital event feeds, or another regulated setting such as HIPAA or SOC 2
• Workflow orchestration (Databricks Workflows, Temporal), event-driven ingestion and serverless compute
• Experience leading or managing engineers, or a clear interest in growing into a data engineering lead

About the company

Vanna Health helps people living with serious mental illness thrive by strengthening the community ecosystem around them. They use innovative payment models and technology to promote collaboration across behavioral health and primary care providers.