Senior Staff Data Engineer, Databricks (R6508)

The role

Design and build data ingestion and transformation pipelines into the Databricks lakehouse using Delta Lake. Own Bronze and Silver layer pipelines, data quality controls, CI/CD for data platform assets, and documentation for downstream teams.

What you will do

• Design and build batch and streaming ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake.
• Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
• Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
• Define and evolve reusable ingestion and transformation patterns and engineering standards for other domains to adopt.
• Implement and maintain Databricks platform constructs, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.
• Build and maintain CI/CD pipelines for data platform assets, including promotion across dev, test, and prod.
• Apply data classification, segregation, and handling requirements within pipeline design.
• Build data quality controls that reflect business meaning, record integrity, completeness, and expected domain behavior.
• Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules.
• Maintain documentation for source objects, ingestion logic, transformations, data quality rules, and known limitations.

Required qualifications

• 12+ years of data engineering experience, including hands-on ownership of production data pipelines.
• Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
• Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
• Experience with batch, CDC, and/or streaming ingestion patterns and their operational trade-offs.
• Experience with CI/CD and deployment automation for data pipelines, including version control, testing, and controlled promotion across environments.
• Strong SQL skills and solid grounding in data modeling fundamentals.
• Demonstrated ability to understand the business and regulatory context of the data being processed.
• Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows.
• Ability to design pipelines with awareness of data sensitivity, ownership, permitted use, and downstream impact.
• Comfort operating in a fast-moving platform build where engineers shape standards.

Nice to have

• Databricks certification such as Data Engineer Associate or Professional.
• Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting.
• Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms.
• Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment.
• Experience enabling multiple business domains on a shared data platform while maintaining governance and ownership boundaries.

Benefits

• Full-time regular employees: pay within the listed range, bonus, benefits, and equity.
• Temporary employees: pay within the listed range and a temporary benefits package, applicable after 60 days of employment.

Before you apply

• All offers are contingent on a cleared background and possible reference check.
• Pay depends on factors including skill set, experience level, licenses and certifications, and work location.

About the company

Shield AI is a venture-backed defense-tech company that builds intelligent systems to protect service members and civilians. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation technologies.