Staff Engineer, Data Platform (R5659)

The role

This person will design and build data services for dataset discovery, versioning, search and metadata. The work supports machine-learning engineers in reusing and tracing training data.

What you will do

• Build dataset discovery, catalog and retrieval services, including LanceDB integration for indexing and search.
• Create APIs that let developers pin an exact dataset version and retrieve only the data they are authorized to access.
• Track where data came from and how it changed so engineers can understand and reproduce training results.
• Design schema validation, compatibility and migration mechanisms that preserve access to historical datasets.
• Define common identifiers and metadata relationships that connect datasets to experiments and model artifacts.
• Work with machine-learning engineers to understand how they create and use datasets, then turn recurring needs into reusable tools.
• Deliver reliable data services with tested upgrades, clear diagnostics and support for large datasets.

Required qualifications

• Experience designing and implementing production data platforms or data-intensive backend services.
• Strong software engineering skills in Python or a comparable backend language.
• Experience with data indexing, versioning, metadata, storage and access control.
• Experience evolving schemas or APIs while supporting existing users and historical data.
• Ability to lead technical design and implement substantial parts of the system.
• Ability to work with data or machine-learning users and translate their needs into reliable services.

Nice to have

• Experience with LanceDB, Arrow, Parquet or comparable columnar and vector-data systems.
• Experience with dataset catalogs, provenance or reproducible machine-learning workflows.
• Experience applying domain models or ontologies to practical data relationships and schema evolution.
• Experience deploying data services on Kubernetes or in on-premises and disconnected environments.

Benefits

• Full-time package includes bonus.
• Full-time package includes benefits.
• Full-time package includes equity.
• Temporary employees receive a temporary benefits package, applicable after 60 days of employment.
• Benefits are not available to military fellows or part-time employees.

Before you apply

• All offers are contingent on a cleared background check and possible reference check.

About the company

Shield AI is a venture-backed defense-tech company that builds intelligent systems to protect service members and civilians. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation technologies.