Data Platform Engineer, Ingestion

New York City. Posted 20 days ago

The role

You will build ingestion pipelines for customer and vendor data. You will automate onboarding and create reusable modules for other Minerva teams.

What you will do

Build pipelines for APIs, files, database extracts and approved scraping sources.
Make reruns safe through incremental loads, backfills and retries.
Automate onboarding from source configuration to first delivery.
Build reusable connectors and storage components for other teams.
Carry source metadata through the pipeline.
Enforce privacy controls.
Own integrations in production including monitoring and recovery runbooks.

Required qualifications

Typically three or more years building and supporting production data pipelines, or equivalent ownership.
Experience bringing in data from many vendors and source types, including messy data at scale.
Practical judgment about incremental processing, backfills, failures and cost.
Experience automating repeated work or building components other engineers used.
Strong Python and SQL.
Ability to pin down what a source owner needs and define support boundaries.
In New York City or willing to relocate.

Nice to have

Connector frameworks, whether off the shelf or your own.
Onboarding automation or self-service data tooling.
Consumer, identity or contact data, and the privacy rules that come with it.
Dagster, Iceberg or Snowflake in production.

Benefits

Meaningful equity
Medical
Dental
Vision

Before you apply

Applications without a few paragraphs walking through one pipeline you built for data you didn't control (the source, what broke, how you made reruns safe and how the next source got easier) won't be

About the company

Minerva builds AI for marketing leaders to automate and accelerate workflows with AI agents. The team includes operators and investors from Citadel, Dentsu, Bridgewater, Meta Superintelligence, Growthloop, Amazon and Lazard, alongside engineers and researchers from Berkeley, MIT, Stanford and Cambridge.