Data Engineer, Core Experimentation

The role

You will build and manage data pipelines and core tables to power analyses, safety systems, product growth, and bad actor prevention. You will also develop canonical datasets to track product metrics and collaborate with various teams to provide data solutions.

What you will do

• Design, build and manage data pipelines.
• Ensure user event data is integrated into the data warehouse.
• Develop canonical datasets to track user growth, engagement, and revenue.
• Work with Infrastructure, Data Science, Product, Marketing, Finance, and Research teams.
• Implement robust and fault-tolerant systems for data ingestion and processing.
• Participate in data architecture and engineering decisions.
• Ensure data security, integrity, and compliance.

Required qualifications

• 3+ years of experience as a data engineer.
• 8+ years of any software engineering experience (including data engineering).
• Proficiency in at least one programming language such as Python, Scala, or Java.
• Experience with distributed processing technologies and frameworks, such as Hadoop, Flink and distributed storage systems (e.g., HDFS, S3).
• Expertise with any of ETL schedulers such as Airflow, Dagster, Prefect or similar frameworks.
• Solid understanding of Spark and ability to write, debug and optimize Spark code.

About the company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.