Data Engineer

The role

The Data Engineer will design and build the backbone of AI-powered data solutions. This includes building PB scale data pipelines, cloud-native data infrastructure, and an intelligent SQL ecosystem.

What you will do

• Building highly performant large scale data infrastructure that can scale to 100K+ jobs and handle PB scale data per day.
• Design and implement robust, scalable data infrastructure on AWS, utilizing Kubernetes and Airflow for efficient resource management and deployment.
• Design and develop a comprehensive SQL intelligence system encompassing query optimization, dynamic pipeline generation, and data lineage tracking.
• Leverage expertise in SQL query profiling, AST analysis, and parsing to create a sophisticated engine focused on query performance improvements, building adaptive data pipelines, and implementing gran
• Contribute to open-source initiatives.

Required qualifications

• 3-5 years of experience in data engineering, with a focus on building scalable data pipelines and systems
• Strong proficiency in Python and SQL
• Extensive experience with SQL query profiling, optimization, and performance tuning
• Deep understanding of SQL Abstract Syntax Tree (AST) and experience working with SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs
• Experience in building data pipelines using Airflow or dbt

Nice to have

• Solid understanding of cloud platforms, particularly AWS
• Familiarity with Kubernetes (K8s) for containerized deployments
• Experience with Snowflake

About the company

Altimate AI was founded in 2022 in San Francisco to revolutionize enterprise data operations through AI. The team consists of Silicon Valley veterans who have previously created category-defining data and AI products.