Data Engineer II

The role

You will lead the design, development, and maintenance of complex data engineering projects and pipelines for the G2 data platform. This role involves implementing ETL and ELT processes within an AWS and Snowflake ecosystem.

What you will do

• Lead the design and development of data pipelines for data integration into the G2 Data Platform.
• Optimize data pipelines for cost effectiveness, scalability, and reliability.
• Innovate to keep the data stack aligned with latest trends.
• Service data requests from various users of the G2 data platform.
• Contribute to data modeling and design reviews.
• Execute project tasks aligned with timelines and objectives.
• Develop repeatable and scalable code for real-time data availability.
• Contribute to the development and advancement of the data platform.
• Promote architectural changes to increase scalability.
• Implement data quality and data governance initiatives.

Required qualifications

• 4+ years of experience as a data engineer or ETL developer.
• 2+ years of development experience with sound skills in data modeling, optimization, and database architecture.
• Experience in the design and development of data pipelines using cloud and open-source tools.
• Proficiency in writing and debugging SQL queries.
• Good programming skills in Python or Java.
• Good knowledge of performance tuning, optimization and debugging of data pipelines.
• Working knowledge of the AWS data services (DynamoDB, RDS, Data Pipeline, EMR, Lambda, Glue, ECS, etc.) and cloud data warehouses like SnowFlake.
• Proficiency in handling structured and unstructured data.
• Proficiency in ELT/ELT tools like AWS Glue, Step Functions, Data Pipelines, Airflow, Airbyte, and DBT.
• Familiarity with distributed computing and frameworks like Apache Spark, Hadoop, and Apache Kafka.
• Familiarity with software engineering principles and best practices.

Nice to have

• Experience with Docker and Kubernetes.
• Proficiency in data modeling, schema design, and optimizing data structures for performance in Snowflake.
• Working experience in startup environments.
• Experience with Agile process methodology, CI/CD automation, Test Driven Development.
• Knowledge of data governance, security, and compliance standards within cloud-based data solutions.
• Understanding of any reporting tools such as Tableau, Qlikview ,Looker, PowerBI, etc.
• Database administration background.

About the company

G2 is the world's largest and most trusted software marketplace. It has joined forces with Capterra, SoftwareAdvice, and GetApp to create a large source of online data and software insights.