Lead Data Engineer

The role

You will design and deliver high-performance data platforms for analytics, AI/ML, and real-time insights. You will mentor junior engineers and drive architectural decisions to align the data platform roadmap with business objectives.

What you will do

• Lead the design and implementation of scalable and reliable data pipelines using Distributed Computing, Kafka, Airflow and Temporal.
• Architect data solutions that support business analytics, and real-time processing.
• Mentor and guide junior engineers, conducting code reviews and setting coding standards.
• Collaborate with cross-functional teams to identify and solve data challenges.
• Drive improvements in infrastructure, automation, and monitoring to enhance scalability and performance.
• Ensure data quality, governance, and compliance with internal policies and industry best practices.
• Stay current with the latest data trends and technologies, and make recommendations to keep the platform ahead of the curve.

Required qualifications

• 6-8 years of experience in data engineering or distributed systems.
• Strong hands on programming skills in Python / Scala / Rust / Golang, and SQL.
• Proven experience with cloud-native services (AWS, GCP) and big data tools (e.g., Spark, Flink, Datafusion, Doris, Trino etc).
• Expertise in building real-time pipelines using Kafka, and batch pipelines.
• Experience working with CI/CD pipelines, Kubernetes, Docker, and IaC tools.
• Demonstrated ability to lead and mentor teams in a fast-paced, agile environment.
• Strong problem-solving and communication skills with the ability to collaborate with multiple stakeholders.

Nice to have

• Contributions to open-source projects
• experience in a fast pace product-based company

Benefits

• Competitive compensation and benefits package.
• Opportunity to work with modern technologies and drive innovation.
• Flexible working hours with Hybrid options.
• A collaborative environment focused on learning and growth.

About the company

Arcana is a front-office portfolio intelligence platform built for hedge funds and asset managers. It empowers investment professionals with tools to understand performance and risk, construct portfolios, evaluate investments, and make better investment decisions.