Senior Data Engineer

The role

You will own the Azure Databricks platform, set its standards and roadmap, and lead the data side of a core lending system migration. You will work in a small team of three engineers to build ELT pipelines and a dimensional data model.

What you will do

• Own the data platform roadmap and engineering standards including CI/CD, environments, testing, monitoring and access governance
• Design and lead the build of robust, scalable ELT pipelines on Azure Databricks
• Lead the data workstream for the core lending platform migration
• Build and extend a curated, dimensional data model
• Lift data quality, freshness and alerting
• Keep the platform cost-efficient through cluster optimisation, job orchestration and smart use of compute
• Mentor and develop other data engineers through code review, pairing and clear standards
• Partner with analysts, data scientists, product managers and Engineering

Required qualifications

• 7+ years in data engineering
• Deep hands-on experience on Azure Databricks (Spark, Delta Lake, Unity Catalog)
• Strong Python and SQL
• Experience with CI/CD and deploying pipelines as code (Azure DevOps or GitHub, Databricks Asset Bundles)
• A track record of lifting standards on an existing platform
• Solid dimensional modelling experience (star schemas, conformed dimensions, medallion architecture)
• Experience mentoring engineers and leading technical decisions through design and code review
• The ability to explain technical direction clearly to non-technical stakeholders

Nice to have

• Experience leading the data side of a core system migration

Benefits

• Hybrid working
• Up to 26 weeks paid parental leave
• Strategy Week twice a year
• Fortnightly all-company business updates
• Flat structure
• Day off on your birthday
• End of trip facilities

About the company

Shift provides business finance to small, established and corporate sized businesses in Australia. Founded in 2014, Shift has funded more than 30,000 Australian businesses.