Data Engineer (Elasticsearch + Datawarehousing)

The role

You will design, build, and maintain data pipelines and infrastructure. You will work on end-to-end data solutions including Elasticsearch clusters and data warehouses.

What you will do

• Design, implement, and optimize Elasticsearch clusters.
• Build and manage Elasticsearch indexes.
• Build and optimize data storage solutions like data lakes and warehouses.
• Integrate structured and unstructured data from various systems.
• Ensure data accuracy, consistency, and completeness through validation and transformation.
• Maintain documentation for data processes, tools, and systems.
• Collaborate with product managers to gather requirements.
• Participate in Agile development processes.
• Conduct testing and debugging to ensure reliability and performance.
• Propose innovative solutions to improve application performance and scalability.

Required qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• Minimum of 3 years of experience as a Data Engineer.
• Proficiency in Elasticsearch and Python programming language.
• Experience with database technologies such as SQL (e.g., MySQL, PostgreSQL) and NoSQL (e.g., MongoDB) databases.
• Utilize Java to build and enhance backend systems.
• Develop APIs, microservices, and automation scripts.
• Competence in working on Linux OS and creating shell scripts.
• Solid understanding of software engineering principles, design patterns, and best practices.
• Excellent problem-solving and analytical skills.
• Effective communication skills, both written and verbal.

Nice to have

• Strong understanding of Programming Libraries/Frameworks and technologies such as Flask, API frameworks, Data warehousing/lakehouse, Principles, Database, ORM, Data analysis, Databricks, Pandas, Spark
• Utilities & Tools: logging, requests, subprocess, regex, pytest
• ELK stack, Redis, distributed task queues
• Strong understanding of data warehousing/lakehousing principles and concurrent/parallel processing concepts.
• Familiarity with at least one cloud data engineering stack (Azure, AWS, or GCP).
• Familiarity with version control systems like Git.

About the company

Particle41 is driven by core values of Empowering, Leadership, Innovation, Teamwork, and Excellence (ELITE).