Lead Data Engineer

Buscojobs; Greater London; On-site; AUD 190,000 - 200,000; Full time

Job summary

An established industry player in technology is searching for a Lead Data Engineer to spearhead the development of cloud Data Platform teams. This role offers a unique opportunity to influence the company's data infrastructure while leading strategic initiatives. The successful candidate will leverage their expertise in AWS, Databricks, and data warehousing methodologies to enhance data quality and governance. Join a collaborative environment that values innovation and provides ample career development opportunities. If you are passionate about data engineering and eager to make a significant impact, this position is perfect for you.

Qualifications

• 6+ years of experience in data engineering with a focus on AWS and Databricks.
• Strong understanding of data architecture and data governance.

Responsibilities

• Lead design and implementation of data pipelines and ETL processes on AWS Databricks.
• Ensure data quality, optimisation, and security best practices are followed.

Skills

Data Engineering

AWS

Databricks

Data Warehousing

ETL Processes

Apache Spark

Problem-Solving

Tools

Microsoft SSIS

Microsoft SSRS

Microsoft SSAS

Power BI

GitLab

Job description

We are seeking a highly skilled Lead Data Engineer with a strong background in Databricks and AWS to spearhead the development and establishment of cloud Data Platform teams. This role requires not only technical expertise but also provides the chance to lead strategic initiatives that will influence the future of the company's data infrastructure.

Our client is seeking a highly skilled Lead Data Engineer with a strong background in Databricks and AWS. This role offers an exciting opportunity to establish and build cloud Data Platform teams, working with Big Data / Data Lake technologies in both cloud and on-premise settings. The successful candidate will have deep knowledge of data warehousing methodologies, and the ability to enhance data quality and documentation processes.

Responsibilities:

• Lead the design and implementation of data pipelines, ETL processes and data models on AWS Databricks.
• Ensure the adoption of best practices in data engineering, including data quality, optimisation and security.
• Conduct code reviews, design reviews and ensure the team follows best practices in version control, CI / CD and DevSecOps for data engineering.
• Collaborate with cross-functional teams to ensure seamless integration of data solutions.
• Communicate technical concepts to non-technical stakeholders, providing insights and recommendations to drive data-driven decision-making.
• Ensure data governance, security and quality standards are maintained throughout the data lifecycle.
• Monitor and support data management initiatives.

What you bring:

• 6+ years of experience in data engineering, with a strong focus on AWS, Databricks.
• Strong understanding of data architecture, data warehousing, data modelling, ETL processes and cloud-based technologies.
• Proven experience with Databricks, particularly in migration and integration projects.
• Solid experience working with Microsoft stack technologies (SSIS, SSRS, SSAS, Power BI).
• Expertise in Apache Spark (or similar) and scalable data processing.
• Experience with CI / CD practices and tools for data pipeline automation ideally using GitLab or similar.
• Excellent problem-solving skills.
• Strong background with AWS and cloud-native data architectures.
• Strong understanding of data governance, security, and compliance.

What sets this company apart:

Our client is a leading player in the technology industry, known for their commitment to innovation and excellence. They offer a supportive and inclusive work environment where every team member is valued for their unique contributions. This is an exciting opportunity to join a company that values collaboration and is committed to providing career development opportunities for its employees.

Aboriginal and Torres Strait Islander Peoples are encouraged to apply. To apply please click apply or call Anthony Buccat for a confidential discussion.

Contract Type: FULL_TIME

Specialism: Information Technology

Salary: AUD190,000 - AUD200,000 per annum