Director: Lead Data Engineer

Latentbridge

Greater London

On-site

GBP 90,000 - 140,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Keka Technologies Private Limited is seeking a Lead Data Engineer to design, build and maintain scalable data pipelines. You will collaborate with data scientists, software engineers and product managers to deliver robust data solutions.

The role requires 7+ years in data engineering, including leadership, and strong expertise in Apache Spark and Azure Data Factory, with hands-on cloud experience across Azure/AWS/GCP.

Qualifications

  • Bachelor's or Master's degree in Computer Science, Engineering, or related field.
  • 7+ years of experience in data engineering, with at least 3 years in a leadership role.
  • Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools.
  • Experience with cloud platforms such as Azure, AWS, or GCP and data governance.

Responsibilities

  • Lead the design, development, and maintenance of scalable data pipelines to support data-driven applications.
  • Collaborate with data scientists, software engineers, and product managers to understand data requirements and deliver solutions.
  • Architect and implement data ingestion, transformation, and storage using Spark, Azure Data Factory, and related tools.
  • Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery.
  • Mentor and guide junior data engineers, promoting learning and improvement.
  • Stay up-to-date with data engineering and AI trends and apply them to projects.
  • Participate in evaluating and selecting new technologies to enhance data capabilities.
  • Contribute to AI-powered features and products leveraging machine learning concepts.

Skills

Leadership
Data engineering
Data pipelines
AI/ML concepts
Programming: Python/Scala/Java
Communication
Mentoring

Education

Bachelor's/Master's in CS/Engineering

Tools

Apache Spark
Azure Data Factory
Azure
AWS
GCP

Job description

Section / Function:Insert Section/Function

Travel Required:No

SMCR Status:N/A

Effective from:

Dual Hat Department:Non-employing entity Dual Hat Department

Dual Hat Reports to:Non-employing entity Manager’s Job Rank/Corporate title and functionaltitle

Dual Hat SMCR:Client Dealing Function, Material Risk Takers, Manager of Certified Employees, Significant Management Function, CASS Oversight function, Algorithmic trading or N/A

Profile Summary

We are seeking a highly skilled and experienced Lead Data Engineer to join our talented team. The ideal candidate will have a strong background in data engineering, with experience in AI and a proven track record of designing and implementing robust data pipelines.


Responsibilities


  • Lead the design, development, and maintenance of scalable and efficient data pipelines to support our data-driven applications

  • Collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to understand data requirements and deliver high-quality solutions

  • Architect and implement data ingestion, transformation, and storage processes using tools such as Apache Spark, Azure Data Factory, and other similar technologies

  • Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery

  • Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement

  • Stay up-to-date with the latest trends and best practices in data engineering and AI, and apply them to our projects

  • Participate in the evaluation and selection of new technologies and tools to enhance our data engineering capabilities

  • Contribute to the development of AI-powered features and products, leveraging your knowledge of machine learning and data science concepts


Required Qualifications


  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field

  • 7+ years of experience in data engineering, with at least 3 years in a leadership role

  • Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools

  • Experience with cloud platforms such as Azure, AWS, or GCP

  • Solid understanding of data modeling, data warehousing, and ETL/ELT processes

  • Familiarity with AI and machine learning concepts, with hands\u2011on experience in implementing AI-driven solutions

  • Excellent problem-solving skills and the ability to design and optimize data architectures

  • Strong programming skills in languages such as Python, Scala, or Java

  • Experience with data quality, data governance, and data security best practices

  • Excellent communication and collaboration skills, with the ability to work effectively in a team environment

  • Ability to thrive in a fast-paced, dynamic environment and manage multiple priorities


Preferred Qualifications


  • Advanced degree in Computer Science, Data Science, or a related field

  • Certifications in relevant technologies, such as Azure Data Engineer or Databricks Certified Developer

  • Experience with real-time data processing and streaming technologies like Apache Kafka or Azure Event Hubs

  • Knowledge of data visualization tools, such as Power BI or Tableau

  • Contributions to open-source projects or active participation in the data engineering community

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

ISx4 Group • Belfast City District

On-site
GBP 70,000 - 100,000
Lead Data Engineer
Lead Data Engineer

Xcede • England

Hybrid
GBP 90,000 - 130,000
Flexible hybrid working approach
Opportunity for leadership responsibilities with technical delivery
Influence over next-generation data and AI capabilities
Lead Data Engineer
Lead Data Engineer

Anson McCade Pty • Greater London

On-site
GBP 110,000 - 150,000
Lead Data Engineer
Lead Data Engineer

Lorien • England

On-site
GBP 90,000 - 120,000
Lead Data Engineer
Lead Data Engineer

Scrumconnect Consulting • Newcastle upon Tyne

On-site
GBP 70,000 - 80,000
Data Engineer
Data Engineer

Arrow Global Group • Manchester

On-site
GBP 60,000 - 90,000
Senior Data Engineer
Senior Data Engineer

Welcome to the Jungle • Greater London

Hybrid
GBP 90,000 - 120,000
Senior Data Engineer
Senior Data Engineer

Advantage Resourcing UK Ltd • Greater London

On-site
GBP 90,000 - 120,000
Data Engineer
Data Engineer

Talent • Greater London

Hybrid
GBP 90,000 - 120,000
Data Engineering Lead – AWS & AI - Manchester
Data Engineering Lead – AWS & AI - Manchester

Adria Solutions Ltd • Manchester

Hybrid
GBP 90,000 - 120,000