Lead Data Engineer, AI & Data Pipelines Architect

Keka Technologies Private Limited; Greater London; On-site; GBP 90,000 - 140,000; Full time

Job summary

Keka Technologies Private Limited is seeking a Lead Data Engineer to design, build and maintain scalable data pipelines. You will collaborate with data scientists, software engineers and product managers to deliver robust data solutions.

The role requires 7+ years in data engineering, including leadership, and strong expertise in Apache Spark and Azure Data Factory, with hands-on cloud experience across Azure/AWS/GCP.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 7+ years of experience in data engineering, with at least 3 years in a leadership role.
• Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools.
• Experience with cloud platforms such as Azure, AWS, or GCP and data governance.

Responsibilities

• Lead the design, development, and maintenance of scalable data pipelines to support data-driven applications.
• Collaborate with data scientists, software engineers, and product managers to understand data requirements and deliver solutions.
• Architect and implement data ingestion, transformation, and storage using Spark, Azure Data Factory, and related tools.
• Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery.
• Mentor and guide junior data engineers, promoting learning and improvement.
• Stay up-to-date with data engineering and AI trends and apply them to projects.
• Participate in evaluating and selecting new technologies to enhance data capabilities.
• Contribute to AI-powered features and products leveraging machine learning concepts.

Skills

Leadership

Data engineering

Data pipelines

AI/ML concepts

Programming: Python/Scala/Java

Communication

Mentoring

Education

Bachelor's/Master's in CS/Engineering

Tools

Apache Spark

Azure Data Factory

Azure

AWS

GCP

Job description

Keka Technologies Private Limited is seeking a Lead Data Engineer to design, build and maintain scalable data pipelines. You will collaborate with data scientists, software engineers and product managers to deliver robust data solutions.

The role requires 7+ years in data engineering, including leadership, and strong expertise in Apache Spark and Azure Data Factory, with hands-on cloud experience across Azure/AWS/GCP.