Director: Lead Data Engineer

Keka Technologies Private Limited; Greater London; On-site; GBP 90,000 - 140,000; Full time

Job summary

Keka Technologies Private Limited is seeking a Lead Data Engineer to design, build and maintain scalable data pipelines. You will collaborate with data scientists, software engineers and product managers to deliver robust data solutions.

The role requires 7+ years in data engineering, including leadership, and strong expertise in Apache Spark and Azure Data Factory, with hands-on cloud experience across Azure/AWS/GCP.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 7+ years of experience in data engineering, with at least 3 years in a leadership role.
• Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools.
• Experience with cloud platforms such as Azure, AWS, or GCP and data governance.

Responsibilities

• Lead the design, development, and maintenance of scalable data pipelines to support data-driven applications.
• Collaborate with data scientists, software engineers, and product managers to understand data requirements and deliver solutions.
• Architect and implement data ingestion, transformation, and storage using Spark, Azure Data Factory, and related tools.
• Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery.
• Mentor and guide junior data engineers, promoting learning and improvement.
• Stay up-to-date with data engineering and AI trends and apply them to projects.
• Participate in evaluating and selecting new technologies to enhance data capabilities.
• Contribute to AI-powered features and products leveraging machine learning concepts.

Skills

Leadership

Data engineering

Data pipelines

AI/ML concepts

Programming: Python/Scala/Java

Communication

Mentoring

Education

Bachelor's/Master's in CS/Engineering

Tools

Apache Spark

Azure Data Factory

Azure

AWS

GCP

Job description

Section / Function:Insert Section/Function

Travel Required:No

SMCR Status:N/A

Effective from:

Dual Hat Department:Non-employing entity Dual Hat Department

Dual Hat Reports to:Non-employing entity Manager’s Job Rank/Corporate title and functionaltitle

Dual Hat SMCR:Client Dealing Function, Material Risk Takers, Manager of Certified Employees, Significant Management Function, CASS Oversight function, Algorithmic trading or N/A

Profile Summary

We are seeking a highly skilled and experienced Lead Data Engineer to join our talented team. The ideal candidate will have a strong background in data engineering, with experience in AI and a proven track record of designing and implementing robust data pipelines.

Responsibilities

• Lead the design, development, and maintenance of scalable and efficient data pipelines to support our data-driven applications


• Collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to understand data requirements and deliver high-quality solutions


• Architect and implement data ingestion, transformation, and storage processes using tools such as Apache Spark, Azure Data Factory, and other similar technologies


• Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery


• Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement


• Stay up-to-date with the latest trends and best practices in data engineering and AI, and apply them to our projects


• Participate in the evaluation and selection of new technologies and tools to enhance our data engineering capabilities


• Contribute to the development of AI-powered features and products, leveraging your knowledge of machine learning and data science concepts

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field


• 7+ years of experience in data engineering, with at least 3 years in a leadership role


• Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools


• Experience with cloud platforms such as Azure, AWS, or GCP


• Solid understanding of data modeling, data warehousing, and ETL/ELT processes


• Familiarity with AI and machine learning concepts, with hands\u2011on experience in implementing AI-driven solutions


• Excellent problem-solving skills and the ability to design and optimize data architectures


• Strong programming skills in languages such as Python, Scala, or Java


• Experience with data quality, data governance, and data security best practices


• Excellent communication and collaboration skills, with the ability to work effectively in a team environment


• Ability to thrive in a fast-paced, dynamic environment and manage multiple priorities

Preferred Qualifications

• Advanced degree in Computer Science, Data Science, or a related field


• Certifications in relevant technologies, such as Azure Data Engineer or Databricks Certified Developer


• Experience with real-time data processing and streaming technologies like Apache Kafka or Azure Event Hubs


• Knowledge of data visualization tools, such as Power BI or Tableau


• Contributions to open-source projects or active participation in the data engineering community