Tech Lead Data Engineer

ELLIOTT MOSS CONSULTING PTE. LTD.; Penarth; On-site; GBP 80,000 - 100,000; Full time

Job summary

A leading data engineering consultancy is seeking a Tech Lead – Data Engineer in Penarth, United Kingdom. This role focuses on driving the design and delivery of a next-generation Air Traffic Management platform. The ideal candidate will have over 7 years of experience in data engineering, with strong capabilities in AWS and Databricks, and will lead a team of engineers while actively contributing to coding and troubleshooting. Competitive salary and opportunities for innovation and mentorship are offered.

Qualifications

• 7+ years of experience in data engineering or complex technical delivery roles.
• 3+ years in a leadership or team lead role.
• Strong hands-on experience with AWS cloud services and Databricks.

Responsibilities

• Lead a team in designing and delivering end-to-end data pipelines.
• Write, review, and optimize code on AWS and Databricks.
• Troubleshoot and resolve complex production issues.

Skills

AWS cloud services (data engineering-focused)

Databricks

Python/Scala

Apache Spark

SQL

Agile/Scrum methodologies

Education

Bachelor’s or Master’s degree in Computer Science or related field

Job description

Job Description

We are seeking a highly skilled and hands-on Tech Lead – Data Engineer to drive the design, delivery, and operational excellence of a next-generation Air Traffic Management (ATM) platform.

This cloud-native platform supports advanced data and analytics workloads.

The Tech Lead will act as a critical bridge between solution architecture, project delivery, and engineering execution.

In addition to leading a team of engineers, the role requires strong hands-on contribution in designing, coding, reviewing, and troubleshooting complex data solutions on AWS and Databricks, ensuring alignment with architectural standards, timelines, and business objectives.

Key Responsibilities

• Technical Leadership & Delivery: Lead a team of data engineers and developers in designing and delivering end-to-end data pipelines (batch, streaming, and event-driven).
• Actively write, review, and optimize code using Python/Scala, Apache Spark, and SQL on Databricks and AWS platforms.
• Translate solution architecture and design blueprints into actionable engineering tasks and contribute directly to implementation.
• Troubleshoot and resolve complex production issues, ensuring system reliability, performance, and uptime.
• Ensure solutions meet stringent performance, security, governance, and compliance requirements within regulated aviation environments.
• Oversee code quality, CI/CD pipelines, observability, and DevOps best practices across the data platform.
• Collaboration & Stakeholder Management: Partner closely with the Solution Architect to validate and refine designs, including building hands-on POCs and prototypes when required.
• Work with the Project Manager to align delivery milestones, manage dependencies, and proactively escalate risks.
• Collaborate with domain experts, data scientists, and external vendors to integrate analytics and AI / GenAI use cases into production systems. Effectively communicate and coordinate with offshore development teams (China) to ensure smooth delivery.
• Innovation & Continuous Improvement: Introduce modern practices in data engineering, MLOps, automation, and cloud-native design to improve platform maturity.
• Proactively identify and address bottlenecks across pipelines, infrastructure, and delivery processes.
• Mentor and guide team members while leading by example through hands-on technical contributions.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Data Science, Information Systems, or a related discipline.
• Experience 7+ years of experience in data engineering, cloud-native platforms, or complex technical delivery roles.
• 3+ years in a technical leadership or team lead role, balancing people leadership with hands-on engineering.
• Strong hands-on experience with AWS cloud services (data engineering-focused) and Databricks (Delta Lake, Spark, Unity Catalog).
• Proven experience delivering solutions in regulated domains such as aviation, finance, or public sector.
• Familiarity with Agile/Scrum methodologies and cross-functional collaboration.
• Experience with AWS Databricks, Delta Lake, live/real-time data streaming, and medallion architecture.
• Exposure to AI / GenAI concepts or intelligent agent-based architectures.
• Familiarity with aviation data standards (e.g., ADS-B, ARINC 424, flight schedules), or strong willingness to learn.
• AWS certifications in Data Analytics, Big Data, or Machine Learning.
• Experience working with high-volume, real-time data processing systems.