Lead AWS Data Engineer

NTT DATA; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Flexible work options

Learning & Development

Job summary

NTT DATA is seeking a Lead Data Engineer (AWS) in the UK to design and deliver scalable data pipelines and cloud-native platforms. You will collaborate with architects, engineers, and analysts to build reliable data solutions using AWS Glue, Python/PySpark, SQL, and YAML.

The role emphasizes governance, security, and performance, with a focus on delivering modern data architectures and data lakehouse implementations for analytics and AI-driven outcomes.

Qualifications

• Proven data engineering experience in cloud-based platforms.
• Strong skills in AWS Glue, PySpark, Python, and SQL.
• Experience with data lakes and Lakehouse-style architectures.
• Knowledge of CI/CD and DevOps practices.

Responsibilities

• Act as a senior engineer within data engineering and cloud platform initiatives.
• Collaborate with architects to define scalable AWS-based data solutions.
• Lead engineering workstreams and ensure high-quality technical delivery.
• Design, build, and optimise scalable data pipelines on AWS.
• Develop and maintain ETL/ELT pipelines using AWS Glue, Python, PySpark, SQL, YAML.
• Implement robust data ingestion, transformation, and processing patterns.
• Develop automated testing, monitoring, and alerting solutions.
• Drive observability and pipeline resilience.

Skills

AWS Glue

Python

PySpark

SQL

YAML

Data pipelines

Spark

CI/CD

DevOps

Tools

AWS Cloud Services

Data Lakes

Lakehouse concepts

Job description

We are seeking an accomplished and detail-oriented Lead Data Engineer - AWS to join our Data & AI practice. The successful candidate will bring deep expertise in data engineering, distributed data processing, and cloud-native platforms, with a strong focus on AWS-based data ecosystems.

This role is critical in designing, building, and optimising end-to-end data pipelines and platforms, enabling scalable data processing, advanced analytics, and AI-driven solutions. You will play a key role in ensuring data quality, integrity, performance, and reliability, supported by strong engineering and testing practices.

As a senior practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing data solutions, leveraging technologies such as AWS Glue, Python/PySpark, SQL, and configuration-driven frameworks (e.g., YAML). You will thrive in a collaborative, client-facing environment, with a passion for solving complex technical challenges, ensuring delivery excellence, and driving modernisation through cloud-native engineering practices.

What you'll be doing:

Client Engagement & Delivery

• Act as a senior engineer within data engineering and cloud platform initiatives, supporting delivery across complex transformation programmes
• Collaborate with architects and stakeholders to define and implement scalable AWS-based data solutions
• Contribute to solution design, estimation, and delivery planning
• Lead engineering workstreams and ensure high-quality technical delivery
• Design, build, and optimise scalable data pipelines and data processing frameworks on AWS
• Develop and maintain ETL/ELT pipelines using:
• AWS Glue
• Python / PySpark
• SQL
• Configuration-driven frameworks (e.g., YAML)
• Implement robust data ingestion, transformation, and processing patterns
• Build reusable data services, components, and frameworks
• Define and implement testing strategies for data pipelines, ensuring reliability and accuracy
• Python / PySpark transformations
• SQL-based validation logic
• Configuration-driven orchestration
• Develop automated testing, monitoring, and alerting solutions
• Ensure:
• Consistent transformation behaviour
• Drive improvements in observability and pipeline resilience

AWS Data Platforms

• Lead development on AWS services including:
• AWS Glue
• S3-based data lakes
• Supporting services within the AWS data ecosystem
• Support implementation of modern data architectures, including data lakes and Lakehouse-style platforms
• Optimise pipelines and jobs for performance, scalability, and cost efficiency
• Define and implement data transformation logic aligned to business requirements
• Support data modelling approaches for analytics and platform use cases
• Ensure consistency, usability, and quality across data assets and pipelines

Collaboration & Technical Leadership

• Collaborate with:
• Solution Architects
• Analysts and ML engineers
• Provide technical leadership and mentoring to engineers within the team
• Promote engineering best practices, automation, and reusable solutions
• Contribute to engineering standards, documentation, and knowledge sharing

Quality, Governance & Security

• Ensure data quality, integrity, and reliability across data platforms
• Implement and enforce secure coding and data handling practices
• Support compliance with:
• GDPR
• Regulated environment standards (where applicable)
• Contribute to monitoring, auditing, and operational processes

What experience you'll bring:

• Proven experience in data engineering and cloud-based platform delivery
• Strong understanding of distributed data processing and scalable system design
• Ability to lead delivery while remaining hands-on technically
• Strong analytical, problem-solving, and communication skills
• Experience working in client-facing and delivery-focused environments
• Ability to mentor and develop engineering teams

Technical Expertise

• Strong hands-on experience with:
• AWS cloud services, especially AWS Glue
• SQL for querying, transformation, and validation
• Experience building and operating:
• Familiarity with:
• Data lakes and Lakehouse concepts
• Distributed processing frameworks (e.g., Apache Spark)
• Strong understanding of:
• ETL vs ELT patterns
• Performance tuning and optimisation
• Experience with:
• Version control (Git)
• CI/CD and DevOps practices

Who we are:

We are a business with a global reach that empowers local teams, and we undertake hugely exciting work that is genuinely changing the world. Our advanced portfolio of consulting, applications, business process, cloud, and infrastructure services will allow you to achieve great things by working with brilliant colleagues, and clients, on exciting projects.

Our inclusive work environment prioritises mutual respect, accountability, and continuous learning for all our people. This approach fosters collaboration, well-being, growth, and agility, leading to a more diverse, innovative, and competitive organisation. We are also proud to share that we have a range of Inclusion Networks such as: the Women’s Business Network, Cultural and Ethnicity Network, LGBTQ+ & Allies Network, Neurodiversity Network and the Parent Network.

What we can offer you:

We offer a range of tailored benefits that support your physical, emotional, and financial wellbeing. Our Learning and Development team ensure that there are continuous growth and development opportunities for our people. We also offer the opportunity to have flexible work options.

You can find more information about NTT DATA UK & Ireland here: https://uk.nttdata.com/

We are an equal opportunities employer. We believe in the fair treatment of all our employees and commit to promoting equity and diversity in our employment practices.

We are also a proud Disability Confident Committed Employer - we are committed to creating a diverse and inclusive workforce. We actively collaborate with individuals who have disabilities and long-term health conditions which have an effect on their ability to do normal daily activities, ensuring that barriers are eliminated when it comes to employment opportunities. In line with our commitment, we guarantee an interview to applicants who declare to us, during the application process, that they have a disability and meet the minimum requirements for the role.

If you require any reasonable adjustments during the recruitment process, please let us know. Join us in building a truly diverse and empowered team.