Senior/Lead Data Engineer (PySpark, AWS)

EPAM; City Of London; Hybrid; GBP 70,000 - 90,000; Full time

Job summary

A leading technology company in the UK is seeking a Senior Data Engineer to design and implement ETL workflows and data solutions on AWS. This role offers a chance to lead and mentor while working with cutting-edge technologies. Ideal candidates should have strong experience in Python and Spark, as well as hands-on AWS expertise. Benefits include employee stock purchase plans, competitive insurance, and professional development opportunities.

Benefits

Employee Stock Purchase Plan (ESPP)

Private medical insurance

Free Wednesday lunch in-office

Learning and development opportunities

Qualifications

• Significant experience as a Senior Data Engineer designing and implementing robust data solutions.
• Hands-on experience with AWS data services such as EMR, AWS Glue, and others.
• Ability to lead and mentor team members while contributing to technical strategy.

Responsibilities

• Design, develop and maintain scalable ETL workflows and data pipelines.
• Implement data solutions leveraging AWS services.
• Mentor junior engineers and contribute to team development.

Skills

Python

PySpark

ETL workflows

AWS

Data modeling

Mentoring

Education

Bachelor's degree in a relevant field

AWS Certified Solutions Architect

Certified Data Analytics

Tools

AWS Glue

AWS Lambda

Athena

API Gateway

AWS Step Functions

Job description

This job is brought to you by Jobs/Redefined, the UK's leading over-50s age inclusive jobs board.

We are looking for a Senior Data Engineer with expertise in Python and Spark to contribute to building state-of-the-art data platforms on AWS. As part of this role, you'll be integral to designing, implementing and optimizing ETL workflows for our robust Lakehouse architecture in a hybrid and collaborative work environment.

Ideal candidates will have strong technical expertise, a proactive mindset for solving complex challenges and the ability to collaborate effectively in an agile team. This role also provides an opportunity for experienced Data Engineers to step into a leadership role while continuing hands-on work with cutting-edge technologies.

RESPONSIBILITIES

• Design, develop and maintain scalable ETL workflows and data pipelines using Python and Spark on AWS
• Implement data solutions leveraging AWS services such as EMR, AWS Glue, AWS Lambda, Athena, API Gateway and AWS Step Functions
• Collaborate with architects, product owners and team members to break down data engineering solutions into Epics and User Stories
• Lead the migration of existing data workflows to the Lakehouse architecture employing Iceberg capabilities
• Ensure reliability, performance and scalability across complex and high-volume data pipelines
• Create clear and concise documentation for solutions and development processes
• Mentor junior engineers and contribute to team development through knowledge sharing, technical leadership and coaching
• Communicate technical concepts effectively to both technical and business stakeholders

REQUIREMENTS

• Significant experience as a Senior Data Engineer designing and implementing robust data solutions
• Expertise in Python, PySpark and Spark, with a solid focus on ETL workflows and data processing practices
• Hands-on experience with AWS data services such as EMR, AWS Glue, AWS Lambda, Athena, API Gateway and AWS Step Functions
• Demonstrable knowledge of Lakehouse architecture and related data services (e.g., Apache Iceberg)
• Proven experience in data modeling for data platforms and preparing datasets for analytics
• Deep technical understanding of data engineering best practices and AWS data services
• Skilled in decomposing technical solutions into Epics/Stories to streamline development in an agile environment
• Strong background in code reviews, QA practices, testing automation and data validation workflows
• Ability to lead and mentor team members while contributing to technical strategy and execution
• A Bachelors degree in a relevant field or certifications (e.g., AWS Certified Solutions Architect, Certified Data Analytics)

WE OFFER

• EPAM Employee Stock Purchase Plan (ESPP)
• Protection benefits including life assurance, income protection and critical illness cover
• Private medical insurance and dental care
• Employee Assistance Program
• Competitive group pension plan
• Cyclescheme, Techscheme and season ticket loans
• Various perks such as free Wednesday lunch in-office, on-site massages and regular social events
• Learning and development opportunities including in-house training and coaching, professional certifications, over 22,000 courses on LinkedIn Learning Solutions and much more
• If otherwise eligible, participation in the discretionary annual bonus program
• If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program
• *All benefits and perks are subject to certain eligibility requirements