Senior Data Engineer

Cognizant; City Of London; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A global technology company is seeking a Senior Data Engineer to join their team in London. Ideal candidates will have expertise in Python and AWS technologies, focusing on building optimal data pipelines. The position allows for hybrid working, requiring 2-3 days on-site. Responsibilities include collaborating with teams to deliver analytical solutions and managing project expectations. This role requires SC-level security clearance or eligibility for it.

Qualifications

• Must be SC cleared, or eligible to obtain SC-level security clearance.
• Experience in building, testing and maintaining optimal AWS data pipeline architecture.
• Experience in dual shore engagement across multiple time zones.

Responsibilities

• Responsible for building AWS Glue, step functions, and maintaining data pipelines.
• Work collaboratively with developers and data scientists on data projects.
• Define and document best practices for application deployment.

Skills

Python programming

AWS Glue

PySpark

Spark SQL

AWS Lambda

Tools

AWS

CICD

Job description

Senior Data Engineer (Python/AWS)

Full-time | London, UK

Hybrid working: 2-3 days per week on-site

Must be SC cleared, or eligible to obtain SC-level security clearance

Job description

We are looking for a developer with expertise in Python, AWS Glue, step function, EMR cluster and redshift, to join our AWS Development Team. You will be serving as the functional and domain expert in the project team to ensure client expectations are met.

The role involves structuring analytical solutions that address business objectives and problem solving. We are looking for hands-on experience in writing code for AWS Glue in Python, PySpark, and Spark SQL.

The successful candidate will translate stated or implied client needs into researchable hypotheses, facilitate client working sessions, and be involved in recurring project status meetings. You will develop long-lasting, trusted advisor relationships with clients, bringing an ability to work in dual shore engagement across multiple time zones and manage business uncertainty, day-to-day project operations.

An in-person interview will be required as part of the interview process.

Key responsibilities

• Hands-on experience in Python programming is essential
• Responsible for Build, Test and Maintain optimal AWS data pipeline architecture. Building AWS glue, step function, AWS Lambda functions and unit test
• Work with fellow developers, data architects, data analysts and data scientists on data initiatives
• Expected to understand the core AWS services and apply best practices regarding security and scalability
• Understand the current application infrastructure, suggesting changes to it
• Define and document best practices and strategies regarding application deployment and infrastructure maintenance
• Define service capacity planning strategies
• Implement the applications CICD pipeline using the AWS CICD stack
• Project tasks deliverables and management
• Deliver on project progress and ensure adherence to client expectation
• Communications, including deck writing
• Delivery of output
• Gathering of client feedback on problem structuring
• Understand and define business problems, get all background information and collect relevant data points
• Create solution hypothesis and get client buy in, discuss and align on end objective, staffing need, timelines and budget

Nice to have

• Hive
• Pig
• No-SQL database