Senior Data Engineer – AWS Data Platform & Pipelines

Tata Consultancy Services Limited; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Tata Consultancy Services UK&I is seeking a hands-on Senior Data Engineer to accelerate delivery and modernization of production data platforms on AWS. You will join an established engineering team and design, develop, test, deploy and operate scalable data solutions on AWS.

The role emphasizes building batch and streaming pipelines, using Python, PySpark, SQL and dbt, with a focus on security, monitoring, and end-to-end ownership from design through production support.

Qualifications

• Strong recent experience building and operating production data platforms on AWS.
• Advanced programming and development skills in Python, PySpark and SQL.
• Production experience with Apache Spark.
• Strong experience using dbt for development, testing, documentation and deployment.
• Experience with Kafka or Amazon MSK and event-driven data processing.
• Practical CDC experience using AWS DMS, Debezium or equivalent technology.
• Hands-on knowledge of AWS services (S3, Glue, EMR, Lambda, Step Functions, MWAA, Kinesis, Redshift, Lake Formation, CloudWatch).
• Strong understanding of dimensional modelling, lakehouse patterns, partitioning, schema evolution and data contracts.
• CI/CD and infrastructure-as-code using Terraform, AWS CDK or CloudFormation.
• Experience implementing automated unit, integration, data-quality and reconciliation testing.
• Experience operating production services, including incident investigation, recovery and performance optimization.
• Ability to write production code and take end-to-end ownership of deliverables.
• Ability to work effectively within an established engineering team.
• Experience constructively challenging designs and proposing practical alternatives.
• Clear professional communication skills with engineers, architects and stakeholders.
• Experience within Banking, Financial Services and Insurance.
• Additional depth in AWS-native data platform and lakehouse architecture.
• Experience optimizing Spark performance and cost.
• Knowledge of Kafka or MSK architecture and operations.
• Understanding of event schemas, ordering, replay and idempotency.
• Experience with real-time monitoring, reconciliation and failure recovery.
• Advanced dbt implementation experience.
• Knowledge of data contracts, semantic modelling, data-quality frameworks, lineage and observability.

Responsibilities

• Build and operate batch, streaming and change-data-capture pipelines on AWS.
• Develop production-quality data solutions using Python, PySpark, SQL and dbt.
• Design reusable ingestion patterns for databases, APIs, files and event streams.
• Implement and operate Kafka-based streaming and CDC solutions.
• Create maintainable data models for analytical and operational use cases.
• Modernize legacy ETL workloads and migrate them to AWS-native services.
• Implement automated testing, data-quality controls and reconciliation.
• Establish monitoring, alerting, lineage and operational runbooks.
• Diagnose pipeline failures, performance issues and data inconsistencies.
• Apply security-by-design principles when handling personal and financial data.
• Use infrastructure as code and CI/CD to enable repeatable deployments.
• Take ownership from design through production support.
• Coach engineers through pairing, code reviews and practical delivery.
• Create reusable components, tests and clear technical documentation.
• Communicate delivery risks early and support technical decisions with evidence.

Skills

AWS

Python

PySpark

SQL

dbt

Kafka

Terraform

CI/CD

Security by design

Tools

dbt

Apache Spark

Kafka

Terraform

AWS CDK

Job description

Tata Consultancy Services UK&I is seeking a hands-on Senior Data Engineer to accelerate delivery and modernization of production data platforms on AWS. You will join an established engineering team and design, develop, test, deploy and operate scalable data solutions on AWS.

The role emphasizes building batch and streaming pipelines, using Python, PySpark, SQL and dbt, with a focus on security, monitoring, and end-to-end ownership from design through production support.