Senior Data Engineer

Tata Consultancy Services; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Tata Consultancy Services UK&I is seeking a hands-on Senior Data Engineer to accelerate delivery and modernization of production data platforms on AWS. You will join an established engineering team and design, develop, test, deploy and operate scalable data solutions on AWS.

The role emphasizes building batch and streaming pipelines, using Python, PySpark, SQL and dbt, with a focus on security, monitoring, and end-to-end ownership from design through production support.

Qualifications

• Strong recent experience building and operating production data platforms on AWS.
• Advanced programming and development skills in Python, PySpark and SQL.
• Production experience with Apache Spark.
• Strong experience using dbt for development, testing, documentation and deployment.
• Experience with Kafka or Amazon MSK and event-driven data processing.
• Practical CDC experience using AWS DMS, Debezium or equivalent technology.
• Hands-on knowledge of AWS services (S3, Glue, EMR, Lambda, Step Functions, MWAA, Kinesis, Redshift, Lake Formation, CloudWatch).
• Strong understanding of dimensional modelling, lakehouse patterns, partitioning, schema evolution and data contracts.
• CI/CD and infrastructure-as-code using Terraform, AWS CDK or CloudFormation.
• Experience implementing automated unit, integration, data-quality and reconciliation testing.
• Experience operating production services, including incident investigation, recovery and performance optimization.
• Ability to write production code and take end-to-end ownership of deliverables.
• Ability to work effectively within an established engineering team.
• Experience constructively challenging designs and proposing practical alternatives.
• Clear professional communication skills with engineers, architects and stakeholders.
• Experience within Banking, Financial Services and Insurance.
• Additional depth in AWS-native data platform and lakehouse architecture.
• Experience optimizing Spark performance and cost.
• Knowledge of Kafka or MSK architecture and operations.
• Understanding of event schemas, ordering, replay and idempotency.
• Experience with real-time monitoring, reconciliation and failure recovery.
• Advanced dbt implementation experience.
• Knowledge of data contracts, semantic modelling, data-quality frameworks, lineage and observability.

Responsibilities

• Build and operate batch, streaming and change-data-capture pipelines on AWS.
• Develop production-quality data solutions using Python, PySpark, SQL and dbt.
• Design reusable ingestion patterns for databases, APIs, files and event streams.
• Implement and operate Kafka-based streaming and CDC solutions.
• Create maintainable data models for analytical and operational use cases.
• Modernize legacy ETL workloads and migrate them to AWS-native services.
• Implement automated testing, data-quality controls and reconciliation.
• Establish monitoring, alerting, lineage and operational runbooks.
• Diagnose pipeline failures, performance issues and data inconsistencies.
• Apply security-by-design principles when handling personal and financial data.
• Use infrastructure as code and CI/CD to enable repeatable deployments.
• Take ownership from design through production support.
• Coach engineers through pairing, code reviews and practical delivery.
• Create reusable components, tests and clear technical documentation.
• Communicate delivery risks early and support technical decisions with evidence.

Skills

AWS

Python

PySpark

SQL

dbt

Kafka

Terraform

CI/CD

Security by design

Tools

dbt

Apache Spark

Kafka

Terraform

AWS CDK

Job description

Job Type: Permanent

Location: London, UK

Working Hours: 40 hours per week (Full-time)

Careers at TCS: It means more

TCS is a purpose-led transformation company, built on belief. We do not just help businesses to transform through technology. We support them in making a meaningful difference to the people and communities they serve - our clients include some of the biggest brands in the UK and worldwide. For you, it means more to make an impact that matters, through challenging projects which demand ambitious innovation and thought leadership.

The Role

We are looking for a hands-on Senior Data Engineer to accelerate the delivery and modernization of production data platforms. You will join an established engineering team and design, develop, test, deploy and operate scalable data solutions on AWS.

Your responsibilities:

• Build and operate batch, streaming and change-data-capture pipelines on AWS
• Develop production-quality data solutions using Python, PySpark, SQL and dbt
• Design reusable ingestion patterns for databases, APIs, files and event streams
• Implement and operate Kafka-based streaming and CDC solutions
• Create maintainable data models for analytical and operational use cases
• Modernize legacy ETL workloads and migrate them to AWS-native services
• Implement automated testing, data-quality controls and reconciliation
• Establish monitoring, alerting, lineage and operational runbooks
• Diagnose pipeline failures, performance issues and data inconsistencies
• Apply security-by-design principles when handling personal and financial data
• Use infrastructure as code and CI/CD to enable repeatable deployments
• Take ownership from design through production support
• Coach engineers through pairing, code reviews and practical delivery
• Create reusable components, tests and clear technical documentation
• Communicate delivery risks early and support technical decisions with evidence

Your Profile

Essential skills/knowledge/experience:

• Strong recent experience building and operating production data platforms on AWS
• Advanced programming and development skills in Python, PySpark and SQL
• Production experience with Apache Spark
• Strong experience using dbt for development, testing, documentation and deployment
• Experience with Kafka or Amazon MSK and event-driven data processing
• Practical CDC experience using AWS DMS, Debezium or equivalent technology
• Hands-on knowledge of relevant AWS services, including Amazon S3, AWS Glue, Amazon EMR, AWS Lambda, AWS Step Functions, Amazon MWAA or Apache Airflow, Amazon Kinesis, Amazon MSK, Amazon Redshift, AWS Lake Formation and Amazon CloudWatch
• Strong understanding of dimensional modelling, lakehouse patterns, partitioning, schema evolution and data contracts
• CI/CD and infrastructure-as-code experience using Terraform, AWS CDK or AWS CloudFormation
• Experience implementing automated unit, integration, data-quality and reconciliation testing
• Experience operating production services, including incident investigation, recovery and performance optimization
• Ability to write production code and take end-to-end ownership of deliverables
• Ability to work effectively within an established engineering team
• Experience constructively challenging designs and proposing practical alternatives
• Clear professional communication skills when working with engineers, architects and business stakeholders
• Experience within Banking, Financial Services and Insurance
• Additional depth in AWS-native data platform and lakehouse architecture
• Experience optimizing Apache Spark performance and cost
• Strong knowledge of Kafka or Amazon MSK architecture and operations
• Understanding of event schemas, ordering, replay and idempotency
• Experience with real-time monitoring, reconciliation and failure recovery
• Advanced dbt implementation experience
• Knowledge of data contracts, semantic modelling, data-quality frameworks, lineage and observability

TCS is consistently voted as a Top Employer in the UK and globally. Our competitive salary packages feature pension, health care, life assurance, laptop, phone, access to extensive training resources and discounts within the larger Tata network.

We offer health & wellness initiatives and sports events; we are the proud sponsor of the London Marathon.

Diversity, Inclusion and Wellbeing

Tata Consultancy Services UK&I is committed to meeting the accessibility needs of all individuals in accordance with the UK Equality Act 2010 and the UK Human Rights Act 1998.

We welcome and embrace diversity in race, nationality, ethnicity, disability, neurodiversity, gender identity, age, physical ability, gender reassignment, sexual orientation. We are a disability inclusive employer and encourage disabled people to apply for this role.

As a Disability Confident Employer, we offer an interview to applicants with disabilities or long-term conditions who meet the minimum criteria for the role. Please email us at [e-mail] if you would like to opt in.

If you are an applicant who needs any adjustments to the application process or interview, please contact us at [e-mail] with the subject line: “Adjustment Request” or call TCS London Office 02031552100 / [phone] to request an adjustment. We welcome requests prior to you completing the application and at any stage of the recruitment process.