Senior AWS Data Engineer (Python & Spark)

Tata Consultancy Services; Bengaluru; On-site; INR 3,500,000 - 5,200,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a Senior Data Engineer to design, build, and maintain scalable data platforms on AWS, leveraging Python and Spark to deliver robust ETL pipelines.

You will drive Lakehouse migrations, collaborate with architects and product teams, and ensure data quality, performance, and governance across enterprise data solutions. Strong communication and mentoring skills are essential for success.

Qualifications

• Proven experience as a Senior Data Engineer.
• Strong expertise in Python and Apache Spark.
• Extensive experience building ETL pipelines and data processing frameworks.
• Hands-on experience with AWS services including EMR, Glue, Lambda, Step Functions, API Gateway, Athena.
• Experience working with Data Lakehouse architectures.
• Strong understanding of data modelling and enterprise data platform design.
• Experience handling large-scale data processing workloads.
• Knowledge of data engineering best practices including testing, validation, and performance optimization.
• Strong communication and stakeholder management skills.

Responsibilities

• Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
• Build and optimize high-performance data pipelines using Python and Spark.
• Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
• Implement and maintain Lakehouse architecture solutions.
• Lead migration of existing data processing workloads to Lakehouse platforms using Apache Iceberg.
• Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
• Break down solution designs into epics, stories, and technical deliverables.
• Ensure data quality, reliability, scalability, and performance across data platforms.
• Conduct code reviews, promote engineering best practices, and support QA initiatives.
• Create and maintain technical documentation, architecture diagrams, and implementation standards.
• Support Agile delivery methodologies and contribute to sprint planning and technical estimations.

Skills

Python

Apache Spark

ETL pipelines

Data Lakehouse

Data modelling

Enterprise data design

Large-scale processing

Testing & validation

Performance optimization

Communication

Stakeholder management

Tools

AWS EMR

AWS Glue

AWS Lambda

AWS Step Functions

API Gateway

Amazon Athena

Apache Iceberg

Job description

Key Responsibilities

• Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
• Build and optimize high-performance data pipelines using Python and Spark.
• Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
• Implement and maintain Lakehouse architecture solutions using modern data engineering principles.
• Lead migration of existing data processing workloads to Lakehouse platforms utilizing Apache Iceberg capabilities.
• Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
• Break down solution designs into epics, stories, and technical deliverables.
• Ensure data quality, reliability, scalability, and performance across data platforms.
• Conduct code reviews, promote engineering best practices, and support quality assurance initiatives.
• Create and maintain technical documentation, architecture diagrams, and implementation standards.
• Support Agile delivery methodologies and contribute to sprint planning and technical estimations.

Required Skills

• Proven experience as a Senior Data Engineer.
• Strong expertise in Python and Apache Spark.
• Extensive experience building ETL pipelines and data processing frameworks.
• Hands-on experience with AWS services including:
• AWS EMR
• AWS Glue
• AWS Lambda
• AWS Step Functions
• API Gateway
• Amazon Athena
• Experience working with Data Lakehouse architectures.
• Strong understanding of data modelling and enterprise data platform design.
• Experience handling large-scale data processing workloads.
• Knowledge of data engineering best practices including testing, validation, and performance optimization.
• Strong communication and stakeholder management skills.

Preferred Skills

• Financial Services or Capital Markets experience.
• Experience working with market data and reference data platforms.
• Knowledge of Data Governance and Data Management frameworks.
• Experience with:
• Metadata Management
• Master Data Management (MDM)
• Data Quality frameworks
• AWS Certifications such as:
• AWS Certified Solutions Architect
• AWS Certified Data Analytics

Bonus Skills

• Knowledge of Indices and Index Construction methodologies.
• Asset Management domain experience.
• Exposure to modern open-table formats such as Apache Iceberg.

Desired Candidate Profile

• Strong technical leadership and mentoring capabilities.
• Excellent problem-solving and analytical skills.
• Ability to communicate complex technical concepts to technical and non-technical audiences.
• Experience working in Agile product teams.
• Strong ownership mindset with a focus on delivering scalable and maintainable solutions.
• Passion for modern cloud-based data engineering and platform modernization.