Lead Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 6,000,000 - 12,000,000; Full time

Job summary

Tata Consultancy Services in Bengaluru, Karnataka, seeks a Principal AWS Data Engineering Architect to lead the design of enterprise-scale data platforms on AWS, build automation frameworks, and drive data governance and engineering excellence.

The role blends hands-on architecture, cloud modernization, and leadership to mentor senior engineers, define strategy, and partner with executives to deliver scalable data solutions across the organization.

Qualifications

• 8–15 years of experience in Data Engineering and Data Architecture.
• Extensive AWS Data Engineering experience.
• Expert-level Python development and framework engineering.
• Strong PySpark and Apache Spark expertise.
• Strong SQL and Data Modeling experience.
• Experience designing enterprise-scale Data Warehouses and Data Lakes.
• Strong AWS expertise across: Glue, EMR, Redshift, Athena, Lambda, Kinesis, DynamoDB, SageMaker, EventBridge, DMS, S3, ECS/EKS, CloudWatch.
• Infrastructure as Code using: Terraform, CloudFormation, CDK.
• Experience with Airflow, DBT, Jenkins, Artifactory, Docker, Kubernetes.
• Strong DevSecOps and CI/CD experience.
• Expertise in Data Governance and Metadata Management.
• Linux/Unix administration experience.
• Experience with large-scale distributed systems.

Responsibilities

• Design and architect large-scale enterprise data platforms and cloud-native data ecosystems.
• Build scalable, resilient, and high-performance data pipelines on AWS.
• Design modern Data Lakes, Data Warehouses, Data Assets, and Data Products.
• Drive enterprise Data Engineering standards and best practices.
• Own end-to-end data engineering lifecycle from requirements gathering through support and operations.
• Lead cloud modernization and automation initiatives.
• Establish scalable cloud architecture patterns and reusable engineering frameworks.
• Design and develop advanced Python-based automation and engineering frameworks.
• Build reusable components for data ingestion, transformation, orchestration, monitoring, testing, governance.
• Automate operational and engineering processes across platforms.
• Drive framework development using SDKs, CDKs, APIs, and automation tooling.
• Develop high-performance PySpark and Apache Spark solutions.
• Optimize big data applications for throughput, scalability, and cost efficiency.
• Work with data formats including Parquet, Avro, ORC, CSV, JSON, XML.
• Implement data modeling, warehousing, and analytics architectures.
• Define technical strategy and roadmap for enterprise data engineering platforms.
• Lead PoCs, innovation initiatives, and modernization programs.
• Mentor senior engineers and architects.
• Drive enterprise architecture decisions and technical governance.
• Partner with executive stakeholders and business leaders.
• Implement enterprise security best practices including encryption at rest and in transit, SSL/TLS, tokenization, data masking, IAM governance.
• Establish metadata management and data governance standards.
• Ensure compliance with enterprise security and regulatory requirements.

Skills

AWS Data Engineering

Python development

PySpark

Apache Spark

SQL

Data Modeling

Data Governance

DevSecOps

CI/CD

Terraform

CloudFormation

CDK

Airflow

DBT

Jenkins

Docker

Kubernetes

Linux

S3

Glue

Tools

Terraform

CloudFormation

CDK

Airflow

DBT

Jenkins

Artifactory

Docker

Kubernetes

Job description

We are seeking a highly experienced Principal AWS Data Engineering Architect with deep expertise in AWS Data Engineering, Python Framework Development, Big Data Processing, Data Architecture, Automation, and Cloud Engineering. The ideal candidate will be a hands‑on technical leader responsible for designing enterprise‑scale data platforms, automation frameworks, and highly scalable cloud‑native data solutions while driving innovation, governance, and engineering excellence.

This role requires a strong blend of Data Architecture, AWS Engineering, DevSecOps, Automation Framework Development, and technical leadership capabilities.

Key Responsibilities

Data Architecture & Engineering

• Design and architect large-scale enterprise data platforms and cloud-native data ecosystems.
• Build scalable, resilient, and high-performance data pipelines on AWS.
• Design modern Data Lakes, Data Warehouses, Data Assets, and Data Products.
• Drive enterprise Data Engineering standards and best practices.
• Own end-to-end data engineering lifecycle from requirements gathering through support and operations.

AWS Cloud Engineering

• Design and implement solutions leveraging AWS services including:
  
  • S3
  • Lambda
  • Glue
  • Athena
  • EMR
  • Redshift
  • DynamoDB
  • Kinesis
  • EventBridge
  • DMS
  • Step Functions
  • SQS
  • SNS
  • ECS
  • EKS
  • Fargate
  • SageMaker
  • CloudWatch
• Lead cloud modernization and automation initiatives.
  
• Establish scalable cloud architecture patterns and reusable engineering frameworks.
  

Automation & Framework Development

• Design and develop advanced Python-based automation and engineering frameworks.
  
• Build reusable components for:
  
  • Data ingestion
  • Data transformation
  • Orchestration
  • Monitoring
  • Testing
  • Governance
• Automate operational and engineering processes across platforms.
  
• Drive framework development using SDKs, CDKs, APIs, and automation tooling.
  

Big Data & Analytics

• Develop high-performance PySpark and Apache Spark solutions.
  
• Optimize big data applications for throughput, scalability, and cost efficiency.
  
• Work with data formats including:
  
  • Parquet
  • Avro
  • ORC
  • CSV
  • JSON
  • XML
• Implement data modeling, warehousing, and analytics architectures.
  

Leadership & Strategy

• Define technical strategy and roadmap for enterprise data engineering platforms.
• Lead Proof of Concepts (PoCs), innovation initiatives, and modernization programs.
• Mentor senior engineers and architects.
• Drive enterprise architecture decisions and technical governance.
• Partner with executive stakeholders and business leaders.

Security & Governance

• Implement enterprise security best practices including:
  
  • Encryption at rest
  • Encryption in transit
  • SSL/TLS
  • Tokenization
  • Data masking
  • IAM governance
• Establish metadata management and data governance standards.
  
• Ensure compliance with enterprise security and regulatory requirements.
  

Required Skills

• 8-15 years of experience in Data Engineering and Data Architecture.
  
• Extensive AWS Data Engineering experience.
  
• Expert-level Python development and framework engineering.
  
• Strong PySpark and Apache Spark expertise.
  
• Strong SQL and Data Modeling experience.
  
• Experience designing enterprise-scale Data Warehouses and Data Lakes.
  
• Strong AWS expertise across:
  
  • Glue
  • EMR
  • Redshift
  • Athena
  • Lambda
  • Kinesis
  • DynamoDB
  • SageMaker
  • EventBridge
  • DMS
  • S3
  • ECS/EKS
  • CloudWatch
• Infrastructure as Code using:
  
  • Terraform
  • CloudFormation
  • CDK
• Experience with:
  
  • Airflow
  • DBT
  • Jenkins
  • Artifactory
  • Docker
  • Kubernetes
• Strong DevSecOps and CI/CD experience.
  
• Expertise in Data Governance and Metadata Management.
  
• Linux/Unix administration experience.
  
• Experience with large-scale distributed systems.
  

Preferred Skills

• Apache Flink.
• AWS MSK (Kafka).
• Apache Iceberg.
• Agentic AI.
• Generative AI.
• Large Language Models (LLMs).
• Data Products Architecture.
• Enterprise Architecture experience.
• Erwin Data Modeling tools.
• Financial Services and Banking domain experience.

Desired Candidate Profile

• Strong technical leadership capabilities.
• Excellent communication and stakeholder management skills.
• Ability to influence enterprise-wide architectural decisions.
• Strong innovation mindset with focus on automation and modernization.
• Experience managing large-scale engineering initiatives.
• Passion for emerging technologies and cloud innovation.