AWS Data Engineer Lead

Softview Infotech; Chennai District; On-site; INR 3,000,000 - 5,000,000; Full time

Job summary

Softview Infotech is seeking an experienced AWS Data Engineer Lead to drive the design and delivery of scalable data engineering solutions on AWS from Chennai/Bangalore. You will lead a team building high-performance data pipelines, lakehouse architectures, and analytics platforms, while championing engineering excellence and cloud-native practices.

The role requires 8–10 years of experience in data engineering with strong expertise in AWS Glue, EMR, Redshift, S3 and real-time streaming.

Qualifications

• Strong hands-on experience with AWS Glue, EMR, Redshift, S3, Lambda, Athena
• Proficiency in PySpark, SQL, and Python for large-scale ETL/ELT development
• Solid understanding of data lake / lakehouse design (Delta/Iceberg/Hudi a plus)
• Experience with data modeling and building scalable data warehouses
• Hands-on experience with CI/CD (CodePipeline/Jenkins) and Infrastructure-as-Code (Terraform/CloudFormation)
• Strong knowledge of AWS security and governance (IAM, Lake Formation, KMS)
• Proven experience leading engineering teams in an Agile delivery environment
• Excellent problem-solving, communication, and stakeholder management skills

Responsibilities

• Lead architecture and development of scalable data pipelines using AWS Glue, EMR, Lambda, and Step Functions
• Design and manage cloud-native data lakes using S3, Lake Formation, and Redshift
• Define engineering standards, review architecture decisions, and drive best practices across the team
• Build real-time and batch data processing solutions using Kinesis / Kafka, Spark, and Glue Streaming
• Partner with data architects and analytics teams to design efficient data models and pipelines
• Implement strong data governance, cataloging, and security controls (IAM, Lake Formation, KMS)
• Optimize infrastructure for performance, scalability, and cost using AWS best practices
• Set up monitoring, alerting, and observability (CloudWatch, CloudTrail)
• Mentor data engineers, lead code/design reviews, and manage sprint deliverables
• Evaluate and adopt new AWS data services to keep the platform modern and efficient

Skills

AWS Data Engineering

Databricks

PySpark

SQL

Tools

AWS Glue

EMR

Redshift

S3

Lambda

Athena

Kinesis

Kafka

Spark

CodePipeline

Jenkins

Terraform

CloudFormation

IAM

Lake Formation

KMS

QuickSight

Delta

Iceberg

Hudi

Job description

Key Skills: AWS Data Engineering, Databricks, PySpark, SQL

About The Role

Position: AWS Data Engineer Lead

Experience: 8–10 Years

Location: Chennai, Bangalore

Key Skills: AWS Data Engineering, Databricks, PySpark, SQL

About The Role

We are seeking an experienced AWS Data Engineer Lead to drive the design and delivery of robust, scalable data engineering solutions on AWS. You'll lead a team building high-performance data pipelines, lakehouse architectures, and analytics platforms, while championing engineering excellence and cloud-native best practices.

Key Responsibilities

• Lead architecture and development of scalable data pipelines using AWS Glue, EMR, Lambda, and Step Functions
• Design and manage cloud-native data lakes using S3, Lake Formation, and Redshift
• Define engineering standards, review architecture decisions, and drive best practices across the team
• Build real-time and batch data processing solutions using Kinesis / Kafka, Spark, and Glue Streaming
• Partner with data architects and analytics teams to design efficient data models and pipelines
• Implement strong data governance, cataloging, and security controls (IAM, Lake Formation, KMS)
• Optimize infrastructure for performance, scalability, and cost using AWS best practices
• Set up monitoring, alerting, and observability (CloudWatch, CloudTrail)
• Mentor data engineers, lead code/design reviews, and manage sprint deliverables
• Evaluate and adopt new AWS data services to keep the platform modern and efficient

Required Skills

• Strong hands-on experience with AWS Glue, EMR, Redshift, S3, Lambda, Athena
• Proficiency in PySpark, SQL, and Python for large-scale ETL/ELT development
• Solid understanding of data lake / lakehouse design (Delta/Iceberg/Hudi a plus)
• Experience with data modeling and building scalable data warehouses
• Hands-on experience with CI/CD (CodePipeline/Jenkins) and Infrastructure-as-Code (Terraform/CloudFormation)
• Strong knowledge of AWS security and governance (IAM, Lake Formation, KMS)
• Proven experience leading engineering teams in an Agile delivery environment
• Excellent problem-solving, communication, and stakeholder management skills

Good To Have

• Experience with Kafka/Kinesis for real-time streaming pipelines
• Exposure to QuickSight or other BI/visualization tools
• Familiarity with Apache Iceberg / Hudi for open table formats
• AWS certifications: AWS Certified Data Analytics – Specialty, AWS Certified Solutions Architect

Why Join Us?

• Lead impactful, large-scale data initiatives on a modern AWS stack
• Collaborative, growth-oriented culture with strong technical mentorship
• Attractive compensation, flexible working arrangements, and career advancement opportunities

Skills: emr,aws data engineering,aws glue,pyspark,amazon redshift,databricks,amazon kinesis,aws data engineer,sql