Senior Big Data Engineer (AWS | PySpark | EMR)

ITC Infotech; Bengaluru; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

ITC Infotech is seeking a Senior Big Data Engineer to lead the design and optimization of large-scale data platforms on AWS. The role focuses on PySpark/Scala development, Spark performance tuning, and end-to-end ETL/ELT pipelines within enterprise-scale environments.

The ideal candidate has 8+ years in Data Engineering, strong experience with Spark clusters, and hands-on work with AWS services such as EMR, S3, Glue, and Redshift. Collaboration with analytics teams is essential.

Qualifications

• 8+ years of Data Engineering / Big Data experience
• Experience working with large Spark clusters
• Caching
• Memory tuning
• Experience solving distributed computing challenges

Responsibilities

• Build and maintain scalable Big Data platforms on AWS
• Develop and optimize PySpark/Scala applications
• Create and maintain Airflow DAGs and orchestration workflows
• Optimize Spark jobs and EMR cluster performance
• Implement monitoring, alerting, and reliability solutions
• Work closely with business and analytics teams to deliver data solutions
• Troubleshoot production issues and improve platform stability

Tools

AWS

PySpark

Scala

Hadoop Ecosystem

Hive

HDFS

Spark SQL

Data Lake / Data Warehouse

Airflow

EMR

S3

Glue

Redshift

Lambda

Step Functions

CloudWatch

Databricks

Snowflake

Terraform

Python

SQL

Job description

Job Description

We are looking for a highly skilled Senior Big Data Engineer with strong hands-on experience in AWS, PySpark/Scala, Apache Spark, EMR, and enterprise-scale data processing.

Candidates should have proven experience building and optimizing large-scale distributed data platforms and must be comfortable working with complex ETL ecosystems handling massive data volumes.

Mandatory Skills

AWS (Must Have)

• Strong hands-on experience with AWS ecosystem
• Expert knowledge of:
• AWS EMR
• S3
• Glue
• Redshift
• Lambda
• Step Functions
• CloudWatch
• Experience designing and managing large-scale AWS data platforms

Big Data Technologies (Must Have)

• PySpark and/or Scala
• Hadoop Ecosystem
• Hive
• HDFS
• Spark SQL
• Distributed Data Processing
• Design and development of scalable ETL/ELT pipelines
• Batch and large-scale data processing
• Data Lake and Data Warehouse solutions
• Performance tuning of Spark applications
• Data quality and validation frameworks

Orchestration (Must Have)

• AWS Step Functions
• Workflow orchestration
• DAG design and optimization
• Job scheduling and dependency management

Required Experience

• 8+ years of Data Engineering / Big Data experience
• Experience working with large Spark clusters
• Caching
• Memory tuning
• Experience solving distributed computing challenges

Preferred Skills

• Kafka
• Spark Streaming
• Databricks
• Snowflake
• Terraform
• Python
• SQL

Responsibilities

• Build and maintain scalable Big Data platforms on AWS
• Develop and optimize PySpark/Scala applications
• Create and maintain Airflow DAGs and orchestration workflows
• Optimize Spark jobs and EMR cluster performance
• Implement monitoring, alerting, and reliability solutions
• Work closely with business and analytics teams to deliver data solutions
• Troubleshoot production issues and improve platform stability