Senior Big Data Engineer (AWS | PySpark | EMR)

ITC Infotech; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

ITC Infotech—Bengaluru seeks a Senior Big Data Engineer to design and optimize large-scale data platforms on AWS. The role focuses on PySpark/Scala, Spark, EMR, and end-to-end ETL/ELT pipelines handling massive data volumes.

The candidate will work with Snowflake, Databricks, and data lake/warehouse architectures, ensuring performance tuning, data quality, and reliable data delivery to analytics teams.

Qualifications

• 8+ years of Data Engineering / Big Data experience
• Experience building and optimizing large-scale distributed data platforms
• Comfortable with complex ETL ecosystems handling massive data volumes
• Strong hands-on experience with AWS ecosystem including EMR, S3, Glue, Redshift, Lambda, Step Functions, CloudWatch
• Experience with PySpark and/or Scala and Hadoop ecosystem
• Designing and managing large-scale AWS data platforms and data pipelines

Responsibilities

• Build and maintain scalable Big Data platforms on AWS
• Develop and optimize PySpark/Scala applications
• Create and maintain Airflow DAGs and orchestration workflows
• Optimize Spark jobs and EMR cluster performance
• Implement monitoring, alerting, and reliability solutions
• Work closely with business and analytics teams to deliver data solutions
• Troubleshoot production issues and improve platform stability

Skills

Distributed computing

Data modeling

Performance tuning

Memory optimization

ETL/ELT development

Spark cluster tuning

Workflow orchestration

Tools

AWS

PySpark

Scala

Hadoop

Hive

HDFS

Spark SQL

Airflow

Databricks

Snowflake

Terraform

Python

SQL

Kafka

Spark Streaming

Job description

Job Description

We are looking for a highly skilled Senior Big Data Engineer with strong hands-on experience in AWS, PySpark/Scala, Apache Spark, EMR, and enterprise-scale data processing.

Candidates should have proven experience building and optimizing large-scale distributed data platforms and must be comfortable working with complex ETL ecosystems handling massive data volumes.

Mandatory Skills

AWS (Must Have)

• Strong hands-on experience with AWS ecosystem
• Expert knowledge of:
• AWS EMR
• S3
• Glue
• Redshift
• Lambda
• Step Functions
• CloudWatch
• Experience designing and managing large-scale AWS data platforms

Big Data Technologies (Must Have)

• PySpark and/or Scala
• Hadoop Ecosystem
• Hive
• HDFS
• Spark SQL
• Distributed Data Processing
• Design and development of scalable ETL/ELT pipelines
• Batch and large-scale data processing
• Data Lake and Data Warehouse solutions
• Performance tuning of Spark applications
• Data quality and validation frameworks

Orchestration (Must Have)

• AWS Step Functions
• Workflow orchestration
• DAG design and optimization
• Job scheduling and dependency management

Required Experience

• 8+ years of Data Engineering / Big Data experience
• Experience working with large Spark clusters
• Caching
• Memory tuning
• Experience solving distributed computing challenges

Preferred Skills

• Kafka
• Spark Streaming
• Databricks
• Snowflake
• Terraform
• Python
• SQL

Responsibilities

• Build and maintain scalable Big Data platforms on AWS
• Develop and optimize PySpark/Scala applications
• Create and maintain Airflow DAGs and orchestration workflows
• Optimize Spark jobs and EMR cluster performance
• Implement monitoring, alerting, and reliability solutions
• Work closely with business and analytics teams to deliver data solutions
• Troubleshoot production issues and improve platform stability