Snr Data Engineer

Alight Solutions; Gurugram District; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Alight Solutions is seeking a Senior Software Engineer (ETL) with 4–8 years of ETL experience to design and deliver scalable data pipelines using Spark and the AWS data stack. You will build end-to-end serverless architectures with Glue, Lambda, S3 and Redshift, orchestrating workflows with Airflow, Step Functions, or Control‑M and ensuring data governance across hybrid data lake environments.

You will implement reusable ingestion frameworks, optimize performance, collaborate with stakeholders,

Qualifications

• 4 to 8 years ETL experience total.

Responsibilities

• Build and maintain high volume ETL/ELT pipelines across Hadoop and AWS.
• Develop distributed data processing with PySpark and Spark SQL.
• Design reusable data ingestion frameworks and orchestration processes.
• Optimize data workflows with partitioning, bucketing, compression, Parquet/ORC.
• Understand hybrid data lake architectures using S3 + HDFS; ensure data governance.
• Deliver complex Agile projects; drive architecture decisions for scalability.
• Define technical specifications and secure code guidelines; implement tests and reviews.
• Troubleshoot issues and optimize Spark performance, jobs, and clusters.
• Understand data flow diagrams and data lineage; design end-to-end data flows.
• Use Airflow, Control‑M, Step Functions for job orchestration; monitor BAU.

Skills

PySpark

Scala

Spark optimization

Hadoop ecosystem

Python

Shell scripting

CI/CD

GitHub

Git commands

Airflow

Control-M

Docker

ECS/EKS

Airflow/Control-M

Security/compliance (Cloud)

Data modelling

Tools

AWS data stack

S3

Glue

EMR

Lambda

Kinesis

Redshift

Step Functions

Spark

HiveQL

Hadoop

Airflow

Control‑M

Job description

JD – Snr Software Engineer (ETL)

Experience & Expectations :

• Leverage extensive experience (4 to 8 years overall ETL experience, to assist in solution design and delivery along with build of new ETLs).
• We are seeking an experienced ETL Developer with strong expertise in Big Data (Spark, Cloudera).
• Experience orchestrating workflows using AWS Step Functions (state machines) for reliable and scalable data pipelines.
• Ability to implement end-to-end serverless data architectures integrating Glue, Lambda, S3, and Redshift

Core Responsibilities :

• Build and maintain high volume ETL/ELT pipelines across Hadoop (HDFS, Hive, Spark, Kafka) and AWS (Glue, EMR, Lambda, Step Functions, Redshift).
• Develop distributed data processing solutions using PySpark, Spark SQL, and scalable cloud serverless patterns.
• Implement reusable data ingestion frameworks for batch, ability to design & implement Orchestration process and Leverage AI
• Optimize data workflows using partitioning, bucketing, compression, file formats (Parquet/ORC).
• Understanding hybrid data lake architectures using S3 + HDFS, ensuring data governance and best practices are adheres
• Experience to deliver complex projects in an Agile environment
• Assist in Design and build the robust, scalable and secure software solutions across the having no/least adoption
• Define clear technical specifications and make architecture decisions that align with business goals and long-term scalability.
• Implement best practices (including secure code guidelines) through the implementation of unit tests, automation, leverage and code reviews. Drive continuous improvement in code quality and maintainability.
• Troubleshooting issues and proactively solving problems as they arise, ensuring the smooth operation of full stack applications
• Ability to understand the data flow diagram, data modelling and Lineages
• Job orchestration using Airflow, Control M, Step Functions, or event-driven triggers.
• Ensure data is protected and compliant with regulatory standards.
• Work closely with business stakeholders to enable high quality datasets.
• Work on best practice adoption and provide guidance to peers/juniors in team.
• Ability to respond on incidents, and troubleshooting Spark performance issues, job failures, and cluster bottlenecks.Collaborate closely with team members, QA and cross product teams to streamline release processes.
• Collaborate with business stakeholders to gather, analyse, and translate data into technical solutions

Technical Skills :

• Strong experience with the AWS data stack (S3, Glue, EMR, Lambda, Kinesis, Redshift, Step Functions etc.,).
• Strong hands‑on expertise in Scala, PySpark , Spark optimization techniques, HiveQL, and distributed computing.
• Good understanding of Hadoop ecosystem (HDFS, Hive, Spark, YARN, Kafka).
• Good work experience in SQL in hive and impala
• Proficiency in at least one scripting/programming language: Python, Shell scripting.
• Strong experience with CI/CD , GitHub, Git commands.
• Expertise = … etc.
• Good understanding of data modelling (star/snowflake), partitioning strategies, and schema evolution.
• Expertise in data profiling and decision making.
• Able to understand, design and create data flow diagrams.
• Able to understand the architecture and design end-to-end data flow.
• Hands‑on experience with Airflow, Control‑M , or other orchestrators.
• To monitor and support BAU and year end activities, if needed.
• Exposure to security and compliance aspects in Cloud.
• Familiarity with serverless patterns and containerization (Docker, ECS/EKS).

Other Requirements

• Strong logical and analytical, problem-solving, and communication skills.
• Communicate effectively and concisely with multiple stakeholders and coordinate and collaborate with cross functional teams.
• AWS certifications (Data Engineer, or Developer) are a plus.

Detail-Oriented and proactive in problem-solving and issue resolution

We offer you a competitive total rewards package, continuing education & training, and tremendous potential with a growing worldwide organization.

DISCLAIMER:

Nothing in this job description restricts management’s right to assign or reassign duties and responsibilities of this job to other entities; including but not limited to subsidiaries, partners, or purchasers of Alight business units.