Senior Data Engineer

Chandigarh News; Chandigarh; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Chandigarh News is seeking a data engineer to design and implement advanced ETL/ELT pipelines on AWS. You will build batch and streaming data flows, optimize PySpark transformations, and collaborate with data architects and analytics teams to enable scalable data access.

The role requires deep experience with AWS services (Glue, Lambda, Kinesis, SQS, DynamoDB) and strong SQL/PySpark skills for analytics. A/CDP exposure is a plus, and you will participate in agile ceremonies and engineering

Qualifications

• Strong hands-on experience with AWS, including Glue, Lambda, Kinesis, SQS, DynamoDB or NoSQL, and Athena/Redshift.
• Proficient in Python and PySpark for batch and streaming data processing.
• Skilled in writing and optimizing complex SQL queries and PySpark transformations.
• Experience building scalable ETL/ELT pipelines for large-scale data platforms.

Responsibilities

• Develop ETL/ELT workflows to integrate data from multiple enterprise sources.
• Build batch and streaming pipelines for large-scale data processing on cloud platforms like AWS.
• Write and optimize complex SQL and PySpark transformations for analytics and reporting.

Skills

AWS

Python

PySpark

SQL

Data Engineering

REST APIs

Kafka

AWS Glue

AWS Lambda

Kinesis Data Streams

DynamoDB

Athena/Redshift

Tools

Kafka

Adobe Experience Platform

NoSQL Databases

REST APIs

Job description

Key Responsibilities

• Develop ETL/ELT workflows to integrate data from multiple enterprise sources.
• Build batch and streaming pipelines for large-scale data processing, preferable using cloud platforms such as AWS
• Write and optimize complex SQL and PySpark transformations for analytics and reporting.
• Ensure pipelines are performant, scalable, and reliable.
• Troubleshoot and resolve issues related to data ingestion, processing, and platform integrations.
• Support data model implementation and pipeline configuration migrations across environments.
• Collaborate with data architects, analytics teams, and platform engineers to enable efficient data access.
• Participate in agile ceremonies and technical discussions while contributing to engineering best practices.

Required Skills

• Strong hands-on experience with AWS (mandatory) Key services include: AWS Glue AWS Lambda Amazon Kinesis Data Streams Amazon SQS DynamoDB or other NoSQL databases Athena / Redshift (preferred) Programming & Data Processing Python PySpark (batch and streaming) SQL Data Engineering
• Experience building scalable data pipelines and ETL/ELT frameworks
• Experience working with large-scale structured and semi-structured datasets
• Ability to write and optimize complex SQL queries and PySpark transformations
• Experience integrating applications using REST APIs. Good to Have
• Exposure to Adobe Experience Platform (AEP) or other Customer Data Platforms.
• Experience with streaming frameworks such as Kafka. Understanding of data modeling, data governance, and large-scale analytics systems.
• Experience 5 years of professional experience in data engineering 4+ years of experience building PySpark-based data pipelines (batch and streaming) 4+ years of experience with AWS data engineering services