Sr Data Engineer

CityMall; Gurugram District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading retail company in Gurugram seeks a Data Engineer to optimize data pipelines and collaborate with stakeholders. The ideal candidate will have expertise in AWS services, experience in data modeling, and a bachelor's or master's degree in a relevant field. This role offers an opportunity to work on complex data infrastructure and contribute significantly to data-driven decision making.

Qualifications

• Minimum 2+ years’ experience as Data Engineer.
• Hands-on experience in big data on AWS environments.
• Expertise in designing complex Data Models.

Responsibilities

• Test and manage data warehouse activity including ETL processes.
• Build and maintain optimized data pipelines.
• Collaborate with stakeholders for business applications.

Skills

Python

SQL

AWS Glue

AWS Redshift

ETL processes

Data modeling

Big data on AWS

Education

Bachelor's or Master’s degree in computer science/Mathematics/statistics

Tools

AWS Athena

Kafka

pySpark

Job description

Roles and Responsibilities

• Tests, monitors, manage and validate data warehouse activity including data extraction, transformation, movement, loading, cleansing, and updating processes
• Building and maintaining optimized and highly available data pipelines that facilitate deeper analysis and reporting
• Collaborate with stakeholders to understand, define and document business applications, data sources/relationships, and needs (metrics, dimensions, charts and dashboards).
• Analyze and Identify new data sources as needed per stakeholder requirements.
>Working closely with analytics team and other business stakeholders to create data processing frameworks that help the business make sense of growing data
• Manage and monitor existing data pipelines and work with other team members to ensure best practices for ETL and data warehousing
• Work on optimization of data storage and processing infrastructure to reduce costs
• Facilitate integration of different BI tools as needed by business stakeholders
• Assist in the design and documentation of data infrastructure and technologies

Skills Required

• Proficient in languages: Python.
• Minimum 2+ years’ experience as Data Engineer
• Experience in AWS Stack - Glue, Athena, Quick sight, RDS, Redshift, Kafka, pySparkExperience setting up data pipelines, archiving data, data lakes
• Bachelors or Master’s degree in computer science/Maths/statistics or related field
• Expertise with designing complex Data Models and Data Engineering solutions
• Write complex SQL for processing raw data, data transformations as well as data validations
• Experience with infrastructure as code using shell scripting
• Hands on experience in working with big data on AWS environments including cleaning/transforming/cataloging/mapping etc.
• Experience in setup of data warehouse using Amazon Redshift, creating Redshift clusters and perform data analysis queries
• Experience in ETL and data modeling on AWS ecosystem components - AWS Glue, Redshift, DynamoDB
• Familiarity with AWS data migration tools such as AWS DMS, Amazon EMR, and AWS Data Pipeline
• Certifications on AWS will be an added plus.