AWS Data Engineer

Durapid Technologies Pvt Ltd; Hyderabad; Presencial; INR 900.000 - 1.350.000; Jornada completa

Descripción de la vacante

Durapid Technologies Pvt Ltd is seeking skilled Data Engineers with hands-on experience in AWS Data Engineering, distributed data processing, and cloud-based data solutions. The role involves building scalable data pipelines, working with Redshift, Spark, PySpark, Kafka, Airflow, Python, and SQL to design and optimize ETL/ELT workflows and data warehouses.

The ideal candidate will develop, maintain, and optimize real-time ingestion solutions, ensure data quality and performance, and collaborate

Formación

• 3–9 years of experience in data engineering.
• Strong hands-on AWS data engineering experience.
• Experience building pipelines with Amazon Redshift.
• Proficiency in Apache Spark and PySpark.
• Strong Python and SQL programming skills.
• Hands-on experience with Apache Airflow and Kafka.
• Understanding of ETL/ELT, Data Warehousing, Data Marts, and analytics models.
• Ability to build scalable, production-grade data pipelines.

Responsabilidades

• Design, develop, and maintain scalable data pipelines using AWS services, particularly Amazon Redshift.
• Build and optimize ETL/ELT workflows for data ingestion, transformation, and processing.
• Develop distributed data processing solutions using Apache Spark and PySpark.
• Implement real-time data streaming and ingestion solutions using Apache Kafka and MQTT.
• Design and maintain Redshift-based data warehouses, data marts, and analytical data models.
• Implement workflow orchestration, scheduling, and monitoring using Apache Airflow.
• Write optimized and reusable Python and SQL code for data processing.
• Ensure data quality, reliability, scalability, and performance across data pipelines.
• Troubleshoot pipeline failures and optimize data processing workflows.
• Collaborate with cross-functional teams to deliver reliable data engineering solutions.

Conocimientos

Data engineering

AWS data engineering

Apache Spark

PySpark

Python

SQL

Apache Airflow

Kafka

ETL/ELT

Data warehousing

Data marts

Analytics modeling

Real-time data ingestion

Herramientas

Amazon Redshift

Apache Airflow

Apache Kafka

MQTT

Descripción del empleo

Role Overview

We are looking for skilled Data Engineers with hands-on experience in AWS Data Engineering, distributed data processing, and cloud-based data solutions. The ideal candidates should have strong expertise in building scalable data pipelines, working with Amazon Redshift, Apache Spark, PySpark, Kafka, Airflow, Python, and SQL.

Candidates should be capable of developing, maintaining, and optimizing ETL/ELT workflows, data warehouses, and real-time data ingestion solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS services, particularly Amazon Redshift.

• Build and optimize ETL/ELT workflows for data ingestion, transformation, and processing.

• Develop distributed data processing solutions using Apache Spark and PySpark.

• Implement real-time data streaming and ingestion solutions using Apache Kafka and MQTT.

• Design and maintain Redshift-based data warehouses, data marts, and analytical data models.

• Implement workflow orchestration, scheduling, and monitoring using Apache Airflow.

• Write optimized and reusable Python and SQL code for data processing.

• Ensure data quality, reliability, scalability, and performance across data pipelines.

• Troubleshoot pipeline failures and optimize data processing workflows.

• Collaborate with cross-functional teams to deliver reliable data engineering solutions.

Mandatory Skills

• 3–9 years of relevant experience in Data Engineering.

• Strong hands-on experience in AWS Data Engineering.

• Experience developing and managing data pipelines using Amazon Redshift.

• Proficiency in Apache Spark and PySpark.

• Strong programming skills in Python and SQL.

• Hands-on experience with Apache Airflow.

• Experience with Apache Kafka for data streaming and ingestion.

• Understanding of ETL/ELT, Data Warehousing, Data Marts, and analytical data modeling.

• Ability to build and maintain scalable, production-ready data pipelines.

Good to Have

• MQTT and real-time data ingestion.

• Tableau or other data visualization tools.

• Manufacturing industry experience.

• Industrial IoT (IIoT) and sensor-based data ingestion.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.

• Ability to independently develop and maintain data pipelines.

• Understanding of cloud-native data engineering architectures.

• Good communication and collaboration skills.

• Experience troubleshooting and optimizing enterprise data workflows.