Data Engineer – PySpark, Snowflake, AWS & GenAI

D4Insight; Hyderabad; On-site; INR 2,000,000 - 4,000,000; Full time

Job summary

D4Insight is seeking an experienced Data Engineer in Hyderabad to design and maintain scalable data pipelines using SQL, PySpark, Python, and Snowflake. You will build reusable engineering frameworks and implement Medallion Architecture across Bronze, Silver, and Gold layers, while leveraging AWS Glue/Lambda and Kafka for ingestion and processing.

You will also work on event-driven and streaming data pipelines, orchestrated with Airflow or similar tools, focusing on data quality, reliability,

Qualifications

• Experience designing and maintaining scalable data pipelines using PySpark, SQL, Python and Snowflake.
• Proven ability to build reusable data engineering frameworks for efficient data processing.
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold) and ETL/ELT processes.
• Hands-on with Kafka, AWS Glue, and AWS Lambda for data ingestion and processing.
• Experience with Apache Airflow or other workflow orchestration tools.
• Focus on data quality, reliability and performance across pipelines.
• Nice to have: GCP, LangChain, LLMs, and GenAI techniques.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, SQL, Python, and Snowflake.
• Build reusable data engineering frameworks for efficient data processing and transformation.
• Implement Medallion Architecture across Bronze, Silver, and Gold layers.
• Develop data ingestion and processing pipelines using AWS Glue and AWS Lambda.
• Implement event-driven and streaming data pipelines using Kafka.
• Orchestrate workflows using Apache Airflow or equivalent tools.
• Optimize SQL queries, Snowflake workloads, and PySpark jobs for performance and scalability.
• Ensure data quality, reliability, and consistency across pipelines.

Skills

SQL

PySpark

Python

Snowflake

Kafka

AWS Glue

AWS Lambda

Airflow

Medallion Architecture

Data pipelines

Tools

Apache Airflow

Kafka

AWS Glue

AWS Lambda

Job description

Role Overview:

We are looking for experienced Data Engineer with strong expertise in SQL, PySpark, Python, and Snowflake to design and maintain scalable data pipelines and engineering frameworks. The role involves cloud data services, Medallion Architecture, ETL/ELT workflows, and orchestration, with exposure to GenAI technologies as an added advantage.

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using PySpark, SQL, Python, and Snowflake.

Build reusable data engineering frameworks for efficient data processing and transformation.

Implement Medallion Architecture across Bronze, Silver, and Gold layers.

Develop data ingestion and processing pipelines using AWS Glue and AWS Lambda.

Implement event-driven and streaming data pipelines using Kafka.

Orchestrate workflows using Apache Airflow or equivalent tools.

Optimize SQL queries, Snowflake workloads, and PySpark jobs for performance and scalability.

Ensure data quality, reliability, and consistency across pipelines.

• Explore GenAI-based data solutions using LLMs, RAG, and LangChain.

Required Skills & Experience:

6-12 years of relevant data engineering experience.

Strong hands-on expertise in SQL, PySpark, Python, and Snowflake.

Experience building scalable data pipelines and reusable engineering frameworks.

Strong understanding of Medallion Architecture and ETL/ELT processes.

Hands-on experience with Kafka, AWS Glue, and AWS Lambda.

Experience with Apache Airflow or other workflow orchestration tools.

Strong problem-solving skills and a focus on data quality and performance.

Good to Have: GCP, additional AWS services, MCP (Model Context Protocol), LLMs, RAG, and LangChain.

We craft, deploy, and manage bespoke services in CRM, data and AI, cybersecurity and consulting.