AWS DATA Engineer (Databricks)

ADP Private Limited; Hyderabad; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

ADP Private Limited in Hyderabad seeks a Data Engineer specializing in AWS Databricks to design scalable ETL/ELT pipelines and build Lakehouse architectures. You will implement Databricks Workflows, CI/CD with GitHub Actions, and contribute to GenAI-enabled data solutions, while ensuring data governance with Unity Catalog and handling batch and streaming workloads.

This role demands hands-on Spark/PySpark, Python, SQL skills and strong AWS cloud knowledge, with a focus on robust data pipelines

Qualifications

• Hands-on experience designing ETL/ELT pipelines with Spark on Databricks.
• Experience building Lakehouse architectures using Delta Lake and Unity Catalog.
• Proficiency in AI/GenAI powered data solutions and data governance.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Spark on Databricks.
• Build and manage Lakehouse architectures using Delta Lake.
• Implement orchestration using Databricks Workflows.
• Develop CI/CD pipelines using GitHub Actions, Databricks Repos, and AWS tools.
• Optimize pipeline performance and troubleshoot production issues.
• Manage data governance using Unity Catalog.
• Support batch and streaming data processing workloads.
• Contribute to AI/GenAI-enabled data solutions.

Skills

AWS Databricks

Apache Spark / PySpark

Python

SQL

Delta Lake

Databricks SQL

Unity Catalog

ETL/ELT Pipeline Development

Data Lakehouse Architecture

CI/CD for Data Pipelines

AWS Cloud Platform

AI / GenAI exposure

Tools

Databricks Workflows

GitHub Actions

Databricks Repos

AWS Tools

Job description

Job ID - IND170694 - EFS FCRM

Data Engineer - AWS Databricks - GenAI

Must-Have Skills

• AWS Databricks (Hands-on implementation experience)
• Apache Spark / PySpark
• Python
• SQL
• Delta Lake
• Databricks SQL
• Unity Catalog
• ETL/ELT Pipeline Development
• Data Lakehouse Architecture
• CI/CD for Data Pipelines
• AWS Cloud Platform
• AI / Generative AI / Agentic AI exposure

Preferred Certifications

• Databricks Certified Data Engineer Associate/Professional
• AWS Certified Data Analytics
• AWS Certified Solutions Architect (Good to have)

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using Spark on Databricks.
• Build and manage Lakehouse architectures using Delta Lake.
• Implement orchestration using Databricks Workflows.
• Develop CI/CD pipelines using GitHub Actions, Databricks Repos, and AWS tools.
• Optimize pipeline performance and troubleshoot production issues.
• Manage data governance using Unity Catalog.
• Support batch and streaming data processing workloads.
• Contribute to AI/GenAI-enabled data solutions.