Azure Data Engineer with Databricks & DLT

IBM; Bengaluru; On-site; INR 1,400,000 - 2,300,000; Full time

Job summary

IBM Bengaluru seeks a strong Azure Data Engineer with 6+ years of experience. The candidate will work on advanced data pipelines using Azure Data Factory, Databricks, and Unity Catalog, with a focus on performance optimisation and cost efficiency in live streaming scenarios.

The role emphasizes ownership, analytical excellence, and clear communication, with opportunities to integrate GenAI tools to accelerate pipeline development and innovation.

Qualifications

• 6+ years of relevant industry experience as a data engineer or similar role.
• Strong ownership, analytical excellence, and excellent communication.

Responsibilities

• Hands-on optimization of DLT pipelines and cost optimization for Databricks in live streaming.
• Design and implement ELT pipelines using Azure data services and Databricks.
• Leverage GenAI-driven approaches to accelerate automated pipeline development.

Skills

Azure Data Engineer

Python

SQL

Databricks

DLT (Delta Live Tables)

Unity Catalog

Streaming Analytics

Azure Data Factory

GenAI integration

Tools

ADLS

Delta Live Tables

Photon

Serverless Databricks

Job description

Total 9-12,Relevant 6+ yrs : Mandate-Strong Azure Data Engineer with advanced skills in Python (coding/scripting),SCD, SQL, Azure Data factory, Data Modeling, and Databricks (PySpark, Delta Live Tables, structured/live streaming, Unity Catalog) Medallion Architecture. Hands-on exp in performance optimisation of DLT pipelines,cost optimisation for Databricks in live streaming (Photon, serverless/non-serverless)./ ELT/Asset Bundle /ADLS/Role requires AI/GenAI-driven innovation- leveraging LLMs/agents to accelerate automated pipeline development.(Gen AI: Genie, Mosaic AI, GitHub CoPilot - Knowledge)

Requires strong ownership, analytical excellence, self-drive, excellent communication and professional conduct .