Azure Data Engineer with Kafka

Hexaware Technologies; Hyderabad; Hybrid; INR 900,000 - 2,100,000; Full time

Job summary

Hexaware Technologies in Hyderabad seeks a Senior Data Engineer to design, develop and optimise scalable ETL/ELT pipelines using Databricks. You will build Bronze/Silver/Gold data layers with Delta Lake, implement batch and streaming ingestion, and ensure data quality and governance through Unity Catalog.

Responsibilities include production-grade transformations with Python, PySpark and SQL, and integrating Databricks with Azure data services like ADLS Gen2 and Synapse, plus CI/CD deployment

Qualifications

• Strong hands-on Databricks, Python/PySpark and SQL.
• Delta Lake and Databricks Workflows.
• ETL/ELT development and performance tuning.
• ADLS Gen2 and Azure data ecosystem.
• Unity Catalog exposure.
• Git and CI/CD experience.
• Allianz-aligned platform JD mentions Databricks/PySpark, Delta Lake, Unity Catalog and Synapse abilities.

Responsibilities

• Design, develop and optimise scalable ETL/ELT pipelines using Databricks.
• Develop production-grade data transformations using Python, PySpark and SQL.
• Build Bronze/Silver/Gold data layers using Delta Lake and medallion architecture.
• Implement batch and, where required, streaming ingestion patterns.
• Work with Databricks Workflows and Unity Catalog for orchestration and governance.
• Integrate Databricks with Azure data services including ADLS Gen2 and Synapse.
• Implement data-quality checks, error handling, monitoring and performance optimisation.
• Develop automated deployments using CI/CD and follow software engineering best practices.
• Troubleshoot production data pipelines and ensure reliability and scalability.

Skills

Databricks

Python

PySpark

SQL

Delta Lake

Unity Catalog

CI/CD

Git

ADLS Gen2

Azure Synapse

Azure data services

ETL/ELT

Performance tuning

Tools

Databricks Workflows

Job description

Job Description

• Design, develop and optimise scalable ETL/ELT pipelines using Databricks.
• Develop production-grade data transformations using Python, PySpark and SQL.
• Build Bronze/Silver/Gold data layers using Delta Lake and medallion architecture.
• Implement batch and, where required, streaming ingestion patterns.
• Work with Databricks Workflows and Unity Catalog for orchestration and governance.
• Integrate Databricks with Azure data services including ADLS Gen2 and Synapse.
• Implement data-quality checks, error handling, monitoring and performance optimisation.
• Develop automated deployments using CI/CD and follow software engineering best practices.
• Troubleshoot production data pipelines and ensure reliability and scalability.

Note

Please do not schedule any candidate below 7 years of total DATA exp.

Mandatory skills:

• Strong hands-on Databricks, Python/PySpark and SQL.
• Delta Lake and Databricks Workflows.
• ETL/ELT development and performance tuning.
• ADLS Gen2 and Azure data ecosystem.
• Unity Catalog exposure.
• Git and CI/CD experience.
• The existing Allianz-aligned platform JD similarly calls for strong Databricks/PySpark, Delta Lake, Unity Catalog and Synapse capabilities, making these good baseline screening criteria