Senior Spark & Azure Data Engineer (Airflow + Databricks)

Zorba Consulting; Hyderabad City Taluka; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Zorba Consulting in India is seeking an experienced Data Engineer to design, develop and optimize scalable data pipelines using Spark, Python, Scala, Databricks, Airflow, and ADLS. The role spans cloud and big-data platforms across Azure services.

You will work with architects and stakeholders to implement CI/CD practices and ensure data quality, performance, and governance across ETL/ELT processes.

Qualifications

• Strong hands-on experience with Apache Spark.
• Proficient in Scala and Python programming.
• Extensive SQL development and data transformation skills.
• Experience with Azure Data Lake Storage and Databricks.
• Knowledge of ETL/ELT concepts and data quality practices.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Spark, Python, and Scala.
• Develop and optimize Spark-based batch and distributed data processing applications.
• Build and manage data pipelines and workflows using Airflow.
• Develop data engineering solutions on Azure Databricks, notebooks, jobs, workflows, and Delta Lake.
• Work with Microsoft Fabric for ingestion, transformation, orchestration, and analytics workloads.
• Design and manage data storage with ADLS/ADLS Gen2.
• Write complex SQL queries for data transformation and validation.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Optimize Spark jobs and pipelines for performance and scalability.
• Collaborate with architects, analysts, developers, and stakeholders; implement CI/CD and production practices.

Skills

Apache Spark

Scala

Python

Databricks

Airflow

Azure Data Lake Storage

SQL

Azure

Data modeling

Big data

Education

Bachelor's degree in CS/IT/Engineering

Tools

Azure Databricks

Kafka

GitHub

Azure DevOps

Job description

Zorba Consulting in India is seeking an experienced Data Engineer to design, develop and optimize scalable data pipelines using Spark, Python, Scala, Databricks, Airflow, and ADLS. The role spans cloud and big-data platforms across Azure services.

You will work with architects and stakeholders to implement CI/CD practices and ensure data quality, performance, and governance across ETL/ELT processes.