Data Engineer

Zorba Consulting; Hyderabad City Taluka; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Zorba Consulting in India is seeking an experienced Data Engineer to design, develop and optimize scalable data pipelines using Spark, Python, Scala, Databricks, Airflow, and ADLS. The role spans cloud and big-data platforms across Azure services.

You will work with architects and stakeholders to implement CI/CD practices and ensure data quality, performance, and governance across ETL/ELT processes.

Qualifications

• Strong hands-on experience with Apache Spark.
• Proficient in Scala and Python programming.
• Extensive SQL development and data transformation skills.
• Experience with Azure Data Lake Storage and Databricks.
• Knowledge of ETL/ELT concepts and data quality practices.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Spark, Python, and Scala.
• Develop and optimize Spark-based batch and distributed data processing applications.
• Build and manage data pipelines and workflows using Airflow.
• Develop data engineering solutions on Azure Databricks, notebooks, jobs, workflows, and Delta Lake.
• Work with Microsoft Fabric for ingestion, transformation, orchestration, and analytics workloads.
• Design and manage data storage with ADLS/ADLS Gen2.
• Write complex SQL queries for data transformation and validation.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Optimize Spark jobs and pipelines for performance and scalability.
• Collaborate with architects, analysts, developers, and stakeholders; implement CI/CD and production practices.

Skills

Apache Spark

Scala

Python

Databricks

Airflow

Azure Data Lake Storage

SQL

Azure

Data modeling

Big data

Education

Bachelor's degree in CS/IT/Engineering

Tools

Azure Databricks

Kafka

GitHub

Azure DevOps

Job description

Job Summary

We are looking for an experienced Data Engineer with strong expertise in Apache Spark, Scala, Python, Databricks, Apache Airflow, Microsoft Fabric, SQL, and Azure Data Lake Storage (ADLS).

The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data engineering solutions across cloud and big-data platforms.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Apache Spark, Python, and Scala
• Develop and optimize Spark-based batch and distributed data processing applications
• Build and manage data pipelines and workflows using Apache Airflow
• Develop data engineering solutions using Azure Databricks, including notebooks, jobs, workflows, and Delta Lake
• Work with Microsoft Fabric for data ingestion, transformation, orchestration, and analytics workloads
• Design and manage data storage solutions using Azure Data Lake Storage (ADLS/ADLS Gen2)
• Write complex and optimized SQL queries for data transformation, validation, and analysis
• Implement data quality, validation, monitoring, and error‑handling mechanisms
• Optimize Spark jobs, SQL queries, and data pipelines for performance and scalability
• Collaborate with architects, analysts, developers, and business stakeholders to understand data requirements
• Implement CI/CD and follow development, testing, deployment, and production‑support best practices
• Troubleshoot pipeline failures and resolve data‑processing and performance issues

Required Skills

• Strong hands‑on experience with Apache Spark
• Strong programming experience in Scala and/or Python
• Hands‑on experience with Apache Airflow and workflow orchestration
• Strong experience with Azure Databricks
• Hands‑on experience with Microsoft Fabric
• Strong SQL development and query optimization skills
• Experience with Azure Data Lake Storage (ADLS/ADLS Gen2)
• Good understanding of ETL/ELT concepts and data engineering principles
• Experience working with large‑scale distributed data processing environments
• Good understanding of data pipeline architecture, data quality, and performance optimization

Good to Have

• Experience with Delta Lake and Medallion Architecture
• Experience with Microsoft Fabric Data Factory, Lakehouse, and Warehouse
• Azure cloud services and data engineering ecosystem
• CI/CD using Azure DevOps or GitHub
• Experience with Kafka or other streaming technologies
• Knowledge of data governance and security practices
• Experience5 to 8 years of overall experience in Data Engineering / Big Data Engineering, with strong hands‑on experience in the required technologies

Education

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field

Location - Pan India-Bengaluru,Hyderabad,Delhi / NCR,Chennai,Pune,Kolkata,Ahmedabad,Mumbai