Senior Data Brick Engineer

Zenithive Engineering; Ahmedabad District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Zenithive Engineering is hiring a Data Engineer to design and build scalable ETL/ELT pipelines on the Databricks platform in Ahmedabad, India. You will develop data lakes, data warehouses (Delta Lake) and optimize data flows for performance and cost efficiency.

You'll collaborate with analysts, scientists and other engineers, implement data quality checks, monitor workflows with Airflow or Databricks Workflows, and contribute to data governance and security best practices.

Qualifications

• Strong proficiency in Apache Spark, PySpark, SQL, Python and Delta Lake.
• Experience with cloud platforms (Azure, AWS, or GCP), preferably Azure Databricks, Azure Data Factory, and Azure Synapse.
• Deep understanding of data modeling, data warehousing, and ETL processes.
• Hands-on experience with job orchestration tools like Databricks Workflows, Airflow, or similar.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Apache Spark on Databricks.
• Build and maintain robust data lakes and data warehouses (Delta Lake, Lakehouse architecture).
• Optimize data flows for performance, scalability, and cost-efficiency on the Databricks platform.
• Collaborate with data analysts, scientists, and other engineers to integrate data from diverse sources.
• Implement data quality checks, monitoring, and alerting solutions.
• Develop CI/CD pipelines for data jobs and workflows.
• Support the migration of legacy data systems to the Databricks ecosystem.
• Contribute to data governance and security best practices

Skills

Apache Spark

PySpark

SQL

Python

Delta Lake

Data warehousing

ETL processes

Cloud platforms

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

Airflow

Job description

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Apache Spark on Databricks.
• Build and maintain robust data lakes and data warehouses (Delta Lake, Lakehouse architecture).
• Optimize data flows for performance, scalability, and cost-efficiency on the Databricks platform.
• Collaborate with data analysts, scientists, and other engineers to integrate data from diverse sources.
• Implement data quality checks, monitoring, and alerting solutions.
• Develop CI/CD pipelines for data jobs and workflows.
• Support the migration of legacy data systems to the Databricks ecosystem.
• Contribute to data governance and security best practices

Required Skills

Strong proficiency in Apache Spark, PySpark, SQL,Python and Delta Lake.

Experience with cloud platforms (Azure, AWS, or GCP), preferably Azure Databricks, Azure Data Factory, and Azure Synapse.

Deep understanding of data modeling, data warehousing, and ETL processes.

Hands-on experience with job orchestration tools like Databricks Workflows, Airflow, or similar.