Azure Databricks Data Engineer

Aspire Systems; Chennai District; Hybrid; INR 1,400,000 - 2,800,000; Full time

Job summary

Aspire Systems is seeking an experienced Data Engineer to design and develop scalable data pipelines on Azure Databricks, utilizing PySpark, Spark SQL, and Python. You will build ETL/ELT solutions and manage Delta Lake storage.

The role involves implementing CI/CD with Azure DevOps, working with ADLS Gen2, ADF, and SQL, and collaborating with architects and stakeholders to ensure performance, security, and governance across data platforms.

Qualifications

• Strong experience with Azure Databricks and PySpark/Python.
• Proficient in Apache Spark and Spark SQL.
• Hands-on with Azure Data Factory and ADLS Gen2.
• Solid knowledge of Delta Lake and data lake architecture.
• Strong SQL and relational database skills.
• Experience with Git and Azure DevOps tooling.
• Familiar with Databricks Unity Catalog and governance concepts.
• Exposure to Azure Synapse Analytics and streaming/event hubs is a plus.
• Strong troubleshooting and performance tuning abilities.

Responsibilities

• Design and develop scalable data pipelines and ETL/ELT processes using Azure Databricks.
• Develop data transformation and processing solutions using PySpark, Spark SQL, and Python.
• Build and maintain data pipelines using Azure Data Factory (ADF).
• Work with Azure storage and data services such as ADLS Gen2 and Azure SQL.
• Implement data ingestion from structured and unstructured sources.
• Develop optimized Spark jobs and troubleshoot performance issues.
• Design data models and implement data transformation logic according to business requirements.
• Work with Delta Lake for reliable and scalable data storage and processing.
• Implement data quality, validation, error handling, and monitoring mechanisms.
• Manage Databricks workflows, jobs, clusters, notebooks, and configurations.
• Implement CI/CD practices for Databricks and data engineering using Azure DevOps/Git.
• Collaborate with data architects, analysts, developers, and business stakeholders.
• Ensure solutions meet security, performance, scalability, and governance requirements.

Skills

Azure Databricks

PySpark

Python

Apache Spark

Spark SQL

Azure Data Factory

Azure Data Lake Storage Gen2

Delta Lake

ETL/ELT

SQL

Git

Azure DevOps

Databricks Unity Catalog

Azure Synapse Analytics

Azure Event Hubs

Tools

Azure Databricks

Azure Data Factory

Azure Data Lake Storage Gen2

Delta Lake

Azure DevOps

Git

Databricks Unity Catalog

Azure Synapse Analytics

Azure Event Hubs

Job description

Role & responsibilities

• Design and develop scalable data pipelines and ETL/ELT processes using Azure Databricks.
• Develop data transformation and processing solutions using PySpark, Spark SQL, and Python.
• Build and maintain data pipelines using Azure Data Factory (ADF).
• Work with Azure storage and data services such as Azure Data Lake Storage (ADLS Gen2) and Azure SQL.
• Implement data ingestion from structured and unstructured data sources.
• Develop optimized Spark jobs and troubleshoot performance issues.
• Design data models and implement data transformation logic according to business requirements.
• Work with Delta Lake for reliable and scalable data storage and processing.
• Implement data quality, validation, error handling, and monitoring mechanisms.
• Manage Databricks workflows, jobs, clusters, notebooks, and associated configurations.
• Implement CI/CD practices for Databricks and data engineering solutions using Azure DevOps/Git.
• Collaborate with data architects, analysts, developers, and business stakeholders.
• Ensure solutions meet security, performance, scalability, and governance requirements.

Mandatory Skills

• Strong experience in Azure Databricks.
• Strong hands‑on experience with PySpark and Python.
• Good knowledge of Apache Spark and Spark SQL.
• Experience with Azure Data Factory.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Strong understanding of Delta Lake and data lake architecture.
• Good understanding of ETL/ELT concepts and data warehousing.
• Experience with SQL and relational databases.
• Knowledge of Git and Azure DevOps.
• Strong troubleshooting and performance‑tuning skills.

Good-to-Have Skills

• Experience with Azure Synapse Analytics.
• Knowledge of Azure Event Hubs or other streaming technologies.
• Experience with Databricks Unity Catalog and data governance.
• Knowledge of real-time/streaming data processing.
• Experience with CI/CD and infrastructure automation.
• Knowledge of dimensional data modeling and Medallion Architecture.
• Experience working in Agile/Scrum environments.