Data Engineer - Databricks

Onedatasoftware; Coimbatore District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A technology company is seeking an experienced Azure Data Engineer to design, develop, and optimize scalable data solutions in Microsoft Azure. The ideal candidate will have strong hands-on experience with Azure Databricks, proficiency in Python, and knowledge of ETL/ELT pipeline development. Responsibilities include implementing Lakehouse architecture and collaborating with analytics teams. Joining this role offers the chance to work within a dynamic tech environment.

Qualifications

• Strong hands-on experience with Azure Databricks (Notebooks, Workflows, Delta Tables).
• Proficiency in Python, PySpark, and SQL.
• Experience with Azure Data Factory (ADF) and ADLS.
• Good understanding of Delta Lake and Medallion Architecture.
• Knowledge of CI/CD and Git version control.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks.
• Implement Lakehouse architecture using Delta Lake.
• Develop batch and streaming data pipelines using PySpark and SQL.
• Build and manage data lakes using Azure Data Lake Storage.
• Orchestrate workflows using Azure Data Factory.
• Design and implement dimensional data models.
• Optimize Databricks jobs for performance and cost efficiency.
• Ensure data quality, governance, and security.

Skills

Azure Databricks

Python

PySpark

SQL

Azure Data Factory

Delta Lake

CI/CD

Git version control

Job description

Job Summary:

We are seeking a skilled Azure Data Engineer with strong hands-on experience in Databricks to design, develop, and optimize scalable data solutions within the Microsoft Azure ecosystem. The ideal candidate should have expertise in ETL/ELT pipeline development, Lakehouse architecture, and cloud-based data engineering practices.

Key Responsibilities:

• Design and develop scalable ETL/ELT pipelines using Azure Databricks.
• Implement Lakehouse architecture using Delta Lake (Bronze / Silver / Gold layers).
• Develop batch and streaming data pipelines using PySpark and SQL.
• Build and manage data lakes using Azure Data Lake Storage (ADLS).
• Orchestrate workflows using Azure Data Factory (ADF).
• Design and implement dimensional data models (Star Schema / Snowflake Schema).
• Optimize Databricks jobs for performance and cost efficiency.
• Ensure data quality, governance, and security using Azure best practices.
• Collaborate with analytics and BI teams to deliver curated datasets for reporting.

Required Skills:

• Strong hands-on experience with Azure Databricks (Notebooks, Workflows, Delta Tables).
• Proficiency in Python, PySpark, and SQL.
• Experience with Azure Data Factory (ADF) and ADLS.
• Good understanding of Delta Lake and Medallion Architecture.
• Knowledge of CI/CD and Git version control.

Preferred Certifications:

• Databricks Data Engineer Professional (Preferred)