Databricks data engineer

NYX TECH SOLUTION; Pune District; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

NYX TECH SOLUTION is seeking an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines in Azure Databricks. The role focuses on Python, PySpark, SQL, and Delta Lake within a Lakehouse architecture to ensure data quality and reliability.

You will build ingestion pipelines from multiple sources, optimize Spark jobs, and collaborate with data architects and analytics teams while using ADF for orchestration and integration.

Qualifications

• 3-4 years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks.
• Excellent knowledge of Python and PySpark.
• Strong SQL skills, including complex queries, joins, CTEs, and window functions.
• Experience with Apache Spark and distributed data processing.
• Hands-on experience with Delta Lake and Delta Tables.
• Good understanding of ETL/ELT concepts and data warehousing.
• Experience with Azure Data Factory (ADF).
• Understanding of cloud storage services such as Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Git and version control.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Work extensively with Python, PySpark, and SQL for data transformation and processing.
• Develop and optimize ETL/ELT workflows for large-scale data processing.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Build and maintain data ingestion pipelines from multiple data sources.
• Implement data cleansing, transformation, validation, and quality checks.
• Optimize Spark jobs and SQL queries for performance and cost efficiency.
• Work with Azure Data Factory (ADF) for orchestration and pipeline integration.
• Collaborate with data architects, analysts, and other engineering teams.
• Troubleshoot data pipeline failures and ensure data accuracy and reliability.

Skills

Python

PySpark

SQL

Azure Databricks

Data pipelines

Delta Lake

Spark

Git

Tools

Azure Data Factory

ADLS Gen2

Git

Azure Synapse Analytics

Job description

Job Summary

We are looking for an experienced Databricks Data Engineer with 34 years of hands-on experience in designing, developing, and maintaining scalable data engineering solutions. The ideal candidate should have strong expertise in Python, PySpark, SQL, Azure Databricks, and data pipeline development.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Work extensively with Python, PySpark, and SQL for data transformation and processing.
• Develop and optimize ETL/ELT workflows for large-scale data processing.
• Work with Delta Lake, Delta Tables, and Lakehouse architecture.
• Build and maintain data ingestion pipelines from multiple data sources.
• Implement data cleansing, transformation, validation, and quality checks.
• Optimize Spark jobs and SQL queries for performance and cost efficiency.
• Work with Azure Data Factory (ADF) for orchestration and pipeline integration.
• Collaborate with data architects, analysts, and other engineering teams.
• Troubleshoot data pipeline failures and ensure data accuracy and reliability.

Required Technical Skills

• 3-4 years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks.
• Excellent knowledge of Python and PySpark.
• Strong SQL skills, including complex queries, joins, CTEs, and window functions.
• Experience with Apache Spark and distributed data processing.
• Hands-on experience with Delta Lake and Delta Tables.
• Good understanding of ETL/ELT concepts and data warehousing.
• Experience with Azure Data Factory (ADF).
• Understanding of cloud storage services such as Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Git and version control.

Good to Have

• Experience with Azure Synapse Analytics.
• Knowledge of Microsoft Fabric.
• Understanding of CI/CD pipelines and DevOps practices.
• Experience working with dimensional modelling and data warehouse concepts.

Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to work independently and within a team.
• Good communication and collaboration skills.
• Ability to handle multiple data engineering tasks and meet project deadlines.