Senior Databricks Developer

Newdream Data Systems; Tiruchirappalli; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Newdream Data Systems is seeking a Senior Databricks Developer to join our data engineering team for a US-based client. The role focuses on building scalable data pipelines and cloud analytics using Azure Databricks, PySpark, Python, SQL, and Azure Data Factory.

You will design and optimize ETL/ELT processes, implement Delta Lake and Medallion Architecture, and collaborate with global teams to deliver reliable data solutions. Strong communication is essential.

Qualifications

• 5+ years of experience in Data Engineering or ETL development.
• Strong hands-on experience with Azure Databricks and PySpark.
• Advanced SQL and Python programming skills.
• Experience with Azure Data Factory, ADLS Gen2, and Delta Lake.
• Knowledge of data warehousing and dimensional modeling.
• Experience with Git, CI/CD, and Azure DevOps.
• Strong debugging, problem-solving, and communication skills.

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Databricks and PySpark.
• Build scalable data processing solutions with Python and SQL.
• Develop and maintain data pipelines using Azure Data Factory.
• Implement Delta Lake architecture and Medallion Architecture (Bronze, Silver, Gold).
• Optimize Databricks jobs, queries, and workflows for performance and cost.
• Integrate data from multiple sources into ADLS.
• Implement data quality checks, validation, monitoring, and error handling.
• Collaborate with US-based teams to deliver reliable data solutions.
• Support production deployments and troubleshoot performance.

Skills

ETL development

PySpark

Python

SQL

Azure Data Factory

Delta Lake

Data Warehousing

Git & CI/CD

Azure DevOps

Communication skills

Tools

Azure Databricks

ADLS Gen2

Delta Lake

Azure Data Factory

Medallion Architecture

Job description

Experience: 5+ Years

Client: US-Based Client

Position: Senior Databricks Developer / Data Engineer

Job Description

Senior Databricks Developer – 5+ Years Experience

Experience: 5+ Years

Client: US-Based Client

Position: Senior Databricks Developer / Data Engineer

Job Description

We are seeking an experienced Senior Databricks Developer to join our team and work on challenging data engineering and cloud analytics projects for a US-based client.

The ideal candidate will have strong hands-on experience with Azure Databricks, PySpark, Python, SQL, and Azure Data Factory (ADF), along with expertise in designing and implementing scalable data pipelines and modern data platforms.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Databricks and PySpark.


• Build scalable data processing solutions using Python, SQL, and Apache Spark.


• Develop and maintain data pipelines using Azure Data Factory.


• Implement Delta Lake architecture and Medallion Architecture (Bronze, Silver, Gold).


• Optimize Databricks jobs, queries, and workflows for performance and cost efficiency.


• Integrate data from multiple sources into Azure Data Lake Storage (ADLS).


• Implement data quality checks, validation, monitoring, and error handling.


• Collaborate with US-based technical and business teams to deliver reliable data solutions.


• Support production deployments, troubleshooting, and performance improvements.

Required Skills

• 5+ years of experience in Data Engineering or ETL development.


• Strong hands-on experience with Azure Databricks and PySpark.


• Advanced SQL and Python programming skills.


• Experience with Azure Data Factory, ADLS Gen2, and Delta Lake.


• Good understanding of data warehousing, dimensional modeling, and ETL/ELT concepts.


• Experience with Git, CI/CD pipelines, and Azure DevOps.


• Strong debugging, problem-solving, and communication skills.

Preferred Qualifications

• Experience with Databricks Unity Catalog and Delta Live Tables/Lakeflow.


• Knowledge of Microsoft Fabric, Azure Synapse Analytics, or Power BI.


• Experience with large-scale data migrations and performance optimization.


• Databricks or Microsoft Azure certifications.


• Prior experience working with US-based clients.

Why Join Us?

This is an excellent opportunity to work on modern cloud data platforms, solve complex engineering challenges, collaborate with experienced professionals, and contribute to impactful projects for a US-based client.