Databricks Developer - ADF

Tata Consultancy Services; Pune District; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Tata Consultancy Services in Pune is seeking a Senior Databricks Engineer to design, build, and optimize data pipelines on Databricks using Spark. The role requires strong Python/Scala skills and Azure cloud experience.

You will work with data owners and analysts to translate requirements into scalable data solutions, ensure data quality, and monitor data flows in a collaborative, fast-paced environment.

Qualifications

• 4–10 years of experience in data engineering with Databricks.
• Proficient in Python or Scala for data processing.
• Experience with Azure cloud and SQL.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks.
• Integrate data from sources such as Azure storage, SFTP, and Fabric.
• Implement data quality checks to ensure accuracy and completeness.
• Collaborate with data owners, analysts, and stakeholders to meet requirements.
• Monitor pipelines to maintain data flow integrity.

Skills

Databricks

Azure Databricks

Apache Spark

Python

Scala

SQL

ETL

Unity Catalog

Job description

Job Description

SN Required Information Details

1 Role Senior Databricks Engineer

2 Required Technical Skill Set Azure Databricks + Apache Spark

4 Desired Experience Range 4-10 years

5 Location of Requirement Pune

Desired Competencies (Technical/Behavioral Competency)

• Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks.
• Implement processes and checks to ensure the accuracy and completeness of data.
• Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity.
• Experience with Databricks-specific tools and features.
• Proficiency in programming languages such as Python or Scala for data processing on Databricks
• Experience with Azure cloud platform.
• Strong understanding of SQL.
• Experience with ETL and processes, Unity Catalog.

Good-to-Have Certifications in Databricks or related cloud technologies

SN Responsibility of / Expectations from the Role

• Design, build, and maintain scalable and efficient data pipelines using Databricks. Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks.
• Implement processes and checks to ensure the accuracy and completeness of data.
• Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions
• Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity.