Senior Azure Databricks Engineer

Tata Consultancy Services; Pune District; On-site; INR 1,200,000 - 2,500,000; Full time

Job summary

Tata Consultancy Services is seeking a Data Engineer to design, build, and maintain scalable data pipelines in Databricks. You will integrate data from Azure Storage, SFTP, and Fabric using Apache Spark and ensure data quality and completeness.

You will collaborate with data owners, analysts, and stakeholders to understand data requirements and deliver robust solutions while monitoring pipelines for issues and ensuring data flow integrity.

Qualifications

• Design and implement scalable data pipelines in Databricks.
• Proficiency in Python or Scala for Spark processing.
• Strong SQL skills and data integration from cloud/storage sources.

Responsibilities

• Design, build, and maintain data pipelines.
• Integrate data from Azure Storage, SFTP, and Fabric using Apache Spark in Databricks.
• Implement data quality checks to ensure accuracy and completeness.
• Collaborate with data owners, analysts, and stakeholders to meet data needs.
• Monitor pipelines and resolve issues to maintain data flow integrity.

Skills

Databricks

Apache Spark

Python

Scala

SQL

Azure Cloud

ETL

Unity Catalog

Data Pipelines

Tools

Azure Storage

SFTP

Fabric

Job description

Design, build, and maintain scalable and efficient data pipelines using Databricks.

Integrate data from various sources (for eg. Azure storage account, SFTP, Fabric) using Apache Spark within Databricks.

Implement processes and checks to ensure the accuracy and completeness of data.

Work closely with data owners, analysts, and other stakeholders to understand data requirements and deliver solutions.

Monitor data pipelines for issues and resolve them promptly to maintain data flow integrity. Experience with Databricks-specific tools and features.

Proficiency in programming languages such as Python or Scala for data processing on Databricks Experience with Azure cloud platform.

Strong understanding of SQL.

Experience with ETL and processes, Unity Catalog.