Databricks Developer(ADF) walk in for pune

Tata Consultancy Services; Pune District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer to design and implement end-to-end data pipelines using Azure Data Factory and Azure Databricks in Pune, India. You will develop ETL/ELT processes, optimize Spark workloads, and integrate data from multiple sources while ensuring quality and security.

The role emphasizes collaboration with analysts and architects, CI/CD delivery, and ongoing platform support in a cloud-first environment.

Qualifications

• Strong hands-on experience with Azure Databricks, PySpark, and Spark SQL.
• Expertise in Azure Data Factory pipeline development and orchestration.
• Experience with Azure Data Lake Storage Gen2, Blob Storage, and SQL databases.

Responsibilities

• Design and develop end-to-end data pipelines using Azure Data Factory and Azure Databricks.
• Build and maintain ETL/ELT workflows for ingesting, transforming, and loading large-scale data.
• Optimize Databricks jobs for performance, scalability, and cost efficiency.
• Develop data processing solutions using PySpark, Spark SQL, and Databricks Notebooks.
• Implement data quality, validation, monitoring, and error-handling frameworks.
• Collaborate with business analysts, data architects, and stakeholders to understand data requirements.
• Develop and manage CI/CD pipelines for data engineering deployments.
• Ensure data security, governance, and compliance standards are followed.
• Troubleshoot production issues and provide ongoing support for data platforms.

Skills

Azure Databricks

PySpark

Spark SQL

Azure Data Factory

ADLS Gen2

Blob Storage

SQL databases

Git

Azure DevOps

CI/CD

SQL tuning

Education

Bachelor's degree in CS/Engineering or related field

Tools

CI/CD pipelines

Job description

• Design and develop end-to-end data pipelines using Azure Data Factory (ADF) and Azure Databricks.
• Build and maintain ETL/ELT workflows for ingesting, transforming, and loading large-scale data.
• Develop data processing solutions using PySpark, Spark SQL, and Databricks Notebooks.
• Integrate data from multiple sources including SQL databases, APIs, flat files, and cloud storage.
• Optimize Databricks jobs for performance, scalability, and cost efficiency.
• Implement data quality, validation, monitoring, and error-handling frameworks.
• Collaborate with business analysts, data architects, and stakeholders to understand data requirements.
• Develop and manage CI/CD pipelines for data engineering deployments.
• Ensure data security, governance, and compliance standards are followed.
• Troubleshoot production issues and provide ongoing support for data platforms.

Required Skills :-

• Strong hands-on experience with Azure Databricks, PySpark, and Spark SQL.
• Expertise in Azure Data Factory (ADF) pipeline development and orchestration.
• Experience with Azure Data Lake Storage (ADLS Gen2), Blob Storage, and SQL databases.
• Good knowledge of ETL/ELT architecture, data modeling, and cloud data solutions.
• Experience with Git, Azure DevOps, and CI/CD processes.
• Strong SQL and performance tuning skills.