Azure Databricks Engineer

Tata Consultancy Services; Pune District; On-site; INR 250,000 - 450,000; Full time

Job summary

Tata Consultancy Services in Pune is seeking an experienced Azure Databricks Engineer to design and optimize data pipelines and data models on Azure Databricks. The role requires strong Spark skills and hands-on implementation.

You will work in an office-based setup, collaborating with cross-functional teams, delivering batch and real-time data workflows, and ensuring robust CI/CD for data applications. Proficiency in PySpark, SQL, and Azure services is essential.

Qualifications

• 5+ years designing and building data pipelines and models.
• Expertise in Spark-based applications and big data optimizations.
• Strong Python and PySpark experience for data processing.
• Familiarity with Azure Databricks, Data Lake, and Data Factory.

Responsibilities

• Design and optimize data pipelines and data models.
• Develop Spark-based applications for big data.
• Implement CI/CD for data apps and automate workflows.
• Work with PySpark for large-scale processing and analysis.
• Tune Spark performance for efficiency and scalability.
• Write SQL queries for large datasets and performance optimization.

Skills

Data pipelines

Data modeling

Spark

Python

PySpark

Big data

SQL

CI/CD

Tools

Azure Databricks

Azure Data Lake

Azure Data Factory

Power BI

GitLab

Job description

Dear Professionals

Greetings from Tata consultancy Services,

Job Title : Azure Databricks Engineer

Experiernce: 6 to 12 Years

Location: Pune

Mode of Work : Work from Office

Mode of Interview: Virtual

Key Responsibilities

Role : Azure Databricks Engineer

• Minimum of 5 years of hands‑on experience in designing, building, and optimizing data pipelines, data models and spark-based applications in Big Data environments.
• Extensive experience and deep expertise in data modeling and data model concepts, particularly with large datasets, ensuring the design and implementation of efficient, scalable, and high‑performing data models.
• Strong software engineer, you take pride in what youre developing with a strong testing ethos.
• String proficiency in Python programming, with a focus on data processing and analysis
• Proven experience working with PySpark for large-scale data processing and analysis
• Extensive experience in designing, building, and optimizing Big Data pipelines and architectures, with a strong focus on supporting both batch and real-time data workflows.
• Hand‑on Experience with cloud technologies and big data concepts, especially in Azure Databricks, Azure Data Lake and Azure Data Factory
• In‑Depth Knowledge of Spark, including experience with Spark performance tuning techniques to optimal processing efficiency
• Strong SQL skills for querying and manipulating large datasets, with experience in optimizing complex queries for performance

• Experience in automating data workflows and implementing CI/CD pipelines for data applications, ensuring efficient deployment and monitoring
• Any experience / knowledge in PowerShell, bash, perl scripting is desirable.
• Any Knowledge in BI Reporting tool (eg. Power BI) is a plus
• Experience with Agile Methodology, Scrum Methodology is a plus
• Experience in the financial sector is a plus, but not essential

Experience with DevOps tools such as GitLab and CI/CD pipelines is a plus