Azure Data Engineer

Tata Consultancy Services; Pune District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Pune invites a senior data engineer with 6–12 years of IT experience to design, build, and optimize data pipelines and data models using PySpark.

You will work on large-scale data processing, ensure performance, and collaborate across teams while adopting CI/CD practices and cloud-native architectures on Azure.

The role requires strong Python and SQL skills, expertise with Azure Databricks and Data Factory, and a passion for robust testing and quality.

Qualifications

• 6–12 years of IT/data engineering experience.
• Hands-on PySpark, data pipelines, and data models.
• Strong Python and SQL skills for large datasets.
• Azure Databricks, Data Lake, and Data Factory experience.
• PySpark performance tuning for efficient processing.
• Cloud and big data concepts with Azure.

Skills

PySpark

Data modeling

Python

SQL

Azure Databricks

Azure Data Lake

Azure Data Factory

Big data

Education

15 years of Full Time Education

Tools

GitLab

PowerShell

Bash

Kubernetes

Job description

Experience Range: - 06 To 12 Years (Mandatory)

(Note: Candidates below 6 years of IT experience shall not be considered)

JOB LOCATION: Pune

JOB DESCRIPTION

Must Have Skills

• Hands-on experience in designing, building, and optimizing data pipelines, data models and PySpark for large-scale data processing and analysis
• Extensive experience and deep expertise in data modeling and data model concepts, particularly with large datasets, ensuring the design and implementation of efficient, scalable, and high-performing data models.
• Strong software engineer, you take pride in what you’re developing with a strong testing ethos.
• String proficiency in Python programming, with a focus on data processing and analysis
• Extensive experience in designing, building, and optimizing ADB/PySpark pipelines and architectures, with a strong focus on supporting both batch and real-time data workflows.
• Hand-on Experience with cloud technologies and big data concepts, especially in Azure Databricks, Azure Data Lake and Azure Data Factory
• In-Depth Knowledge of PySpark, including experience with PySpark performance tuning techniques to optimal processing efficiency
• Strong SQL skills for querying and manipulating large datasets, with experience in optimizing complex queries for performance

Good to Have Skills

• Experience in automating data workflows and implementing CI/CD pipelines for data applications, ensuring efficient deployment and monitoring
• Any experience / knowledge in PowerShell, bash, perl scripting is desirable.
• Any Knowledge in BI Reporting tool (eg. Power BI) is a plus
• Experience with Agile Methodology, Scrum Methodology is a plus
• Experience in the financial sector is a plus, but not essential
• Experience with DevOps tools such as GitLab and CI/CD pipelines is a plus

15 years of Full Time Education