Azure Data Engineer (6 years To 12 years)

Tata Consultancy Services; Pune District; On-site; INR 1,200,000 - 2,300,000; Full time

Job summary

Tata Consultancy Services in Pune invites a data engineer to design, build and optimize end-to-end data pipelines and models for large-scale analytics.

The role emphasizes PySpark, Python, SQL and cloud data platforms (Azure Databricks, Data Lake, Data Factory) with emphasis on batch and real-time workflows, strong testing ethos, and scalable architectures.

Qualifications

• Experience designing, building, and optimizing data pipelines and data models for large-scale processing.

Responsibilities

Skills

PySpark data processing

Data modelling concepts

Python for data processing

SQL for large datasets

PySpark performance tuning

Data pipelines design & optimization

Cloud big data concepts

Azure Databricks

Azure Data Lake

Azure Data Factory

Big data concepts

Tools

Azure Databricks

Azure Data Lake

Azure Data Factory

PowerShell

Bash scripting

GitLab CI/CD

Power BI

Job description

Must Have Skills:

• Hands-on experience in designing, building, and optimizing data pipelines, data models and PySpark for large-scale data processing and analysis
• Extensive experience and deep expertise in data modelling and data model concepts, particularly with large datasets, ensuring the design and implementation of efficient, scalable, and high-performing data models.
• Strong software engineer, you take pride in what youre developing with a strong testing ethos.
• String proficiency in Python programming, with a focus on data processing and analysis
• Extensive experience in designing, building, and optimizing ADB/ PySpark pipelines and architectures, with a strong focus on supporting both batch and real-time data workflows.
• Hand-on Experience with cloud technologies and big data concepts, especially in Azure Databricks, Azure Data Lake and Azure Data Factory
• In-Depth Knowledge of PySpark, including experience with PySpark performance tuning techniques to optimal processing efficiency
• Strong SQL skills for querying and manipulating large datasets, with experience in optimizing complex queries for performance

Good to Have Skills:

• Experience in automating data workflows and implementing CI/CD pipelines for data applications, ensuring efficient deployment and monitoring
• Any experience / knowledge in PowerShell, bash, Perl scripting is desirable.
• Any Knowledge in BI Reporting tool (eg. Power BI) is a plus
• Experience with Agile Methodology, Scrum Methodology is a plus
• Experience in the financial sector is a plus, but not essential
• Experience with DevOps tools such as GitLab and CI/CD pipelines is a plus