SDR Data Engineer

Tata Consultancy Services; Kolkata District, Pune District, Bengaluru; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Tata Consultancy Services in India (Kolkata area) is seeking a Data Engineer with strong Python, Azure, PySpark and Databricks to design, develop, and maintain scalable data processing solutions. You will build data transformation pipelines, handle large datasets, and implement complex calculation engines in collaboration with engineering teams.

The role involves developing ETL/ELT pipelines, optimizing databases such as PostgreSQL and MongoDB, ensuring data quality and performance, and

Qualifications

• Experience designing ETL/ELT pipelines.
• Strong SQL and database concepts.
• Experience with PostgreSQL and/or MongoDB.
• Experience handling large-scale datasets.

Responsibilities

• Design, develop, and maintain data transformation pipelines using Azure services.
• Build and optimize scalable data processing solutions with PySpark and Databricks.
• Design and implement complex data calculation engines and business logic.
• Develop efficient data ingestion, transformation, and aggregation processes for large datasets.
• Manage and optimize databases such as PostgreSQL and MongoDB.
• Ensure data quality, integrity, performance, and reliability across data platforms.
• Collaborate with software engineers and cross-functional teams to translate requirements into solutions.
• Participate in code reviews, architecture discussions, design sessions, and sprints.
• Contribute to automation, testing, deployment, and monitoring processes.
• Troubleshoot and resolve data pipeline and application performance issues.
• Create and maintain technical documentation.

Skills

Python

Azure Cloud Services

PySpark

Databricks

SQL

PostgreSQL

MongoDB

ETL/ELT pipelines

Data modeling

Git/CI/CD

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

Azure Data Factory

Docker

Kubernetes

Kafka

Job description

Job Title: Data Engineer | Python | Azure | PySpark | Databricks | PostgreSQL/MongoDB

Location: Pune, Bengaluru, NCR, Kolkata

Experience: 6 to 12 Years

Job Summary

We are seeking a skilled Data Engineer with expertise in Python, Azure, PySpark, and Databricks to design, develop, and maintain scalable data processing solutions. The ideal candidate will be responsible for building data transformation pipelines, handling large datasets, implementing complex calculation engines, and collaborating with engineering teams to deliver high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain data transformation pipelines using Azure services such as Azure Functions and related cloud technologies.
• Build and optimize scalable data processing solutions using PySpark and Databricks.
• Design and implement complex data calculation engines and business logic frameworks.
• Develop efficient data ingestion, transformation, and aggregation processes for large datasets.
• Manage and optimize databases such as PostgreSQL and MongoDB.
• Ensure data quality, integrity, performance, and reliability across data platforms.
• Collaborate with Software Engineers and cross-functional teams to translate business requirements into technical solutions.
• Participate in code reviews, architecture discussions, technical design sessions, and sprint planning.
• Contribute to automation, testing, deployment, and monitoring processes.
• Troubleshoot and resolve data pipeline and application performance issues.
• Create and maintain technical documentation.

Required Skills & Qualifications

Technical Skills

• Strong programming experience in Python.
• Hands-on experience with Azure Cloud Services.
• Expertise in PySpark and Databricks.
• Experience designing and maintaining ETL/ELT pipelines.
• Strong knowledge of SQL and database concepts.
• Experience with PostgreSQL and/or MongoDB.
• Experience handling large-scale structured and unstructured datasets.
• Understanding of data modeling, data warehousing, and distributed computing concepts.
• Familiarity with Git, DevOps practices, and CI/CD pipelines.
• Knowledge of testing frameworks and automated deployment methodologies.

Preferred Skills

• Experience with Azure Data Factory, Azure Synapse Analytics, Event Hub, or Data Lake.
• Knowledge of containerization technologies such as Docker and Kubernetes.
• Exposure to Kafka or other streaming platforms.
• Experience working in Agile/Scrum environments.

Education

• Bachelor's or Master's Degree in Computer Science, Information Technology, Engineering, or a related field.
• Education requirements are flexible for candidates with strong relevant experience.

Certifications (Optional)

• Microsoft Azure Certifications (AZ-900, DP-203, AZ-204, or equivalent).
• Databricks Certifications.