Senior Data Engineer - S

Tata Consultancy Services; Dadri, Chennai District; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

Tata Consultancy Services is seeking a senior Data Engineer to design, develop, and optimize scalable data pipelines using Databricks, Python, and PySpark. You will work across data platforms to deliver robust data solutions.

Join a collaborative team and contribute to data lakehouse architectures, ETL/ELT workflows, and cloud modernization initiatives while mentoring junior engineers and upholding data governance standards.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, or a related field.
• 5+ years of experience in Data Engineering and Data Warehousing.
• Strong hands-on experience with Databricks platform.
• Expertise in Python and PySpark development.
• Strong proficiency in SQL and data modeling concepts.
• Experience building scalable ETL/ELT pipelines.
• Knowledge of distributed data processing and big data technologies.
• Experience working with large datasets in cloud environments.
• Understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
• Strong analytical, debugging, and problem-solving skills.
• Experience with Agile development methodologies.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Python, and PySpark.
• Build and optimize ETL/ELT workflows for large-scale data.
• Develop and maintain data lakes, data warehouses, and Lakehouse architectures.
• Implement batch and real-time data processing solutions to support business requirements.
• Collaborate with Data Architects, Data Scientists, Analysts, and Business stakeholders to deliver data-driven solutions.
• Optimize data ingestion, transformation, storage, and processing performance.
• Ensure data quality, integrity, security, and governance standards across data platforms.
• Develop reusable frameworks, utilities, and automation solutions for data engineering processes.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Participate in data platform modernization and cloud transformation initiatives.
• Document data flows, technical designs, and operational processes.
• Mentor junior engineers and contribute to engineering best practices.

Skills

Databricks

Python

PySpark

SQL

Data Modeling

ETL/ELT

Cloud Data Platforms

Data Lakehouse

Agile

Big Data

Education

Bachelor's degree in CS/IT/Engineering/Math

Job description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Python, and PySpark.
• Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.
• Develop and maintain data lakes, data warehouses, and Lakehouse architectures.
• Implement batch and real-time data processing solutions to support business requirements.
• Collaborate with Data Architects, Data Scientists, Analysts, and Business stakeholders to deliver data-driven solutions.
• Optimize data ingestion, transformation, storage, and processing performance.
• Ensure data quality, integrity, security, and governance standards across data platforms.
• Develop reusable frameworks, utilities, and automation solutions for data engineering processes.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Participate in data platform modernization and cloud transformation initiatives.
• Document data flows, technical designs, and operational processes.
• Mentor junior engineers and contribute to engineering best practices.

Required Skills & Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, or a related field.
• 5+ years of experience in Data Engineering and Data Warehousing.
• Strong hands-on experience with Databricks platform.
• Expertise in Python and PySpark development.
• Strong proficiency in SQL and data modeling concepts.
• Experience building scalable ETL/ELT pipelines.
• Knowledge of distributed data processing and big data technologies.
• Experience working with large datasets in cloud environments.
• Understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
• Strong analytical, debugging, and problem-solving skills.
• Experience with Agile development methodologies.