Walk-in | Pyspark Data engineer

Tata Consultancy Services; Pune District; On-site; INR 1,000,000 - 1,400,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer in Pune to design, build, and optimize scalable data pipelines using PySpark and Python. You will work with large, diverse datasets, implement batch and real-time processing, and collaborate with analysts and scientists to deliver robust data solutions.

Responsibilities include developing Spark jobs for cleansing and transformation, optimizing performance, and ensuring governance, security, and compliance across data workloads.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Python.
• Build ETL/ELT frameworks for ingesting, transforming, and loading large datasets.
• Develop Spark jobs for data cleansing, transformation, aggregation, and validation.
• Optimize Spark applications for performance using partitioning, caching, broadcast joins, and tuning techniques.
• Work with structured and unstructured data from multiple sources.
• Implement batch and real-time data processing solutions.
• Develop and optimize SQL queries, stored procedures, and data models.
• Collaborate with business analysts, architects, and data scientists to deliver data solutions.
• Ensure data quality, governance, security, and compliance standards.
• Implement CI/CD and DevOps practices for data engineering workloads.
• Troubleshoot production issues and provide operational support.

Job description

Role & responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Python.
• Build ETL/ELT frameworks for ingesting, transforming, and loading large datasets.
• Develop Spark jobs for data cleansing, transformation, aggregation, and validation.
• Optimize Spark applications for performance using partitioning, caching, broadcast joins, and tuning techniques.
• Work with structured and unstructured data from multiple sources.
• Implement batch and real-time data processing solutions.
• Develop and optimize SQL queries, stored procedures, and data models.
• Collaborate with business analysts, architects, and data scientists to deliver data solutions.
• Ensure data quality, governance, security, and compliance standards.
• Implement CI/CD and DevOps practices for data engineering workloads.
• Troubleshoot production issues and provide operational support.