Python Pyspark & Automation Engineer

Tata Consultancy Services; India; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced Python PySpark & Automation Engineer to design, develop, and optimize large-scale data processing solutions.

Ideal candidates have automation scripting experience, cloud platform familiarity (AWS/Azure), and strong data engineering skills to drive scalable pipelines and robust CI/CD practices.

Qualifications

• 6–10 years IT experience with Python and PySpark.

Responsibilities

• Develop scalable data pipelines using Python and PySpark.
• Design automation solutions to improve efficiency.
• Build reusable automation frameworks and scripts.
• Optimize Spark jobs for performance and scalability.
• Develop and consume REST APIs for integrations.
• Collaborate with data engineers, developers and stakeholders.
• Implement CI/CD pipelines and best practices.
• Troubleshoot production issues with timely fixes.
• Ensure code quality via reviews and tests.

Skills

Python

PySpark

Automation Frameworks

Data Engineering

SQL

Shell Scripting

CI/CD

REST API

Cloud AWS

Cloud Azure

Git

Test Automation

Tools

Databricks

Airflow

Docker

Kubernetes

Selenium

Terraform

Job description

Job Details

Python PySpark & Automation Engineer
Experience: 6 to 10 Years
Location: PAN India
Mode of Work: Work from Office
Mode of Interview: Virtual

Job Summary

We are seeking an experienced Python PySpark & Automation Engineer with strong expertise in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate should have experience in automation frameworks, scripting, cloud platforms, and data engineering practices.

Required Skills

• Python Programming
• PySpark
• Automation Framework Development
• Data Engineering & ETL Processes
• SQL and Database Management
• Shell Scripting
• CI/CD Pipelines (Jenkins, GitHub Actions, Azure DevOps)
• REST API Development & Integration
• Cloud Platforms (AWS/Azure)
• Git Version Control
• Test Automation (PyTest, Robot Framework, Selenium preferred)

Key Responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement automation solutions to improve operational efficiency.
• Build reusable automation frameworks and scripts.
• Optimize Spark jobs for performance and scalability.
• Develop and consume REST APIs for system integrations.
• Collaborate with data engineers, developers, and business stakeholders.
• Implement CI/CD pipelines and automation best practices.
• Troubleshoot production issues and provide timely resolutions.
• Ensure code quality through reviews, testing, and documentation.
• Follow Agile development methodologies.

Required Qualifications

• 6 to 10 years of IT experience.
• Strong hands-on experience in Python and PySpark.
• Experience in automation scripting and framework development.
• Strong knowledge of SQL and data processing concepts.
Experience with AWS or Azure cloud services.
• Familiarity with CI/CD tools and DevOps practices.
• Excellent problem-solving and communication skills.

Preferred Skills

• Databricks
• Airflow
• Docker & Kubernetes
• Selenium Automation
• Terraform
• Machine Learning pipeline automation