Python Developer with PySpark/Spark expertise

Tata Consultancy Services; Bengaluru, Mumbai; On-site; INR 1,500,000 - 2,700,000; Full time

Job summary

Tata Consultancy Services is seeking a Python Developer with PySpark expertise to design, develop, and optimize scalable data pipelines. You will work with Python, Spark, and SQL across distributed data systems in a data-intensive environment.

The role emphasizes clean code, SOLID principles, system design, and collaboration for production-grade solutions with observability and operational excellence.

Qualifications

• 5+ years of professional experience in Python development with a focus on data engineering.
• Hands-on expertise with Apache Spark and Big Data processing.
• Strong SQL skills and experience with both relational and distributed data systems.
• Solid understanding of software engineering principles, clean code, and design patterns.
• Experience in system design and architecture for scalable, data-intensive applications.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Python and Apache Spark.
• Build high-performance data workflows with pushdown optimization, streaming, replication, partitioning, and clustering techniques.
• Evaluate and integrate modern data platforms and tools into our enterprise architecture.
• Engineer and manage feature pipelines to support real-time fraud detection systems.
• Design data models and processing strategies that align with distributed system principles, ensuring scalability, consistency, and performance across large-scale environments.
• Develop solutions that are production-ready, maintainable, and built with observability and operational excellence in mind.
• Apply clean code practices, SOLID principles, and architecture patterns to deliver robust and extensible systems.
• Participate in code reviews, testing, and deployment activities.
• Contribute to architectural decisions and continuous improvement initiatives.

Skills

Python development

Apache Spark

Big Data processing

SQL skills

Software engineering principles

Design patterns

System design

Communication & collaboration

Analytical problem-solving

Tools

Snowflake

Databricks

Kafka

Redis

Airflow

Jenkins

Git

OpenTelemetry

Grafana

Loki

Tempo

Kubernetes

Helm

GitOps

Job description

Greetings from TCS!!

TCS is hiring for: Python Developer with PySpark expertise

Desired Experience: 5 Yrs to 8 Yrs

Job Locations: Mumbai / Bangalore

Mode Of Interview: Virtual

Mode Of Work: 5-day WFO

Job Description:

Key Responsibilities:

• Design, develop, and optimize scalable data pipelines using Python and Apache Spark.
• Build high-performance data workflows with pushdown optimization, streaming, replication, partitioning, and clustering techniques.
• Evaluate and integrate modern data platforms and tools into our enterprise architecture.
• Engineer and manage feature pipelines to support real-time fraud detection systems.
• Design data models and processing strategies that align with distributed system principles, ensuring scalability, consistency, and performance across large-scale environments.
• Develop solutions that are production-ready, maintainable, and built with observability and operational excellence in mind.
• Apply clean code practices, SOLID principles, and architecture patterns to deliver robust and extensible systems.
• Participate in code reviews, testing, and deployment activities.
• Contribute to architectural decisions and continuous improvement initiatives.

Required Skills:

• 5+ years of professional experience in Python development with a strong focus on data engineering.
• Hands-on expertise with Apache Spark and Big Data processing.
• Strong SQL skills and experience with both relational and distributed data systems.
• Solid understanding of software engineering principles, clean code, and design patterns.
• Experience in system design and architecture for scalable, data-intensive applications.
• Ability to evaluate new technologies and recommend solutions aligned with enterprise needs.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration abilities.

Beneficial Skills (Nice to have)

• Exposure to cloud platforms such as Snowflake, Databricks, or similar.
• Familiarity with Kafka, Redis, Airflow, Jenkins, and Git.
• Understanding of observability practices and tools like Open Telemetry, Grafana, Loki, or Tempo, with a mindset for embedding observability into system design.
• Awareness of Kubernetes, Helm, GitOps, and containerization concepts.
• Background in real-time systems, or financial services domains.