Python_PySpark Developer on GCP

Tata Consultancy Services; Hyderabad; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Tata Consultancy Services in Hyderabad invites experienced data engineering professionals to design and optimize scalable data pipelines. Proficiency in Python, PySpark, Spark, and Big Data tools is essential to deliver robust ETL and analytics solutions.

You will collaborate across teams, implement REST APIs, ensure data quality, and leverage cloud platforms like GCP/AWS/Azure to process large datasets efficiently. 6-8 years of hands-on experience is expected for this role.

Qualifications

• 6–8 years of experience using Python and PySpark.
• Strong proficiency in Python programming.
• Hands-on PySpark and Apache Spark experience.
• Knowledge of Hadoop, Hive, Kafka, and other Big Data tech.
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Experience with REST APIs, JSON/XML and data serialization.
• Exposure to cloud platforms (GCP, AWS or Azure).

Responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL processes.
• Optimize PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like GCP, AWS or Azure.
• Implement data storage solutions and manage large-scale datasets.

Skills

Python

PySpark

Apache Spark

Big Data

SQL

Distributed computing

REST APIs

Job description

• 6to 8 + Years of experience using Python and Pyspark.
• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Experience with REST APIs, JSON/XML, and data serialization.
• Exposure to GCP services and cloud computing environments.

Key Responsibilities*

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well‑documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like GCP, AWS or Azure.
• Implement data storage solutions and manage large‑scale datasets.