Python, Pyspark, GCP_Virtual drive in Week days

Tata Consultancy Services Limited; Hyderabad; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Tata Consultancy Services Limited in Hyderabad is seeking an experienced Data Engineer specializing in Python, PySpark and cloud data platforms to build scalable data pipelines.

You will design ETL processes, optimize PySpark workloads, ensure data quality, and collaborate with cross-functional teams to meet data requirements. The role emphasizes strong SQL skills and production-ready code.

Qualifications

• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.
• Exposure to cloud computing environments.

Responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like AWS, Azure, or GCP.
• Implement data storage solutions and manage large-scale datasets.

Skills

Python

PySpark

Spark

REST APIs

JSON/XML

Cloud computing

Distributed computing

Data engineering

SQL

Databases

Parallel processing

Big Data

Hadoop

Hive

Kafka

Data serialization

Tools

Hadoop

Hive

Kafka

Job description

Role: Python, Pyspark ,GCP(Data engineer)

Experience : 6 to 10 Yrs

Location : Hyderabad

Role & responsibilities

Desired Competencies (Technical/Behavioral Competency)

Must-Have**

(Ideally should not be more than 3-5)

• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.
• Exposure to cloud computing environments.

Good-to-Have

• 5+ years of experience in Python and PySpark development.
• Experience with data warehousing and data lakes.
• Knowledge of machine learning libraries (e.g., MLlib) is a plus.
• Strong problem-solving and debugging skills.
• Excellent communication and collaboration abilities.

SN

Responsibility of / Expectations from the Role

1

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like AWS, Azure, or GCP.
• Implement data storage solutions and manage large-scale datasets.

Preferred candidate profile