Data Engineer- Python Pyspark GCP

Tata Consultancy Services; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer to develop and maintain scalable data pipelines using Python and PySpark. You will design ETL processes, optimize PySpark applications for performance, and ensure data integrity across the lifecycle.

You will collaborate with cross-functional teams, write clean code, conduct reviews, and integrate with cloud platforms such as GCP, AWS, or Azure. 4–8+ years of experience and deep Big Data knowledge are expected.

Qualifications

• 4 to 8+ years of experience with Python and PySpark.
• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding of data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.

Responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL processes.
• Optimize and troubleshoot PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like GCP, AWS or Azure.

Skills

Python

PySpark

Apache Spark

Big Data

SQL

Distributed computing

Data engineering

REST APIs

JSON/XML

GCP

AWS

Azure

Hadoop

Hive

Kafka

Tools

Apache Hadoop

Hive

Kafka

Job description

Role & responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like GCP, AWS or Azure.

Implement data storage solutions and manage large-scale datasets.

Preferred candidate profile

• 4 to 8 + Years of experience using Python and Pyspark.
• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and elational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding of data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.

Exposure to GCP services and cloud computing environments