GCP Data engineer with Pyspark

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services in Bengaluru, India, seeks a seasoned Data Engineer to design and optimize data pipelines on a cloud-first platform. You will contribute to building scalable data systems and work closely with analytics teams.

Ideal candidates have 5–10 years of experience with PySpark, Python, SQL, and BigQuery, and will work with Airflow/Cloud Composer, CI/CD, and data governance to ensure reliable, secure data solutions.

Qualifications

• 5–10 years of experience in data engineering with strong GCP and PySpark.
• Hands-on experience developing scalable data pipelines using Python, PySpark, and SQL.
• Strong knowledge of GCP services such as BigQuery, Cloud Storage, Dataproc, Dataflow, and Pub/Sub.
• Experience designing and implementing ETL/ELT pipelines for large-scale data processing.
• Proficiency in Apache Spark/PySpark for distributed data transformation.

Responsibilities

• Develop scalable data pipelines using Python, PySpark, and SQL.
• Build and optimize data models and storage on GCP services.
• Collaborate with analytics teams to ensure data quality and governance.
• Design and implement ETL/ELT processes for large-scale data workloads.
• Implement cloud security best practices and CI/CD for data solutions.

Skills

Data Engineering

PySpark

Python

SQL

GCP

ETL/ELT

Apache Spark

BigQuery

Cloud Security

CI/CD

Airflow

Tools

BigQuery

Cloud Storage

Dataproc

Dataflow

Pub/Sub

Git

Cloud Composer

Airflow

Job description

Mode of Interview: Virtual

Responsibilities :-

• 5–10 years of experience in Data Engineering, with strong expertise in GCP and PySpark.
• Hands-on experience developing scalable data pipelines using Python, PySpark, and SQL.
• Strong knowledge of GCP services such as BigQuery, Cloud Storage, Dataproc, Dataflow, and Pub/Sub.
• Experience designing and implementing ETL/ELT pipelines for large-scale data processing.
• Proficiency in Apache Spark/PySpark for distributed data transformation and optimization.
• Strong experience with BigQuery, including SQL development and query performance optimization.
• Experience integrating data from multiple structured and unstructured data sources.
• Familiarity with workflow orchestration tools such as Cloud Composer/Airflow.
• Knowledge of Git, CI/CD, data quality, monitoring, and cloud security best practices.
• Strong analytical and troubleshooting skills with experience delivering reliable GCP-based data solutions.