GCP Data engineer with Python/Pyspark,Airflow,Dataproc expertise

Tredence; Bengaluru; On-site; INR 2,500,000 - 3,500,000; Full time

Benefits offered by this job

Health insurance

Job summary

Tredence is seeking a Senior GCP Data Engineer to design and optimize enterprise data pipelines and architectures. You will lead batch and real-time processing using GCP, Python, Airflow, Dataproc, and PySpark to deliver scalable data solutions.

The role requires 6–9 years IT experience with 3+ years in GCP, and expertise in BigQuery, GCS, Pub/Sub, and SQL. Strong problem-solving and collaboration across teams are expected.

Qualifications

• 6–9 years IT experience with 3+ years in GCP.
• Hands-on with BigQuery, GCS, Pub/Sub, Cloud Functions.
• Proficiency in Python and PySpark for data pipelines.

Responsibilities

• Design, build, and optimize batch and real-time data pipelines.
• Develop data architectures using GCP, PySpark, and Airflow.
• Tune and scale distributed processing with Dataproc.
• Manage data warehouses with BigQuery and efficient querying.
• Implement Medallion Architecture or data lake approaches.

Education

Google Certified Professional Data Engineer

Tools

BigQuery

Cloud Storage

Pub/Sub

Cloud Functions

Python

PySpark

Dataproc

Airflow

Cloud Composer

SQL

Job description

Job Description: Senior GCP Data Engineer

Job Title: Senior Data Engineer - GCP & Big Data Architecture

Experience Level: 6 to 9 Years of overall IT experience (with 3+ years of dedicated GCP)

Location: Bangalore / Chennai / Pune / Gurgram / Kolkata / Hyderabad

Role Overview

We are looking for a highly skilled Senior GCP Data Engineer with 6 to 9 years of experience to design, build, and optimize our next-generation enterprise data platform. In this role, you will lead the development of robust, scalable batch and real-time data pipelines. You will be the subject matter expert for our big data infrastructure, translating complex business needs into clean, production-ready architectures leveraging GCP, Python, Airflow, Dataproc, and PySpark.

Key Responsibilities

• Data Pipeline Development: Design, build, and maintain enterprise-grade ETL/ELT pipelines using Python and PySpark to process massive, unstructured, and structured datasets
• Orchestration & Workflow Management: Author, schedule, and monitor complex Directed Acyclic Graphs (DAGs) using Apache Airflow (Google Cloud Composer) to orchestrate modern data workflows
• Big Data Processing: Architect and tune scalable distributed computing environments using GCP Dataproc (Hadoop/Spark clusters) to optimize large-scale batch processing
• Data Warehousing: Manage and optimize data schemas, indexing, partitioning, and cost-efficient querying inside Google BigQuery.
• Data Architecture: Implement and scale modern frameworks like the Medallion Architecture (Bronze, Silver, Gold layers) or data lakes ensuring high performance

Required Technical Skills & Qualifications

Core Technical Stack (Must Haves)

• Google Cloud Platform (GCP): In-depth, hands-on experience with core data components including BigQuery, Cloud Storage (GCS), Pub/Sub, and Cloud Functions.
• Programming Language: Expert proficiency in Python for general programming and data engineering tasks.
• Big Data Processing: Deep technical expertise in PySpark / Apache Spark for distributed processing, including performance tuning and configurations on GCP Dataproc.
• Orchestration: Extensive experience designing, writing, and debugging Apache Airflow / Cloud Composer DAGs.
• Advanced SQL: Superior skills in writing, optimizing, and tuning highly complex SQL queries and analytic functions.

Preferred / Good to Have Skills

• An active Google Certified Professional Data Engineer certification