Sr. GCP Data Engineer (Job-ID 3301)

KANINI; United States; Remote; USD 140,000 - 200,000; Full time

Job summary

KANINI is seeking a Senior Data Engineer with deep GCP expertise to design scalable data pipelines and implement Medallion Architecture for enterprise-grade data solutions.

The role requires hands-on work with BigQuery, PySpark, Dataflow, Airflow, GCS, and CI/CD practices, plus strong SQL and data modeling skills. Remote position for the United States with governance and security emphasis.

Qualifications

• Experience with Google Cloud Platform (GCP) in enterprise data environments.
• Strong SQL skills including advanced joins, window functions, and optimization.
• Hands-on experience building scalable batch and real-time data pipelines.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines on GCP.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data processing.
• Build high-performance data transformations using Python and PySpark.
• Develop and optimize complex SQL queries for analytical workloads.
• Work with BigQuery for large-scale data processing and performance tuning.
• Develop and deploy pipelines using Cloud Dataflow.
• Orchestrate workflows with Cloud Composer (Airflow).
• Manage data storage and lifecycle using Google Cloud Storage (GCS).
• Implement version control and CI/CD pipelines using Git-based tools.
• Ensure data security, governance, and access control using GCP IAM.
• Optimize data solutions for performance, scalability, reliability, and cost-efficiency.

Skills

GCP

BigQuery

Medallion Architecture

Python

PySpark

Java

SQL

Dataflow

Airflow

GCS

CI/CD

IAM

Tools

Git

Cloud Composer

Job description

This is a remote position. KANINI is seeking a highly skilled Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands‑on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise‑grade data solutions. This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.

Key Responsibilities

• Design, develop, and maintain scalable batch and real‑time data pipelines on GCP
• Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
• Build high‑performance data transformations using Python and PySpark
• Develop and optimize complex SQL queries for analytical workloads
• Work extensively with BigQuery for large‑scale data processing and performance tuning
• Develop and deploy pipelines using Cloud Dataflow
• Orchestrate workflows using Cloud Composer (Apache Airflow)
• Manage data storage and lifecycle using Google Cloud Storage (GCS)
• Implement version control and CI/CD pipelines using Git‑based tools
• Ensure data security, governance, and access control using GCP IAM
• Optimize data solutions for performance, scalability, reliability, and cost‑efficiency

Required Skills & Experience

• Strong hands‑on experience with Google Cloud Platform (GCP)
• Expertise in BigQuery (partitioning, clustering, query optimization)
• Proven experience implementing Medallion Data Architecture
• Strong programming skills in Python and PySpark
• Hands‑on exposure on Java
• Advanced proficiency in SQL (complex joins, window functions, performance tuning)
• Hands‑on experience with Cloud Dataflow
• Experience with Cloud Composer (Airflow) for orchestration
• Experience working with Google Cloud Storage (GCS)
• Knowledge of version control systems (Git) and CI/CD practices
• Strong understanding of GCP IAM and cloud security best practices

Preferred Qualifications

• Experience working with large‑scale enterprise data platforms
• Knowledge of data warehousing and data lake concepts
• Familiarity with real‑time streaming frameworks
• Experience in data governance and data quality frameworks
• Exposure to