GCP Platform Engineer

TechDigital Group

Virginia

On-site

USD 120,000 - 180,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

TechDigital Group seeks an experienced Cloud/Data Engineer to enhance Dataflow/Dataproc pipelines and manage GCP Composer environments. You will troubleshoot Airflow DAGs, optimize performance, and maintain infrastructure across GCS, BigQuery, Dataflow, Dataproc, Functions, and Run.

Strong SQL/PLSQL, Java or Python, and Terraform skills are essential. You will collaborate with DevOps to implement CI/CD pipelines using Jenkins, design APIs with Spring Boot, and ensure security, compliance, and

Qualifications

  • Good understanding of GCP Composer (Apache Airflow) environments.
  • Manage Airflow DAGs and troubleshoot Airflow-related issues.
  • Monitor and optimize Composer performance and scalability.
  • Maintain GCP infrastructure including services like GCS, BigQuery, Cloud Dataflow, Cloud Dataproc, Cloud Functions, Cloud Run.
  • Have good understanding of IAM and service account permissions, Secret Manager.
  • Monitor GCP usage and quotas, and proactively prevent quota-related issues.
  • Set up and manage alerting, logging, and monitoring using Stackdriver (Cloud Monitoring & Logging).
  • DB design experience (with Excellent RDBS-PLSQL knowledge).
  • Design Springboot APIs.
  • Analyze, design, create Dockerfile.
  • Knowledge on Angular (Good to Have).
  • Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources.
  • Collaborate with DevOps and development teams to manage CI/CD pipelines using Jenkins and other automation tools.
  • Ensure security, compliance, and cost-efficiency across the cloud environment.
  • Document system configurations, procedures, and best practices.
  • Proficient in either Java or Python programming languages.
  • Knowledge on DNS and SSO configuration setup.

Responsibilities

  • Enhance and maintain Dataflow/Dataproc data ingestion pipelines (batch & streaming).
  • Oversee GCP Composer environments and Airflow DAG management.
  • Troubleshoot Airflow issues and optimize performance at scale.
  • Maintain and secure GCP infrastructure including GCS, BigQuery, Dataflow, Dataproc, Functions, and Run.
  • Manage IAM, Secret Manager, DNS and SSO configurations.
  • Set up alerting, logging, and monitoring with Cloud Monitoring/Stackdriver.
  • Design and refine DB architectures and SQL routines.
  • Develop Spring Boot APIs and related backend services.
  • Create Dockerfiles and containerized deployment artifacts.
  • Collaborate with DevOps to implement robust CI/CD pipelines with Jenkins.
  • Ensure cost-efficiency and compliance across cloud resources.
  • Document configurations, runbooks, and best practices.

Skills

Airflow
CI/CD
Java/Python
SQL/PLSQL

Tools

Terraform
Docker
Jenkins
Cloud Monitoring

Job description

Note - More on Composer/Airflow Support & Maintenance, Expert knowledge in Terraform, CI/CD , GCP Infrastructure

Role Overview

ix and enhance Dataflow and/or Dataproc Data ingestion pipelines (batch & Streaming).

  • Have Good Understanding of GCP Composer (Apache Airflow) environments.
  • Manage Airflow DAGs and troubleshoot Airflow-related issues.
  • Monitor and optimize Composer performance and scalability.
  • Maintain GCP infrastructure including services like:
    • Google Cloud Storage (GCS)
    • BigQuery (BQ)
    • Cloud Dataflow (Apache Beam)
    • Cloud Dataproc
    • Cloud Functions
    • Cloud Run
  • Have Good Understanding of:
    • IAM and service account permissions
    • Secret Manager
  • Monitor GCP usage and quotas, and proactively prevent quota-related issues.
  • Set up and manage alerting, logging, and monitoring using tools like Stackdriver (Cloud Monitoring & Logging).
  • DB design experience (with Excellent RDBS-PLSQL knowledge)
  • Design Springboot APIs
  • Analyze, design, create Dockerfile
  • Knowledge on Angular (Good to Have)
  • Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources.
  • Collaborate with DevOps and development teams to manage CI/CD pipelines using Jenkins and other automation tools.
  • Ensure security, compliance, and cost-efficiency across the cloud environment.
  • Document system configurations, procedures, and best practices.
  • Proficient in either Java or Python programming languages.
  • Knowledge on DNS and SSO configuration setup.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

GCP Platform Engineer: Airflow, Terraform & CI/CD
GCP Platform Engineer: Airflow, Terraform & CI/CD

TechDigital Group • Virginia

On-site
USD 120,000 - 180,000
GCP Developer
GCP Developer

TechDigital Group • Virginia

On-site
USD 80,000 - 110,000
GCP Dataflow/Dataproc Engineer (Infra & CI/CD)
GCP Dataflow/Dataproc Engineer (Infra & CI/CD)

TechDigital Group • Virginia

On-site
USD 80,000 - 110,000
GCP Cloud Engineer
GCP Cloud Engineer

TechDigital Group • St. Louis (MO)

On-site
USD 90,000 - 120,000
GCP Cloud Engineer - Data Pipelines & Infra Expert
GCP Cloud Engineer - Data Pipelines & Infra Expert

TechDigital Group • St. Louis (MO)

On-site
USD 90,000 - 120,000
Senior Azure Data Factory Architect – Cloud & Data
Senior Azure Data Factory Architect – Cloud & Data

TechDigital Group • United States

On-site
USD 120,000 - 180,000
GCP Cloud Engineer
GCP Cloud Engineer

Rapinno • Florida

On-site
USD 130,000 - 160,000
GCP- Data Engineer
GCP- Data Engineer

TechDigital Group • Chicago (IL)

On-site
USD 100,000 - 120,000
GCP Data Engineer: Scalable Pipelines, IAM & Airflow
GCP Data Engineer: Scalable Pipelines, IAM & Airflow

TechDigital Group • Seattle (WA)

On-site
USD 130,000 - 160,000
GCP Data Engineer
GCP Data Engineer

TechDigital Group • Raleigh (NC)

On-site
USD 100,000 - 130,000