GCP Platform Engineer: Airflow, Terraform & CI/CD

TechDigital Group

Virginia

On-site

USD 120,000 - 180,000

Full time

14 days+

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

TechDigital Group seeks an experienced Cloud/Data Engineer to enhance Dataflow/Dataproc pipelines and manage GCP Composer environments. You will troubleshoot Airflow DAGs, optimize performance, and maintain infrastructure across GCS, BigQuery, Dataflow, Dataproc, Functions, and Run.

Strong SQL/PLSQL, Java or Python, and Terraform skills are essential. You will collaborate with DevOps to implement CI/CD pipelines using Jenkins, design APIs with Spring Boot, and ensure security, compliance, and

Qualifications

  • Good understanding of GCP Composer (Apache Airflow) environments.
  • Manage Airflow DAGs and troubleshoot Airflow-related issues.
  • Monitor and optimize Composer performance and scalability.
  • Maintain GCP infrastructure including services like GCS, BigQuery, Cloud Dataflow, Cloud Dataproc, Cloud Functions, Cloud Run.
  • Have good understanding of IAM and service account permissions, Secret Manager.
  • Monitor GCP usage and quotas, and proactively prevent quota-related issues.
  • Set up and manage alerting, logging, and monitoring using Stackdriver (Cloud Monitoring & Logging).
  • DB design experience (with Excellent RDBS-PLSQL knowledge).
  • Design Springboot APIs.
  • Analyze, design, create Dockerfile.
  • Knowledge on Angular (Good to Have).
  • Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources.
  • Collaborate with DevOps and development teams to manage CI/CD pipelines using Jenkins and other automation tools.
  • Ensure security, compliance, and cost-efficiency across the cloud environment.
  • Document system configurations, procedures, and best practices.
  • Proficient in either Java or Python programming languages.
  • Knowledge on DNS and SSO configuration setup.

Responsibilities

  • Enhance and maintain Dataflow/Dataproc data ingestion pipelines (batch & streaming).
  • Oversee GCP Composer environments and Airflow DAG management.
  • Troubleshoot Airflow issues and optimize performance at scale.
  • Maintain and secure GCP infrastructure including GCS, BigQuery, Dataflow, Dataproc, Functions, and Run.
  • Manage IAM, Secret Manager, DNS and SSO configurations.
  • Set up alerting, logging, and monitoring with Cloud Monitoring/Stackdriver.
  • Design and refine DB architectures and SQL routines.
  • Develop Spring Boot APIs and related backend services.
  • Create Dockerfiles and containerized deployment artifacts.
  • Collaborate with DevOps to implement robust CI/CD pipelines with Jenkins.
  • Ensure cost-efficiency and compliance across cloud resources.
  • Document configurations, runbooks, and best practices.

Skills

Airflow
CI/CD
Java/Python
SQL/PLSQL

Tools

Terraform
Docker
Jenkins
Cloud Monitoring

Job description

Note - More on Composer/Airflow Support & Maintenance, Expert knowledge in Terraform, CI/CD , GCP Infrastructure

Role Overview:
ix and enhance Dataflow and/or Dataproc Data ingestion pipelines (batch & Streaming) .
  • Have Good Understanding of GCP Composer (Apache Airflow) environments.
  • Manage Airflow DAGs and troubleshoot Airflow-related issues.
  • Monitor and optimize Composer performance and scalability.
  • Maintain GCP infrastructure including services like:
    • Google Cloud Storage (GCS)
    • BigQuery (BQ)
    • Cloud Dataflow ( Apache Beam )
    • Cloud Dataproc
    • Cloud Functions
    • Cloud Run
  • Have Good Understanding of:
    • IAM and service account permissions
    • Secret Manager
  • Monitor GCP usage and quotas, and proactively prevent quota-related issues.
  • Set up and manage alerting, logging, and monitoring using tools like Stackdriver (Cloud Monitoring & Logging).
  • DB design experience (with Excellent RDBS-PLSQL knowledge )
    Design Springboot APIs
  • Analyze, design, create Dockerfile
  • Knowledge on Angular ( Good to Have )
  • Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources.
  • Collaborate with DevOps and development teams to manage CI/CD pipelines using Jenkins and other automation tools.
  • Ensure security, compliance, and cost-efficiency across the cloud environment
  • Document system configurations, procedures, and best practices.
  • Proficient in either Java or Python programming languages.
  • Knowledge on DNS and SSO configuration setup
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

GCP Platform Engineer
GCP Platform Engineer

TechDigital Group • Virginia

On-site
USD 120,000 - 180,000
GCP Dataflow/Dataproc Engineer (Infra & CI/CD)
GCP Dataflow/Dataproc Engineer (Infra & CI/CD)

TechDigital Group • Virginia

On-site
USD 80,000 - 110,000
GCP Cloud Engineer
GCP Cloud Engineer

TechDigital Group • St. Louis (MO)

On-site
USD 90,000 - 120,000
GCP Cloud Engineer - Data Pipelines & Infra Expert
GCP Cloud Engineer - Data Pipelines & Infra Expert

TechDigital Group • St. Louis (MO)

On-site
USD 90,000 - 120,000
GCP Developer
GCP Developer

TechDigital Group • Virginia

On-site
USD 80,000 - 110,000
Senior Azure Data Factory Architect – Cloud & Data
Senior Azure Data Factory Architect – Cloud & Data

TechDigital Group • United States

On-site
USD 120,000 - 180,000
GCP Data Engineer: Scalable Pipelines, IAM & Airflow
GCP Data Engineer: Scalable Pipelines, IAM & Airflow

TechDigital Group • Seattle (WA)

On-site
USD 130,000 - 160,000
GCP Cloud Engineer
GCP Cloud Engineer

Rapinno • Florida

On-site
USD 130,000 - 160,000
GCP Data Engineer
GCP Data Engineer

TechDigital Group • Seattle (WA)

On-site
USD 130,000 - 160,000
GCP Architect
GCP Architect

Compunnel Inc. • Mount Laurel Township (NJ)

On-site
USD 120,000 - 150,000