GCP Cloud Engineer

TechDigital Group; St. Louis (MO); On-site; USD 90,000 - 120,000; Full time

Job summary

A technology services firm is seeking a skilled cloud developer to manage GCP Composer environments and optimize data ingestion pipelines. The role includes enhancing Dataflow and Dataproc processes, managing Airflow DAGs, and implementing infrastructure as code with Terraform. The ideal candidate is proficient in Java or Python, has strong troubleshooting skills, and communicates effectively with both DevOps and development teams. Comprehensive knowledge of GCP services and tools is essential.

Qualifications

• Proficient in either Java or Python programming languages.
• Excellent problem‑solving and troubleshooting skills.
• Strong communication and documentation abilities.

Responsibilities

• Fix and enhance Dataflow and Dataproc data ingestion pipelines.
• Manage Airflow DAGs and troubleshoot Airflow issues.
• Monitor and optimize Composer performance.

Skills

GCP Composer (Apache Airflow)

Docker

Java

Python

Spring Boot

Terraform

Angular

IAM and service account permissions

Tools

Jenkins

Google Cloud Storage (GCS)

BigQuery (BQ)

Cloud Dataflow (Apache Beam)

Cloud Dataproc

Cloud Functions

Cloud Run

Stackdriver (Cloud Monitoring & Logging)

Job description

Mandatory skills:

• GCP Composer (Apache Airflow)

Job Description:

• Fix and enhance Dataflow and/or Dataproc data ingestion pipelines (batch & streaming).
• Have a good understanding of GCP Composer (Apache Airflow) environments.
• Manage Airflow DAGs and troubleshoot Airflow‑related issues.
• Monitor and optimize Composer performance and scalability.
• Maintain GCP infrastructure including services like:
  • Google Cloud Storage (GCS)
  • BigQuery (BQ)
  • Cloud Dataflow (Apache Beam)
  • Cloud Dataproc
  • Cloud Functions
  • Cloud Run
• Have a good understanding of:
  • IAM and service account permissions
  • Secret Manager
• Monitor GCP usage and quotas, and proactively prevent quota-related issues.
• Set up and manage alerting, logging, and monitoring using tools like Stackdriver (Cloud Monitoring & Logging).
• DB design experience (excellent RDBS‑PLSQL knowledge).
• Design Spring Boot APIs.
• Analyze, design, create Dockerfile.
• Knowledge on Angular (good to have).
• Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources.
• Collaborate with DevOps and development teams to manage CI/CD pipelines using Jenkins and other automation tools.
• Ensure security, compliance, and cost‑efficiency across the cloud environment.
• Document system configurations, procedures, and best practices.
• Proficient in either Java or Python programming languages.
• Excellent problem‑solving and troubleshooting skills.
• Strong communication and documentation abilities.
• Knowledge on DNS and SSO configuration setup.
• Knowledge on CDC Pipelines/Application (good to have).