GCP Developer

TechDigital Group; Virginia (MN); On-site; USD 80,000 - 110,000; Full time

Job summary

TechDigital Group is seeking a talented individual to enhance data ingestion pipelines using GCP services. The ideal candidate will manage Airflow workflows, optimize performance, and ensure security across cloud applications.

Responsibilities include working with Docker and Terraform to provision resources, collaborating closely with DevOps teams, and designing Springboot APIs. Proficiency in Java or Python and knowledge of CI/CD are essential for success in this role.

Qualifications

• Good understanding of GCP Composer for troubleshooting and performance monitoring.
• Experience in managing Airflow-related issues and optimizing pipeline performance.
• Proficient in designing infrastructure as code and managing GCP resources efficiently.

Responsibilities

• Design and implement data ingestion pipelines using GCP services.
• Collaborate with teams to manage CI/CD processes effectively.
• Ensure cloud environment security, compliance, and cost efficiency.

Skills

GCP Composer (Apache Airflow)

DB design

Springboot APIs

Java

Python

Terraform

Docker

Angular

CI/CD pipelines

Tools

Stackdriver (Cloud Monitoring & Logging)

Jenkins

Job description

Enhance Dataflow and/or Dataproc Data ingestion pipelines (batch & Streaming)

• Have Good Understanding of GCP Composer (Apache Airflow) environments.
• Manage Airflow DAGs and troubleshoot Airflow-related issues.
• Monitor and optimize Composer performance and scalability.
• Maintain GCP infrastructure including services like:
  • Google Cloud Storage (GCS)
  • BigQuery (BQ)
  • Cloud Dataflow (Apache Beam)
  • Cloud Dataproc
  • Cloud Functions
  • Cloud Run
• Have Good Understanding of:
  • IAM and service account permissions
  • Secret Manager
• Monitor GCP usage and quotas, and proactively prevent quota-related issues.
• Set up and manage alerting, logging, and monitoring using tools like Stackdriver (Cloud Monitoring & Logging).
• DB design experience (with Excellent RDBS-PLSQL knowledge).
• Design Springboot APIs.
• Analyze, design, create Dockerfile.
• Knowledge on Angular (Good to Have).
• Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources.
• Collaborate with DevOps and development teams to manage CI/CD pipelines using Jenkins and other automation tools.
• Ensure security, compliance, and cost-efficiency across the cloud environment.
• Document system configurations, procedures, and best practices.
• Proficient in either Java or Python programming languages.
• Knowledge on DNS and SSO configuration setup.