DevOps Lead (GCP, AWS) – Data Engineering

Harmony Data Integration Technologies; India; Hybrid; INR 2,500,000 - 4,000,000; Full time

Job summary

Harmony Data Integration Technologies is seeking a DevOps Lead to design, build, and operate scalable cloud and data platforms, primarily on GCP with AWS as secondary. This hybrid role covers platform engineering, Terraform, Kubernetes, CI/CD, security, observability, and data pipelines.

The ideal candidate combines data engineering experience with DevOps/Platform Engineering, wearing infrastructure and automation hats while managing production data workloads.

Qualifications

• -8+ years of total professional experience in software, data, or infrastructure engineering.
• -Minimum 4 years in a dedicated DevOps / SRE / Platform Engineering role.
• -Strong hands-on data engineering experience with pipelines, warehousing, and orchestration.
• -Proficient in GCP and solid AWS experience; IaC with Terraform.
• -Kubernetes, Docker, and Helm in production; CI/CD with Jenkins, GitLab CI, GitHub Actions.
• -Strong Python for maintainable pipelines and Bash; SQL with cloud data warehouse (BigQuery).
• -Experience with Airflow / Cloud Composer and data workflows (Dataflow/Beam, Spark, dbt).
• -Experience with streaming (Pub/Sub, Kafka, Kinesis) and SonarQube integration.
• -Cloud security basics (IAM, VPC, firewalls) and secrets management; secure SDLC.
• -Excellent written and verbal communication.

Responsibilities

• Design, implement, and manage cloud infrastructure across GCP and AWS using Terraform.
• Deploy and manage Kubernetes clusters; use Helm and ArgoCD for workloads.
• Build secure, scalable CI/CD pipelines with GitHub Actions, Jenkins, GitLab CI, ArgoCD.
• Enforce container security, image hardening, and runtime protection.
• Integrate quality gates in CI/CD with tests, coverage, and SonarQube.
• Maintain secure CI/CD: secrets management, least privilege, hardened agents.
• Apply cloud security best practices: IAM roles, network segmentation, encryption.
• Embed DevSecOps with policy-as-code and pipeline scanning.
• Contribute to system architecture, monitoring, and observability.
• Participate in incident response, RCAs, and post-incident reviews.
• Own data platform infrastructure: ETL/ELT pipelines, orchestration, and cost management.
• Collaborate with analytics/ML teams to ensure secure, governed data access.

Skills

GCP

AWS

Terraform

Kubernetes

Docker

Helm

ArgoCD

CI/CD

Python

Airflow

Cloud Composer

BigQuery

Dataflow/Apache Beam

Snowflake/Redshift

Pub/Sub/Kafka/Kinesis

SonarQube

IAM/VPC/Security

Secrets Management

Tools

Jenkins

GitLab CI

GitHub Actions

Airflow

Cloud Composer

Terraform

ArgoCD

Job description

We are looking for a highly skilled and motivated DevOps Lead to design, build, and operate scalable, secure, and highly available cloud and data platforms, primarily on Google Cloud Platform (GCP) with Amazon Web Services (AWS) as a secondary environment.

This is a hybrid platform role. Alongside core DevOps ownership - Terraform, Kubernetes, CI/CD, cloud security, and observability - you will also contribute substantially to our data engineering function: building and operating production Python data pipelines, orchestrating batch and streaming workflows, and owning the infrastructure that sits underneath our data platform.

The ideal candidate has spent several years building data pipelines as a data engineer and has since moved into a DevOps / SRE / Platform Engineering role, or has run platform engineering for data-intensive products. We are looking for someone who can credibly wear both hats: infrastructure and automation first, data engineering a close second.

DevOps & Cloud Infrastructure Responsibilities:

• Design, implement, and manage cloud infrastructure across GCP and AWS using Terraform and other Infrastructure-as-Code practices.
• Deploy and maintain Kubernetes clusters and manage application workloads using Helm and/or ArgoCD.
• Build and enhance secure, scalable CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI, and ArgoCD.
• Implement best practices in containerization using Docker, including image hardening and container runtime security.
• Integrate quality gates in CI/CD, including automated testing, code coverage, and SonarQube configuration.
• Ensure secure CI/CD processes, including secrets management, least-privilege access, and hardened build agents.
• Implement and maintain cloud security best practices: IAM roles and policies, network segmentation, encryption, and vulnerability management.
• Collaborate with security teams to embed DevSecOps practices (shift-left security, scanning in pipelines, policy-as-code).
• Contribute to system architecture, monitoring, and observability using modern tools and practices.
• Participate in incident response, post-incident reviews, and root cause analysis.

Data Engineering & Data Platform Responsibilities:

• Design, build, and maintain production-grade ETL/ELT pipelines in Python for both batch and streaming workloads.
• Build and operate workflow orchestration using Airflow / Cloud Composer, including scheduling, dependency management, retries, and alerting.
• Develop and optimise data models and transformations in BigQuery (and equivalents such as Snowflake or Redshift), with attention to partitioning, clustering, and query cost.
• Own the infrastructure and deployment lifecycle of data services - provisioning, autoscaling, upgrades, and cost management.
• Implement data quality, validation, lineage, and observability checks so pipeline failures and data drift are caught early.
• Apply software engineering discipline to data workloads: version control, code review, automated testing, and CI/CD for pipelines and transformations.
• Work with analytics, ML, and product teams to make data reliably available, secure, and well governed.

Must have skills:

-8+ years of total professional experience in software, data, or infrastructure engineering.
-Minimum 4 years in a dedicated DevOps / SRE / Platform Engineering role.
-Substantial prior hands‑on data engineering experience at a senior level (pipeline development, data warehousing, and orchestration).
-Strong proficiency in Google Cloud Platform (GCP) and solid working experience with AWS.
-Hands‑on experience with Terraform and Infrastructure as Code (IaC).
-Strong experience with Kubernetes, Docker, and Helm in production.
-Expertise with CI/CD tools: Jenkins, GitLab CI, GitHub Actions, ArgoCD.
-Strong production‑grade Python - not scripting only, but maintainable pipeline and tooling code - plus solid Bash.
-Advanced SQL and hands‑on experience with a cloud data warehouse (BigQuery preferred; Snowflake or Redshift also considered).
-Experience with workflow orchestration (Airflow / Cloud Composer) and a distributed processing framework (Dataflow/Apache Beam, Spark, or dbt).
-Experience with streaming or event‑driven data (Pub/Sub, Kafka, or Kinesis).
-Experience with SonarQube and static code analysis integration.
-Practical cloud security experience (IAM, VPC, firewalls/security groups).
-Secrets management experience (Secret Manager / Secrets Manager / Vault).
-Experience with image and dependency scanning.
-Knowledge of secure software development practices.
-Excellent written and verbal communication skills.

Good to Have:

-Certifications: GCP Professional DevOps Engineer, GCP Professional Data Engineer, GCP Architect, AWS DevOps Engineer, etc.
-Experience with dbt and modern analytics engineering practices.
-Data modelling experience (dimensional modelling, lakehouse formats such as Iceberg or Delta Lake).
-Data quality and testing frameworks (Great Expectations, Soda, dbt tests).
-Exposure to ML platform support (Vertex AI, MLflow, feature stores, model deployment pipelines).
-Knowledge of compliance frameworks (ISO 27001, CIS Benchmarks, NIST) and data governance / PII handling.
-Experience with monitoring tools (Prometheus, Grafana).
-Experience with logging stacks (ELK, Loki, etc.).
-Exposure to policy-as-code and security tools (OPA, scanners).
-Cloud cost optimisation / FinOps experience, particularly for data workloads.
-Scripting or service development in Go.