Senior /Staff Data & Cloud Platform Engineer

Vibehackers; Hyderabad; On-site; INR 3,600,000 - 6,000,000; Full time

Job summary

Vibehackers seeks a Senior/Staff Data & Cloud Platform Engineer to design, build, and operate scalable data ingestion and processing platforms across cloud and on-prem environments, focusing on ETL/ELT pipelines and workflow orchestration for manufacturing and analytics workloads.

The role emphasizes security, infrastructure automation with Terraform, Kubernetes and Docker, and AI-assisted coding to automate workflows.

Qualifications

• Bachelor’s degree in CS or related field.
• 5+ years in data engineering or cloud/platform roles.
• Strong Python and SQL skills.
• Experience with GCP and AWS services.
• IaC, Kubernetes, Docker, CI/CD experience.
• Knowledge of schema/metadata management and CDC/delta processing.
• AI-assisted coding and AI tool usage to automate workflows.

Responsibilities

• Build scalable data ingestion pipelines for diverse data sources.
• Develop batch, scheduled, and event-driven workflows with Airflow and cloud services.
• Implement ETL/ELT frameworks with data quality checks.
• Manage schema registry, versioning, and evolution.
• Design delta detection and incremental refresh mechanisms.
• Set up secure landing zones for multi-cloud/on-prem validation.
• Automate infrastructure with Terraform and manage container workloads.
• Implement IAM/RBAC, secrets management, and cost controls.
• Monitor, troubleshoot, and optimize platform reliability and cost.
• Collaborate with IT, AI/Data Science, Product Eng, and manufacturing teams.

Skills

Python

SQL

Apache Airflow

Terraform

Kubernetes

Docker

CI/CD

Data Modeling

Metadata Management

Incremental Processing

Cloud Security

IAM/RBAC

Cost Optimization

AI-assisted coding

Education

Bachelor’s degree in Computer Science or related field

Tools

GCP

AWS

BigQuery

Dataflow

Pub/Sub

Cloud Functions

S3

Redshift

IAM

Terraform

Kubernetes

Docker

Job description

Explicitly requires AI-assisted (vibe) coding skills and use of AI tools to automate workflows and improve efficiency.

About the Role

Senior/Staff Data & Cloud Platform Engineer responsible for designing, building, and operating scalable data ingestion and processing platforms across cloud and on-prem environments. The role focuses on ETL/ELT pipelines, workflow orchestration, infrastructure automation, security, and cross-functional delivery to support TB-scale manufacturing and analytics workloads.

Job Description

Role

Senior/Staff Data & Cloud Platform Engineer focused on building scalable, secure data platforms for ingestion, processing, storage, and analytics across AWS, GCP, and on-prem environments. The role emphasizes pipeline engineering, workflow orchestration, schema and metadata management, infrastructure automation, and operational reliability for large-scale manufacturing and engineering data.

Key Responsibilities

• Build scalable ingestion pipelines for structured, semi-structured, and engineering data sources.
• Develop batch, scheduled, and event-driven workflows using Apache Airflow and cloud-native services.
• Implement ETL/ELT frameworks, reusable pipeline patterns, incremental ingestion, and data quality checks.
• Implement schema registry/versioning, schema evolution, and compatibility controls.
• Design delta detection, CDC-style processing, watermarking, and incremental refresh mechanisms.
• Extract and normalize manufacturing identifiers (e.g., Lot, Die, Wafer, Test, Flow, Product, Step).
• Set up secure cloud landing zones for AWS, GCP, and on-prem validation and analytics pipelines.
• Automate infrastructure provisioning with Terraform and CI/CD, and manage platform operations for containerized workloads (Kubernetes, Docker).
• Implement IAM, RBAC, hybrid connectivity, secrets management, and cost optimization for TB-scale data.
• Monitor, troubleshoot, and optimize platform reliability, performance, and cost.
• Partner cross-functionally with IT, AI/Data Science, Product Engineering, and manufacturing teams.

Requirements

• Bachelor’s degree in Computer Science, Data Engineering, Cloud Engineering, Information Systems, or related technical field.
• 5+ years of experience in data engineering, cloud/platform engineering, DevOps, or infrastructure engineering.
• Strong hands-on experience with Python and SQL.
• Practical experience with GCP services (Dataflow, Pub/Sub, Cloud Functions, BigQuery, Cloud Storage, IAM) and AWS services (Glue, Lambda, S3, Redshift, IAM).
• Experience with Terraform, Kubernetes, Docker, CI/CD, cloud networking, and operational automation.
• Experience implementing schema/metadata management, incremental processing (CDC/delta), and data quality validation.
• Knowledge of cloud security, IAM/RBAC, hybrid connectivity, and cost management for large-scale data.
• Proven ability to leverage AI-assisted (vibe) coding techniques and AI tools to automate workflows and improve efficiency.

Preferred

• Domain exposure to semiconductor, manufacturing, validation, reliability, test, yield, or RCA data environments.
• Familiarity with AI/analytics platform foundations, data governance, lineage, and governed data products.
• Certifications in AWS, GCP, Kubernetes, Terraform, or data engineering are a plus.

Tools & Technologies (explicitly referenced)

Apache Airflow, Python, SQL, GCP (Dataflow, Pub/Sub, Cloud Functions, BigQuery, Cloud Storage, IAM), AWS (Glue, Lambda, S3, Redshift, IAM), Terraform, Kubernetes, Docker, CI/CD.

Apache Airflow Python SQL GCP Dataflow GCP Pub/Sub GCP Cloud Functions GCP BigQuery GCP Cloud Storage GCP IAM AWS Glue AWS Lambda AWS S3 AWS Redshift AWS IAM Terraform Kubernetes Docker CI/CD

Skills

Data Engineering Cloud Engineering Infrastructure-as-Code DevOps Platform Operations Workflow Orchestration Data Modeling Schema & Metadata Management Incremental/CDC Processing Monitoring & Observability Security & Governance Cost Optimization Collaboration Troubleshooting Automation AI-assisted (vibe) coding