Sr. Data Engineer

Humyn Labs; Bengaluru; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

Humyn Labs seeks a Senior Data Engineer to own, extend, and harden the multimodal data infrastructure powering our AI training data supply chain. You will build and maintain production-grade pipelines moving real-world audio, video, and egocentric capture data from ingestion through AWS-scale storage into GPU-backed technical validation and delivery-ready datasets for frontier AI lab customers.

This role sits at the intersection of data engineering and ML infrastructure; you will own the full

Qualifications

• 4+ years in a data engineering role with end-to-end pipeline ownership.
• Strong Python async patterns, subprocess management, API clients, and data processing at scale.
• Hands-on AWS Athena, Glue, S3 DynamoDB, and Lake Formation; production-grade, not just familiarity.
• Apache Hudi or Delta Lake schema evolution, partition strategies, and upsert patterns.
• SQL proficiency: able to write and optimize complex analytical queries.
• Experience with Airflow or an equivalent workflow orchestrator.
• Demonstrated experience with large-scale media data pipelines: audio/video format conversion, metadata extraction, chunking, and egocentric or multimodal datasets (hard requirement, not a plus).
• Understanding of how data format and I/O design affect downstream GPU compute workloads (WebDataset, sharded Parquet, tfrecord, or equivalent).

Responsibilities

• Own the full data pipeline from ingestion to delivery-ready datasets for AI training.
• Build and maintain production-grade pipelines moving audio, video, and egocentric data through AWS-scale storage to GPU-backed validation.
• Collaborate with Head of Data and ML teams to ensure data quality and performance.
• Optimize data formats and I/O to improve downstream GPU compute workloads.

Skills

Python async patterns

End-to-end pipeline ownership

AWS data stack (Athena, Glue, S3, Lake

Lake Formation

Hudi/Delta Lake

SQL

Airflow

Media data pipelines

GPU compute awareness

Data formats (WebDataset, Parquet, tf)

Tools

AWS Athena

AWS Glue

S3

DynamoDB

Lake Formation

Apache Hudi

Delta Lake

SQL

Airflow

WebDataset

Parquet

TFRecord

Job description

We are looking for a Senior Data Engineer to own, extend, and harden the multimodal data infrastructure that powers our AI training data supply chain. You will work directly with the Head of Data, building and maintaining production-grade pipelines that move real-world audio, video, and egocentric capture data from ingestion through AWS-scale storage into GPU-backed technical validation and finally to delivery-ready datasets for frontier AI lab customers.

This role sits at the intersection of data engineering and ML infrastructure; you will own the full pipeline: ingestion, lake management, quality validation, and compute handoff.

Requirements:

• 4+ years in a data engineering role with end-to-end pipeline ownership.
• Strong Python async patterns, subprocess management, API clients, and data processing at scale.
• Hands-on AWS Athena, Glue, S3 DynamoDB, and Lake Formation; production-grade, not just familiarity.
• Apache Hudi or Delta Lake schema evolution, partition strategies, and upsert patterns.
• SQL proficiency: able to write and optimize complex analytical queries.
• Experience with Airflow or an equivalent workflow orchestrator.
• Demonstrated experience with large-scale media data pipelines: audio/video format conversion, metadata extraction, chunking, and egocentric or multimodal datasets (hard requirement, not a plus).
• Understanding of how data format and I/O design affect downstream GPU compute workloads (WebDataset, sharded Parquet, tfrecord, or equivalent).