Senior Data Engineer

Antal International; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Antal International is seeking a Senior Data Engineer with 4+ years of experience to build and scale data pipelines powering AI training datasets. You will work with Python, SQL, AWS services, and Airflow to orchestrate workflows and manage large multimodal data.

You'll contribute to data ingestion, processing, quality validation, and optimize GPU-accelerated workflows for audio/video data, WebDataset and Parquet formats, aligning with AI infrastructure needs.

Qualifications

• 4+ years of experience as a Data Engineer.
• Strong Python and SQL proficiency.
• Hands-on experience with AWS services (S3, Athena, Glue, DynamoDB, Lake Formation).
• Experience with Airflow or similar workflow orchestration.
• Experience with multimodal data pipelines (audio/video) and WebDataset/Parquet.
• Understanding GPU workloads and data ingestion/validation.

Responsibilities

• Build and scale data pipelines powering AI training datasets.
• Work at the intersection of Data Engineering, AI infrastructure and multimodal data.

Skills

Python

SQL

Airflow

Large-scale audio/video data

Multimodal data

Tools

AWS S3

AWS Athena

AWS Glue

AWS DynamoDB

Lake Formation

WebDataset

Parquet

Job description

Looking for a Senior Data Engineer with 4+ years of experience to build and scale data pipelines powering AI training datasets.

Key requirements:

• Strong Python & SQL
• Hands-on AWS – S3, Athena, Glue, DynamoDB, Lake Formation
• Airflow / workflow orchestration
• Strong experience with large-scale audio/video or multimodal data pipelines
• Understanding of GPU workloads, WebDataset / Parquet / sharded data
• Experience with data ingestion, processing and quality validation

This is a great opportunity for someone who enjoys working at the intersection of Data Engineering, AI infrastructure and multimodal data.