Sr. Bigdata Engineer

UST; Hyderabad; On-site; INR 4,000,000 - 9,000,000; Full time

Job summary

UST is looking for a Sr. Big Data Engineer to lead the design and delivery of large-scale, production-grade ETL pipelines on AWS. You will architect batch and streaming solutions, optimize cost and performance, and set engineering standards across the team.

The role requires a minimum of 10 years in data engineering, hands-on experience with Glue, EMR, PySpark, and Step Functions, and the ability to mentor others while partnering with data scientists, analysts, and product teams.

Qualifications

• Minimum 10 years in data engineering / big data is mandatory.
• Production ETL delivery experience with data quality controls and schema evolution.
• Hands-on experience with Glue, EMR, PySpark, Step Functions and cost/perf optimization.

Responsibilities

• Lead design and implementation of batch and streaming ETL pipelines on AWS.
• Architect and deliver scalable pipelines, optimize cost and performance, and set engineering standards.
• Mentor engineers and collaborate with data scientists, analysts, and product teams.

Skills

Python

SQL

Spark

Data engineering

ETL design

Education

Bachelor's or Master's in CS/Engineering or equivalent

Tools

AWS Glue

Amazon EMR

AWS Step Functions

S3/Lake Formation

Terraform/CDK/CFN

Job description

Who We Are

At UST, we help the world’s best organizations grow and succeed through transformation. Bringing together the right talent, tools, and ideas, we work with our client to co-create lasting change. Together, with over 30,000 employees in 30+ countries, we build for boundless impact—touching billions of lives in the process. Visit us at .

Role Description

UST Tittle: Lead I - Data Engineering

Who We Are

At UST, we help the world’s best organizations grow and succeed through transformation. Bringing together the right talent, tools, and ideas, we work with our client to co-create lasting change. Together, with over 30,000 employees in 30+ countries, we build for boundless impact—touching billions of lives in the process. Visit us at .

Summary

UST is looking for Sr. Big Data Engineer

The Opportunity

• Hands-on senior engineer and technical lead who has personally built and shipped large-scale, production-grade ETL pipelines on AWS.
• The resource will architect and deliver batch/streaming pipelines, optimize cost and performance at scale, and set the team's engineering standards.

Required

• Minimum 10 years in data engineering / big data is mandatory.
• Mandatory Qualifications Production experience (not POCs) required in all of: -
• AWS Glue — Glue ETL jobs, Data Catalog, crawlers, job bookmarks, dynamic frames; tuning for cost/runtime. -
• Amazon EMR — Cluster lifecycle (transient, EMR Serverless / on EKS), Spark tuning, instance fleets / spot. -
• PySpark — DataFrame APIs, UDFs, partitioning, join strategies, skew handling, and Spark memory/executor tuning. -
• AWS Step Functions — Orchestrating multi-stage ETL with retries, error handling, and parallel/map states. -
• Production ETL delivery — Delivered ETL at scale with data quality controls, schema evolution, idempotency, and backfills.

Core Skills

• Apache Spark internals; Parquet/ORC/Avro and Iceberg/Hudi/Delta; partitioning and small-file management - S3 data lake, Athena, Redshift/Spectrum, Lake Formation; Lambda, SQS/SNS, Event Bridge; Kinesis/MSK (plus)
• Expert Python and strong SQL; IAC (Terraform / CDK / CloudFormation); CI/CD; Git; Docker - Dimensional modelling, Lakehouse / medallion architectures, batch and CDC ingestion
• Production-Readiness Expectations - Reliability (retries, checkpointing, safe re-runs, SLAs/SLOs)
• Observability (CloudWatch/Datadog metrics, logging, lineage)
• Data quality (validation frameworks, late/malformed data handling)
• Cost and performance optimization at multi-terabyte scale
• Security/compliance (PII, encryption, masking, access governance)
• Operational ownership (on-call, incident response, RCA)

Leadership

• Mentor engineers, run design/code reviews, and lead architecture decisions.
• Partner with data scientists, analysts, and product to deliver multi-quarter initiatives.

Nice To Have

• AWS certifications; streaming (Kafka/Kinesis/Structured Streaming/Flink);
• Airflow/MWAA; dbt; regulated-domain experience (financial / credit / rental data).

Education

• Bachelor's or Master's in CS/Engineering or equivalent practical experience.

What We Believe

We’re proud to embrace the same values that have shaped UST since the beginning. Since day one, we’ve been building enduring relationships and a culture of integrity. And today, it's those same values that are inspiring us to encourage innovation from everyone, to champion diversity and inclusion and to place people at the centre of everything we do.

Humility

We will listen, learn, be empathetic and help selflessly in our interactions with everyone.

Humanity

Through business, we will better the lives of those less fortunate than ourselves.

Integrity

We honour our commitments and act with responsibility in all our relationships.

Equal Employment Opportunity Statement

UST is an Equal Opportunity Employer. We believe that no one should be discriminated against because of their differences, such as age, disability, ethnicity, gender, gender identity and expression, religion, or sexual orientation.

All employment decisions shall be made without regard to age, race, creed, colour, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

UST reserves the right to periodically redefine your roles and responsibilities based on the requirements of the organization and/or your performance.

• To support and promote the values of UST.
• Comply with all Company policies and procedures

Skills

AWS, AWS CDK, AWS CloudFormation, AWS Glue, AWS Step Functions, Amazon Athena, Amazon CloudWatch, Amazon EMR, Amazon EventBridge, Amazon SNS, Amazon S3