Big Data Engineer

UST; Maharashtra; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

UST in Pune, India, is seeking a highly motivated Big Data Engineer with 6 to 8 years of hands-on experience to design, build and maintain large-scale data pipelines and platforms.

You will work across batch and streaming workloads using Spark, Databricks, AWS services, and modern data engineering practices, owning components end-to-end and collaborating with product, architecture, and engineering teams to deliver scalable, reliable data solutions.

Qualifications

• 6–8 years of hands-on Big Data or Data Engineering experience.
• Strong knowledge of Data Lake / Lakehouse architectures.
• Proficiency in Java or Scala; strong Python is acceptable.
• Extensive Spark experience for distributed data processing.
• Experience with Apache Airflow or similar workflow tools.
• Experience with Databricks for large-scale workloads.
• Strong AWS experience: S3, Glue, EMR.
• Ability to write clean, scalable code and use AI-assisted coding tools.
• Understanding of Data Structures, Algorithms, and OOP.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Ingest, transform, and process large-scale structured datasets.
• Ensure data quality, reliability, scalability, and availability.
• Collaborate with PMs, Architects, Engineers to implement solutions.
• Participate in code reviews, CI/CD, and deployments.
• Own data platform components and drive end-to-end delivery.
• Troubleshoot performance bottlenecks in pipelines.
• Mentor junior team members and contribute as senior IC.

Skills

Big Data / Data Engineering

Apache Spark

Databricks

Airflow

Python

Java

Scala

AWS

S3

Glue

EMR

Data Lake / Lakehouse

Tools

Dagster

Terraform

Job description

About UST

UST is a leading digital transformation solutions company that partners with global enterprises to drive innovation and business value. We are looking for an experienced

Role Description

UST Title : Lead I - Data Engineering

Location:

Pune, India

Experience:

6 to 8 Years

About UST

UST is a leading digital transformation solutions company that partners with global enterprises to drive innovation and business value. We are looking for an experienced Big Data Engineer to join our growing Data & Analytics team and play a key role in building scalable data platforms and pipelines that power analytics and business insights.

Job Summary

We are seeking a highly motivated Big Data Engineer with 6 to 8 years of experience in designing, building, and maintaining large-scale data processing solutions. The ideal candidate should possess strong expertise in distributed data processing frameworks, cloud technologies, and modern data engineering practices. This role requires an individual contributor who can take ownership of data pipelines, optimize performance, and collaborate with cross-functional teams to deliver reliable and scalable data solutions.

Key Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build robust solutions for ingesting, transforming, and processing large-scale structured datasets.
• Ensure data quality, reliability, scalability, and availability across the data platform.
• Collaborate with Product Managers, Architects, and Engineering teams to define and implement technical solutions.
• Participate in code reviews, testing, CI/CD, and deployment activities to maintain engineering excellence.
• Own data platform components and drive end-to-end delivery and support.
• Troubleshoot and resolve performance bottlenecks within data pipelines and distributed systems.
• Continuously improve data engineering practices, development workflows, and platform capabilities.
• Leverage modern AI-assisted and agentic development tools to improve productivity, testing, and code quality.
• Mentor junior team members and contribute as a senior individual contributor across multiple product initiatives.

Required Qualifications

• 6 to 8 years of hands-on experience in Big Data or Data Engineering.
• Strong understanding of distributed storage and processing architectures including Data Lake and Lakehouse concepts.
• Proficiency in Java or Scala (preferred); strong Python experience is also acceptable.
• Extensive experience with Apache Spark for distributed data processing.
• Hands-on experience with workflow orchestration tools such as Apache Airflow or equivalent.
• Experience working with Databricks for large-scale data engineering workloads.
• Strong experience with AWS services, including:
  • Amazon S3
  • AWS Glue
  • Amazon EMR
• Ability to write clean, optimized, maintainable, and scalable code.
• Experience utilizing AI-powered coding or agentic development tools within the software development lifecycle.
• Strong understanding of:
  • Data Structures
  • Algorithms
  • Object‑Oriented Programming (OOP)

Technical Skills

Big Data & Data Engineering

• Apache Spark
• Databricks
• Data Lake / Lakehouse Architecture
• Batch & Streaming Data Processing

Programming Languages

• Java
• Scala
• Python

Cloud Platforms

• AWS
  • S3
  • Glue
  • EMR

Data Orchestration

• Apache Airflow
• Dagster (Good to Have)

Data Formats

• Parquet
• Avro
• ORC
• JSON

DevOps & Development Tools

• Git
• CI/CD Tools
• Claude Code or equivalent AI‑assisted coding tools

Containerization & Infrastructure

• Docker
• Terraform (Good to Have)

Preferred Skills

• Exposure to modern data platform architectures.
• Experience implementing data quality and monitoring frameworks.
• Knowledge of cloud‑native data engineering best practices.
• Strong analytical and problem‑solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work in a fast‑paced, collaborative environment.

Skills

Scala, Apache Spark, Data Lakes, AWS, Amazon EMR, Amazon S3, Python, AWS Glue, Databricks, Java, Data Engineering