Lead II - Data Engineering

UST; Dadri; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

UST in India (Uttar Pradesh, Dadri) is seeking a Senior Software Engineer to build and maintain data platforms and infrastructure. You will work with Python/PySpark, SQL, Terraform, and cloud services to design robust data pipelines and APIs.

The role emphasizes MLOps, Databricks Unity Catalog, Delta Lake, CI/CD, and strong observability. Collaboration and knowledge sharing across teams are expected to drive engineering excellence.

Qualifications

• 5–8 years of senior software engineering experience, preferably in infrastructure or data platform roles.
• Proficiency in Python/PySpark and strong SQL knowledge.
• Experience with Terraform or similar Infrastructure-as-Code tools.
• Strong knowledge of at least one cloud platform (GCP/AWS).
• Solid CI/CD experience.
• ,
• Data & MLOps expertise including Unity Catalog, medallion architecture, and DABs.
• Proficiency in Python and API frameworks such as FastAPI or Flask.
• Strong experience designing, developing, and maintaining RESTful APIs.
• Good understanding of data modeling and workflow orchestration.
• Experience with Spark/Delta Lake/MLflow/ETL frameworks (e.g., Fivetran).
• Operational excellence including Github workflows and container versioning.
• Proficiency with incident management and observability tools (e.g., PagerDuty).
• An AI as a partner mindset and collaborative approach to knowledge sharing.

Responsibilities

• 5–8 years of Senior Software engineering experience, ideally in infrastructure or data platform roles.
• Proficiency in Python/PySpark and strong knowledge of SQL.
• Experience with Terraform or similar Infrastructure-as-Code (IaC) tools.
• Strong knowledge of at least one cloud platform (GCP/AWS).
• Solid CI/CD experience.
• Data & MLOps Expertise: Strong Background in Databricks Unity Catalog, medallion architecture, and Databricks Asset Bundles (DABs).
• Proficiency in Python and API frameworks such as FastAPI or Flask.
• Strong experience designing, developing, and maintaining RESTful APIs.
• Good understanding of data engineering concepts such as data modeling and workflow orchestration.
• Experience with MLOps and Data Engineering like Spark/Delta Lake/MLflow/ETL frameworks (e.g., Fivetran, Lakeflow Connect).
• Operational Excellence: Github workflows, container versioning, incident management and observability tools (PagerDuty).
• An AI as a partner mindset and collaborative knowledge sharing.

Skills

Senior software engineer

Python/PySpark

SQL

CI/CD

DataOps/MLOps

Databricks Unity Catalog

DABs (Databricks Asset Bundles)

Terraform / IaC

Cloud platforms (AWS/GCP)

RESTful APIs

API frameworks (FastAPI/Flask)

Data modeling & orchestration

Observability / incident mgmt

Tools

AWS

Databricks

PySpark

Python

Docker

Terraform

CI/CD tools

PagerDuty

Job description

• 5 to 8 years of Senior Software engineering experience, ideally in infrastructure or data platform roles.


• Proficiency in Python/PySpark and strong knowledge of SQL.


• Experience with Terraform or similar Infrastructure-as-Code (IaC) tools.


• Strong knowledge of at least one cloud platform (GCP/AWS)


• Solid CI/CD experience


• Data & MLOps Expertise:


• Strong Background in Databricks, specifically Unity Catalog, medallion architecture, and Databricks Asset Bundles (DABs).


• Proficiency in Python and API frameworks such as FastAPI or Flask.


• Strong experience designing, developing, and maintaining RESTful APIs.


• Good understanding of data engineering concepts such as data modeling and workflow orchestration.


• Experience with MLOps and Data Engineering like having worked on Spark/Delta Lake/MLflow/ETL frameworks (e.g., Fivetran, Lakeflow Connect).


• Operational Excellence:


• Experience with Github workflows, including container versioning.


• Proficiency with incident management and observability tools (e.g., PagerDuty, system monitoring).


• An AI as a partner mindset.


• A collaborative approach, including asking questions, navigating unclear requirements, and actively contributing to knowledge sharing/community enablement.

Role Description

• 5 to 8 years of Senior Software engineering experience, ideally in infrastructure or data platform roles.


• Proficiency in Python/PySpark and strong knowledge of SQL.


• Experience with Terraform or similar Infrastructure-as-Code (IaC) tools.


• Strong knowledge of at least one cloud platform (GCP/AWS)


• Solid CI/CD experience


• Data & MLOps Expertise:


• Strong Background in Databricks, specifically Unity Catalog, medallion architecture, and Databricks Asset Bundles (DABs).


• Proficiency in Python and API frameworks such as FastAPI or Flask.


• Strong experience designing, developing, and maintaining RESTful APIs.


• Good understanding of data engineering concepts such as data modeling and workflow orchestration.


• Experience with MLOps and Data Engineering like having worked on Spark/Delta Lake/MLflow/ETL frameworks (e.g., Fivetran, Lakeflow Connect).


• Operational Excellence:


• Experience with Github workflows, including container versioning.


• Proficiency with incident management and observability tools (e.g., PagerDuty, system monitoring).


• An AI as a partner mindset.


• A collaborative approach, including asking questions, navigating unclear requirements, and actively contributing to knowledge sharing/community enablement.

Skills

• AWS


• Databricks


• PySpark


• Python


• CI/CD


• FastAPI


• Docker


• Infrastructure as Code


• Terraform


• PagerDuty


• ETL