Lead I - Data Engineering

UST; Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

UST in Bengaluru is seeking a Senior Data Engineer to design and optimize data pipelines using Azure Databricks, PySpark, and Python. You will build ETL/ELT processes with ADF, Synapse, Informatica, and Fabric, ensure data quality and governance, and collaborate with stakeholders to deliver scalable analytics solutions.

Responsibilities include implementing DataOps, automating deployments with CI/CD, and working with cross-functional teams to translate requirements into robust data architectures.

Qualifications

• Minimum 5+ years of data engineering experience.
• Hands-on experience with Azure Databricks and PySpark.
• Strong Python programming skills.
• Exposure to Microsoft Fabric and modern data platforms.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Azure Databricks, PySpark, and Python.
• Build and maintain ETL/ELT workflows with ADF, Synapse, Informatica, and Fabric.
• Develop robust data engineering solutions to support BI and analytics initiatives.
• Ensure data quality, governance, and monitoring across platforms.

Skills

Azure Databricks

PySpark

Python

Microsoft Fabric

Azure Data Factory

Azure Synapse Analytics

Informatica

ETL/Data Engineering

SQL

Data Quality Frameworks

DataOps

CI/CD Pipeline implementation

Solution Design & Architecture

Strong communication & stakeholder mgm

Tools

Apache Spark

Azure Cloud Services

Data Modeling

Performance Optimization

Azure DevOps

Git

Agile/Scrum

JavaScript

Job description

Role Description

Senior Data Engineer (Azure Databricks | Fabric | PySpark)

Position

Senior Data Engineer

Experience

5+ Years

Primary (Must-Have) Skills

• Azure Databricks
• PySpark
• Python
• Microsoft Fabric
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Informatica
• ETL/Data Engineering
• SQL
• Data Quality Frameworks
• DataOps
• CI/CD Pipeline implementation
• Solution Design & Architecture
• Strong communication and stakeholder management skills

Secondary Skills

• Apache Spark
• Azure Cloud Services
• Data Modeling
• Performance Optimization
• Azure DevOps
• Git
• Agile/Scrum methodology
• JavaScript (basic knowledge is acceptable)

Roles & Responsibilities

• Design, develop, and optimize scalable data pipelines using Azure Databricks, PySpark, and Python.
• Build and maintain ETL/ELT workflows using Azure Data Factory, Synapse, Informatica, and Microsoft Fabric.
• Develop robust data engineering solutions to support business intelligence and analytics initiatives.
• Ensure data quality, integrity, governance, and monitoring across data platforms.
• Implement DataOps best practices and automate deployments using CI/CD pipelines.
• Translate functional and business requirements into technical solutions and development tasks.
• Collaborate with business stakeholders, product owners, architects, and cross-functional teams.
• Troubleshoot production issues, optimize performance, and ensure high availability of data solutions.
• Participate in solution design discussions, technical reviews, and estimation activities.
• Follow coding standards, documentation practices, and industry best practices.

Good-to-Have Skills

• Domain experience in Manufacturing or Retail
• Experience with Azure DevOps
• Knowledge of Delta Lake and Lakehouse architecture
• Experience with Data Warehousing concepts
• Exposure to Power BI
• Experience with cloud migration projects
• Certification in Azure Data Engineering or Databricks
• Knowledge of AI/ML data preparation pipelines

Required Soft Skills

• Excellent communication and interpersonal skills
• Strong analytical and problem-solving abilities
• Client-facing experience
• Ability to work independently as well as in a team
• Strong requirement analysis and solutioning skills

Skills

Apache Spark, Azure, Databricks, JavaScript, Python