Azure & Databricks - Senior Data Engineer

Valtech; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Valtech is seeking a Senior Data Engineer to architect cloud-native data solutions on the Microsoft Azure stack, focusing on Databricks, Delta Lake, and scalable data pipelines. You will implement medallion and lakehouse patterns, ensure governance and security, and collaborate with analytics teams to deliver reliable data platforms.

You will work hands-on with Azure Data Factory, Synapse, and Spark, shaping data architectures that support real-time and batch analytics across a global

Qualifications

• Design enterprise data platforms on Azure with Databricks and Delta Lake.
• Develop and optimize ELT/ETL pipelines using Azure Data Factory and Databricks.
• Implement lakehouse architectures and medallion patterns for analytics.
• Collaborate with data scientists and analysts to enable self-service analytics.

Responsibilities

• Architect scalable cloud-native data solutions on Azure.
• Build robust data ingestion from streaming and batch sources.
• Tune Spark jobs for performance and cost efficiency.
• Ensure data governance and security within Azure and Databricks.

Skills

Azure Data Factory

Databricks

Azure Synapse

Spark PySpark

Python/Scala/SQL

Tools

Delta Lake

Job description

Why Valtech? We’re advisors, visionaries, creative and techies. We embrace all things digital. We talk to each other. We have fun. We love our clients. We’re looking ahead We are global

Join our innovative, fast-growing data team at the forefront of cloud data architecture on Microsoft Azure. We're building scalable, secure, and modern data platforms using cutting‑edge Azure services and Databricks unified analytics platform. If you're passionate about creating high‑performance data infrastructure and solving complex big data challenges in a cloud‑native environment, this is the perfect opportunity for you.

As a Senior Data Engineer specializing in Azure and Databricks, you will architect and implement enterprise‑grade cloud‑native data solutions on the Microsoft Azure ecosystem. This is a hands‑on engineering role with significant architectural influence, where you'll work extensively with Azure Data Factory, Databricks, Delta Lake, and other modern data tools to create efficient, maintainable, and scalable data pipelines using medallion architecture and lakehouse patterns.

What you do

4–7 years of hands‑on experience in data engineering with strong focus on cloud data platforms. Proven experience with big data technologies and distributed computing

Technical Expertise

• Azure Mastery: Deep expertise in Azure data services including Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure SQL Database
• Databricks Proficiency: Advanced proficiency in Databricks including cluster management, notebook development, and Delta Lake
• Big Data Processing: Strong experience with Apache Spark (PySpark, Scala), distributed computing concepts, and performance optimization
• Programming: Proficiency in Python, Scala, and SQL for data processing and transformation

Data Architecture

• Experience with lakehouse architecture, medallion architecture, and modern data warehouse design patterns
• Streaming Technologies: Knowledge of real‑time data processing using Azure Event Hubs, Kafka, and Spark Streaming
• Cloud Platforms: Deep understanding of Microsoft Azure ecosystem and native data services

Data Management & DevOps

• Data Governance: Understanding of data governance frameworks, Unity Catalog, and data quality practices
• DevOps: Experience with CI/CD practices, Git‑based workflows, and Infrastructure as Code
• Orchestration: Knowledge of data orchestration tools like Azure Data Factory, Airflow, or similar platforms
• Security: Understanding of Azure security best practices, RBAC, and data encryption

Advanced Technologies

• Experience with Azure Machine Learning and MLOps practices
• Knowledge of containerization (Docker, Kubernetes) and Azure Container Instances
• Experience with Power BI for data visualization and reporting
• Understanding of data mesh architecture and domain‑driven design principles
• Experience with Azure DevOps for CI/CD pipeline management
• Familiarity with monitoring tools like Azure Monitor, Application Insights, and Databricks monitoring

Specialized Skills

• Experience with real‑time analytics and complex event processing
• Knowledge of graph databases and Azure Cosmos DB
• Understanding of data lake optimization techniques and performance tuning
• Experience with multi‑cloud or hybrid cloud architectures

Big Data & Processing

• Compute: Apache Spark (PySpark, Scala), Databricks Runtime
• Data Formats: Delta Lake, Parquet, JSON, Avro
• Architecture: Medallion Architecture, Lakehouse, Data Mesh
• Orchestration: Azure Data Factory, Airflow, Azure Logic Apps

Certifications

• Microsoft Azure Data Engineer Associate (DP‑203)
• Databricks Certified Associate Developer for Apache Spark
• Databricks Certified Professional Data Engineer
• Azure Solutions Architect or Azure Data Fundamentals certifications

What we ask

Data Architecture & Engineering

• Design and implement enterprise‑scale, robust data solutions using Azure and Databricks
• Architect scalable ELT/ETL pipelines using Azure Data Factory, Azure Synapse Analytics, and Databricks
• Build automated data ingestion processes from various sources including streaming and batch data
• Develop and maintain data transformation workflows using PySpark, Scala, and SQL on Databricks

Data Platform & Optimization

• Implement lakehouse architecture using Delta Lake and Databricks Delta tables
• Design and optimize Databricks clusters for performance, cost management, and resource utilization
• Apply medallion architecture (Bronze, Silver, Gold) for data processing and transformation
• Optimize Spark jobs and queries for maximum performance and cost efficiency

Big Data & Analytics

• Build streaming data pipelines using Azure Event Hubs, Kafka, and Databricks Structured Streaming
• Implement machine learning pipelines using MLflow and Databricks ML capabilities
• Design and maintain data models for analytical workloads and real‑time processing
• Work with large‑scale datasets and implement partitioning strategies for optimal performance

Quality & Governance

• Develop comprehensive data quality tests and monitoring using Great Expectations and custom frameworks
• Implement data governance, lineage, and security policies within Azure and Databricks
• Create and maintain comprehensive data documentation and lineage tracking using Unity Catalog
• Build data validation and testing frameworks for proactive data monitoring

DevOps & Automation

• Build and maintain CI/CD pipelines for Databricks notebooks and data workflows
• Develop custom Python/Scala scripts for data processing, manipulation, and automation tasks
• Implement Infrastructure as Code (IaC) using ARM templates or Terraform for Azure resources
• Work with orchestration tools for pipeline scheduling and dependency management

Collaboration & Analytics

• Collaborate with data scientists, analysts, and ML engineers to support advanced analytics use cases
• Enable self‑service analytics capabilities using Databricks SQL and Azure Analytics services
• Work closely with stakeholders to understand data requirements and deliver scalable solutions

What we offer

You’ll join an international network of data professionals within our organisation. We support continuous development through our dedicated Academy. If you're looking to push the boundaries of innovation and creativity in a culture that values freedom and responsibility

At Valtech, we’re here to engineer experiences that work and reach every single person. To do this, we are proactive about creating workplaces that work for every person at Valtech. Our goal is to create an equitable workplace which gives people from all backgrounds the support they need to thrive, grow and meet their goals (whatever they may be). You can find out more about what we’re doing to create a Valtech for everyone here .

Please do not worry if you do not meet all of the criteria or if you have some gaps in your CV. We’d love to hear from you and see if you’re our next member of the Valtech team!