Senior Data Engineer

DS STREAM; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

DS STREAM in Bengaluru, India is seeking a Senior Data Engineer to expand a high-availability data hub on Azure Databricks. You will design and implement robust data pipelines using PySpark and Structured Streaming, ensuring seamless integration of new data sources within a Medallion Architecture framework.

Collaboration with a skilled team to optimize Unity Catalog and performance is essential. The ideal candidate will have 5–8 years’ experience in data engineering, strong Azure Databricks

Qualifications

• 5–8 years of data engineering experience focusing on large-scale data pipelines.
• Proven experience with Azure cloud and Databricks deployments.
• Strong knowledge of Spark Structured Streaming.
• Proficiency in Python and SQL.
• Ability to build pipelines using GitHub or similar tools.

Responsibilities

• Developing Databricks pipelines in PySpark / SQL
• Ingesting data from APIs
• Handling Spark Structure Streaming pipelines
• Designing and managing objects in Unity Catalog according to Medallion Architecture

Skills

Data engineering experience

Azure cloud and Databricks

Spark Structured Streaming

Python

SQL

GitHub

Tools

Databricks

Unity Catalog

PySpark

Job description

Role Overview

We are looking for an experienced Senior Data Engineer to drive the expansion of a high-availability data hub on Azure Databricks, working in the US time zone. In this role, you will design and implement robust data pipelines using PySpark and Structured Streaming, ensuring seamless integration of new data sources within a Medallion Architecture framework. You will work closely with the team to manage Unity Catalog objects and optimize performance, building scalable solutions that power critical data ecosystems. This position is ideal for an expert in Azure and Databricks who thrives on handling large-scale data challenges.

This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance. The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem.

Key Responsibilities

• Developing Databricks pipelines in PySpark / SQL
• Ingesting data from APIs
• Handling Spark Structure Streaming pipelines
• Designing and managing objects in Unity Catalog according to Medallion Architecture

Required Skills

• Commercial experience of 5-8 years in Data Engineering with focus on large scale data pipelines.
• Proven track record using advanced solutions in Azure cloud and Databricks
• Knowledge of Spark Structured Streaming
• Proficiency in Python and SQL
• Ability to build pipelines using Github or similar tools

Nice to Have

• Cloud or Data Engineering certifications or specialization certifications (e.g. Google Professional Machine Learning Engineer, Microsoft Certified: Azure AI Engineer Associate Exam AI-102, AWS Certified Machine Learning Specialty (MLS-C01), Databricks Certified Machine Learning).
• Contributions to open-source AI projects.
• Experience with Agile/Scrum methodologies.

What We Offer

• Work with a senior team of 10+ years experienced ML/AI experts
• Hands-on experience with latest Generative AI and agentic technologies
• Global exposure working on projects for leading brands
• Technology-agnostic environment fostering innovation
• Competitive compensation package
• Opportunity to shape AI strategy in our growing India operations

Location & Work Mode

Bangalore, Karnataka

Those who are willing to work in US time zone are preferred.