Senior Data Engineer (Azure/GCP)

DS STREAM; Bengaluru; Hybrid; INR 1,600,000 - 2,400,000; Full time

Benefits offered by this job

Competitive compensation package

Senior ML/AI team exposure

Global project experience

Technology-agnostic environment

Shape AI strategy in India operations

Job summary

DS STREAM seeks a Senior Data Engineer to expand a high-availability data hub on Azure Databricks and GCP BigQuery. You will design robust data pipelines using PySpark and Structured Streaming, integrating new sources within a Medallion Architecture framework while managing Unity Catalog objects for governance.

Ideal candidates have 5–8 years of data engineering experience, strong Python/SQL skills, and a proven Azure/Databricks track record, with familiarity across GCP services and GitHub for

Qualifications

• Commercial experience of 5-8 years in Data Engineering focusing on large-scale data pipelines.
• Proven track record with Azure cloud and Databricks; also GCP and GCP BigQuery.
• Knowledge of Spark Structured Streaming.
• Proficiency in Python and SQL.
• Ability to build pipelines using GitHub or similar tools.

Responsibilities

• Develop Databricks pipelines in PySpark/SQL.
• Ingest data from APIs.
• Handle Spark Structured Streaming pipelines.
• Design and manage Unity Catalog objects according to Medallion Architecture.

Skills

Data engineering

Azure Databricks

GCP BigQuery

Spark Structured Streaming

Python

SQL

GitHub

Tools

Databricks

Job description

Role Overview

We are looking for an experienced Senior Data Engineer to drive the expansion of a high-availability data hub on Azure Databricks, and GCPBigquery. In this role, you will design and implement robust data pipelines using PySpark and Structured Streaming, ensuring seamless integration of new data sources within a Medallion Architecture framework. You will work closely with the team to manage Unity Catalog objects and optimize performance, building scalable solutions that power critical data ecosystems. This position is ideal for an expert in Azure and Databricks who thrives on handling large-scale data challenges.

This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance. The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem.

Key Responsibilities

• Developing Databricks pipelines in PySpark / SQL
• Ingesting data from APIs
• Handling Spark Structure Streaming pipelines
• Designing and managing objects in Unity Catalog according to Medallion Architecture

Required Skills

• Commercial experience of 5-8 years in Data Engineering with focus on large scale data pipelines.
• Proven track record usng advanced solutions in Azure cloud and Databricks also GCP and GCP Bigquery
• Knowledge of Spark Structured Streaming
• Proficiency in Python and SQL
• Ability to build pipelines using Github or similar tools

Nice to Have

• Cloud or Data Engineering certifications or specialization certifications (e.g. Google Professional Machine Learning Engineer, Microsoft Certified: Azure AI Engineer Associate Exam AI-102, AWS Certified Machine Learning Specialty (MLS-C01), Databricks Certified Machine Learning).
• Contributions to open-source AI projects.
• Experience with Agile/Scrum methodologies.

What We Offer

• Work with a senior team of 10+ years experienced ML/AI experts
• Hands-on experience with latest Generative AI and agentic technologies
• Global exposure working on projects for leading brands
• Technology-agnostic environment fostering innovation
• Competitive compensation package
• Opportunity to shape AI strategy in our growing India operations

Location & Work Mode

Bangalore, Karnataka