Senior Databricks Engineer

Value Vision; Bengaluru; On-site; INR 4,200,000 - 6,600,000; Full time

Job summary

Value Vision in Bengaluru seeks a Senior Databricks Engineer to own design and delivery of a Databricks-first data platform. You will build high-performance data pipelines, architect scalable Lakehouse solutions, and champion engineering best practices across the team.

You will lead Lakehouse architecture, Delta Lake, Unity Catalog, and Delta Live Tables, while mentoring engineers and enforcing governance and SDLC standards within the organization.

Qualifications

• 10+ years of hands-on data engineering experience.
• Databricks Certified Data Engineer Professional or Associate mandatory.
• Deep expertise in Databricks components (Delta Lake, Unity Catalog, Delta Live Tables, Workflows).
• Advanced PySpark and Python for large-scale distributed processing.
• Strong SQL skills, data modeling and warehousing concepts.
• Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
• Hands-on experience with Azure DevOps CI/CD and IaC (Terraform or Bicep).
• Understanding of Data Warehousing and Data Lakehouse patterns.
• Experience with Structured Streaming and data governance tools (Purview).
• Familiarity with dbt for SQL-based transformations and MLflow for ML integration.

Responsibilities

• Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
• Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
• Unity Catalog: Govern data assets through Unity Catalog, enforcing fine-grained access control and lineage tracking.
• Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies.
• Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling and alerting.
• ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and files into the Lakehouse.
• Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
• Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables.
• CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
• Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and Azure infrastructure using Terraform or Bicep.
• Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible deployments.
• Data Quality & Lineage: Implement data validation, lineage, and governance using Purview or similar tooling.
• Documentation: Produce thorough technical documentation for pipelines, data models, and runbooks.
• Mentorship & Stakeholder Engagement: Coach junior engineers and translate business requirements into robust technical solutions.
• Standards & Governance: Enforce SDLC processes and platform governance policies.

Skills

Mentorship

Stakeholder communication

Team leadership

Education

Databricks Certified Data Engineer Professional or Associate

Tools

Databricks

Delta Lake

Unity Catalog

Delta Live Tables

Azure Data Factory

ADLS Gen2

Azure Synapse

Azure DevOps

Terraform

Bicep

dbt

MLflow

Databricks Asset Bundles

Microsoft Purview

Structured Streaming

Job description

We are looking for a Senior Databricks Engineer to become a cornerstone of our Data & Analytics team. In this role, you will own the design and delivery of our Databricks-first data platform - building high-performance data pipelines, architecting scalable Lakehouse solutions, and championing engineering best practices across the team. You will work closely with business analysts, a data modeler and business stakeholders to turn raw data into reliable, production-grade analytics products.

Responsibilities

• Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
• Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
• Unity Catalog: Govern data assets - tables, volumes, and models - through Unity Catalog, enforcing fine-grained access control and lineage tracking.
• Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
• Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
• ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
• Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
• Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
• Code
• Write clean,
• and
• coding standards.
• CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
• Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
• Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
• Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
• Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
• Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
• Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
• Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
• Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.

Requirements

• 10+
• Databricks Certified Data Engineer Professional or Associate (mandatory).
• 10+ years of hands-on experience in Data Engineering roles.
• Deep expertise in Databricks - Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
• Advanced PySpark and Python skills for large-scale distributed data processing.
• Strong SQL skills: query authoring, performance tuning, and data modeling.
• Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
• Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
• Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
• Experience with Structured Streaming or other real-time data processing frameworks.
• Experience with Microsoft Purview for data cataloguing and governance.
• Familiarity with dbt (data build tool) for SQL-based transformation workflows.
• Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
• Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
• Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures.
• Background in data mesh or data product thinking.