Databricks Data Specialist – R01569707

Brillio; Bengaluru; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Brillio is seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data pipelines on the Databricks Lakehouse Platform, with strengths in Databricks, PySpark, and SQL.

The role focuses on building batch and real-time data workflows, implementing Delta Lake architectures, governance via Unity Catalog, and collaborating with data analysts and stakeholders to deliver robust data solutions.

Qualifications

• Bachelor’s or Master’s degree in CS, Data Eng, IT, or related field.
• Proven experience designing cloud-based data pipelines and scalable data platforms.
• Strong analytical, troubleshooting, and problem-solving skills.
• Experience in agile and collaborative environments.
• Excellent communication and stakeholder management abilities.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark.
• Build and implement Lakehouse architectures with Bronze, Silver, and Gold layers.
• Develop Delta Lake solutions with ACID and schema enforcement.
• Create, monitor, and optimize Delta Live Tables pipelines.
• Implement scalable data ingestion with Auto Loader.
• Develop real-time data processing with Structured Streaming.
• Orchestrate, schedule, and monitor data workflows with Databricks Workflows.
• Design Lakehouse data models for reporting, analytics, and BI.
• Establish data governance and access controls using Unity Catalog.
• Optimize Spark jobs and SQL queries for performance.
• Collaborate with analysts, architects, and stakeholders to deliver high-quality data solutions.

Skills

Databricks

PySpark

SQL

Apache Spark

Structured Streaming

Auto Loader

Delta Lake

Delta Live Tables

Unity Catalog

Lakehouse Architecture

Databricks Workflows

Power BI

Education

Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related discipline

Tools

Azure

AWS

Airflow

Kafka

DBT

Fivetran

Informatica

Power BI

Job description

We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL, with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark.
• Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.
• Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.
• Create, monitor, and optimize Delta Live Tables (DLT) pipelines.
• Implement scalable and efficient data ingestion processes using Auto Loader.
• Develop and manage real-time data processing solutions using Structured Streaming.
• Orchestrate, schedule, and monitor data workflows using Databricks Workflows.
• Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.
• Establish and enforce data governance, security, and access controls using Unity Catalog.
• Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.
• Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions.

Required Skills (Must Have)

• Databricks Platform
  • Delta Lake
  • Delta Live Tables (DLT)
  • Unity Catalog
  • Databricks Workflows
• PySpark and Apache Spark
• Structured Streaming
• Auto Loader
• SQL
• Lakehouse Data Modeling
• Strong understanding of data engineering best practices and scalable data architectures

Preferred Skills (Good to Have)

• Microsoft Purview
• Microsoft Fabric

AWS Ecosystem

• AWS Glue
• AWS Lambda
• AWS Step Functions
• DBT
• Fivetran
• Informatica

Streaming & Analytics

• Power BI

Data Governance

• Collibra
• Alation

GCP

• BigQuery

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.
• Strong analytical, troubleshooting, and problem-solving capabilities.
• Experience working in agile and collaborative environments.
• Excellent communication and stakeholder management skills.

Preferred Candidate Profile

• Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.
• Strong understanding of data governance, security, and compliance frameworks.
• Experience delivering both batch and real-time data processing solutions.
• Ability to work independently while collaborating effectively across global teams.

Key Technologies

Databricks | PySpark | Apache Spark | Delta Lake | Delta Live Tables (DLT) | Unity Catalog | Structured Streaming | Auto Loader | SQL | Lakehouse Architecture | Azure | AWS | Airflow | Kafka | Power BI

Role Overview

We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL, with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark.
• Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.
• Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.
• Create, monitor, and optimize Delta Live Tables (DLT) pipelines.
• Implement scalable and efficient data ingestion processes using Auto Loader.
• Develop and manage real-time data processing solutions using Structured Streaming.
• Orchestrate, schedule, and monitor data workflows using Databricks Workflows.
• Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.
• Establish and enforce data governance, security, and access controls using Unity Catalog.
• Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.
• Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions.

Required Skills (Must Have)

• Databricks Platform
  • Delta Lake
  • Delta Live Tables (DLT)
  • Unity Catalog
  • Databricks Workflows
• PySpark and Apache Spark
• Structured Streaming
• Auto Loader
• SQL
• Lakehouse Data Modeling
• Strong understanding of data engineering best practices and scalable data architectures

Preferred Skills (Good to Have)

• Microsoft Purview
• Microsoft Fabric

AWS Ecosystem

• AWS Glue
• AWS Lambda
• AWS Step Functions
• DBT
• Fivetran
• Informatica

Streaming & Analytics

• Power BI

Data Governance

• Collibra
• Alation

GCP

• BigQuery

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.
• Strong analytical, troubleshooting, and problem-solving capabilities.
• Experience working in agile and collaborative environments.
• Excellent communication and stakeholder management skills.

Preferred Candidate Profile

• Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.
• Strong understanding of data governance, security, and compliance frameworks.
• Experience delivering both batch and real-time data processing solutions.
• Ability to work independently while collaborating effectively across global teams.

Key Technologies

Databricks | PySpark | Apache Spark | Delta Lake | Delta Live Tables (DLT) | Unity Catalog | Structured Streaming | Auto Loader | SQL | Lakehouse Architecture | Azure | AWS | Airflow | Kafka | Power BI