Senior Databricks Data Engineer

RapidBrains; India; On-site; INR 1,200,000 - 1,900,000; Full time

Job summary

RapidBrains is seeking a Data Engineer with 6–8 years of experience and 3+ years of hands‑on Databricks expertise. You will design enterprise ETL/ELT pipelines using Databricks, PySpark, and Spark SQL on the Azure stack, building Delta Lake pipelines across Bronze/Silver/Gold and enforcing governance via Unity Catalog.

You will also implement Lakeflow Jobs / DLT and drive cost and performance optimization of Spark clusters in a production environment. A Databricks certification is a plus.

Qualifications

• 6–8 yrs Data Engineering experience with 3+ yrs hands‑on Databricks.
• Strong fluency in Python, PySpark, Spark SQL, and Azure stack (ADLS Gen2, ADF, Key Vault, Event Hubs).
• Hands‑on experience with Unity Catalog, Delta Lake, and production pipeline orchestration.

Responsibilities

• Design enterprise ETL/ELT pipelines using Databricks, PySpark and Spark SQL.
• Build Delta Lake pipelines across Medallion Architecture (Bronze/Silver/Gold).
• Enforce unified data governance and access control via Unity Catalog.
• Implement Lakeflow Jobs / DLT and optimize Spark cluster cost/performance.

Skills

Databricks

Python

PySpark

Spark SQL

Azure stack

Unity Catalog

Delta Lake

Production pipelines

Job description

Experience: 6–8 Years (3+ yrs hands-on Databricks)

Type: Full-time

What You’ll Work On

• Design enterprise ETL/ELT pipelines using Databricks, PySpark, and Spark SQL.
• Build robust Delta Lake pipelines across Medallion Architecture (Bronze/Silver/Gold).
• Enforce unified data governance & access control via Unity Catalog.
• Implement Lakeflow Jobs / DLT and drive Spark cluster cost/performance optimization.

Ideal Profile (Must-Haves)

• 6–8 yrs Data Engineering experience with 3+ yrs deep hands-on Databricks.
• Strong fluency in Python, PySpark, Spark SQL, and Azure stack (ADLS Gen2, ADF, Key Vault, Event Hubs).
• Hands‑on experience with Unity Catalog, Delta Lake, and production pipeline orchestration.
• Bonus: Databricks Certified Data Engineer Professional certification.