Databricks Lead Job ID: 411651

Blumetra Solutions LLC

Hyderabad

Hybrid

INR 1.200.000 - 1.900.000

Vollzeit

14 Tage+

Eine vollständige Bewerbung in einer Minute

Maßgeschneiderter Lebenslauf und Anschreiben – versandbereit.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

Blumetra Solutions LLC seeks a hands-on Databricks Lead to design, build and productionize governed financial data products on an Azure Databricks lakehouse in Hyderabad. You will own the end-to-end Databricks solution, including ingestion, medallion pipelines, hierarchy modelling, and row-level security, while guiding a small engineering team.

The role emphasizes driving architecture decisions, collaborating with finance and security teams, and delivering low-latency data access with strong

Qualifikationen

  • 7–10 years of overall experience in data engineering or equivalent.
  • Hands-on leadership of data engineering teams on Databricks and Azure.
  • Experience delivering production-grade lakehouse solutions with row-level security.

Aufgaben

  • Own the Databricks solution design end to end.
  • Lead 2–4 engineers; set coding, testing and documentation standards.
  • Collaborate with stakeholders on security, data contracts and governance.

Kenntnisse

Databricks core
Unity Catalog
Fine-grained security
Delta Lake
Performance
PySpark and SQL
Python engineering
Azure
DevOps
Data modelling
Data quality

Jobbeschreibung

Job Title: Databricks – Lead Data Engineer

Work Location: Hyderabad (Hybrid)

Experience: 7 – 10 years

Role summary

We are looking for a hands-on Databricks Lead to design, build and take to production governed financial data products on an Azure Databricks lakehouse. You will own the Databricks solution end to end – file-based ingestion, medallion pipelines, hierarchy modelling, row-level security, zero-outage publishing and performance – and lead a small engineering team while working closely with source-system owners, finance stakeholders, and security and platform teams.

What you will build

A typical engagement publishes multi-dimensional financial data (for example a management P&L with entity, department, project, product, version, period and account dimensions) from an enterprise planning system as a single, row-secured consumption table in Databricks, refreshed several times a day. The quality bar:

  • Exact reconciliation to source control totals; no data loss.
  • Full fidelity of hierarchies, alternate hierarchies, aliases and attributes.
  • Zero consumer outage during refresh; a failed run never replaces the last good data.
  • Row-level security that matches the source system's access rules.
  • Sub-second response for agreed benchmark queries.
  • Additive schema changes absorbed without code changes; full rebuild from Git.
Key responsibilities
Architecture and technical leadership
  • Own the Databricks solution design and drive open decisions with stakeholders: landing zone, publish mechanism, orchestration pattern, compute, retention and recovery targets.
  • Translate integration specifications into buildable designs, data contracts and a delivery plan; lead design reviews with integration architects and platform teams.
  • Lead, mentor and review the work of 2–4 data engineers; set coding, testing and documentation standards.
  • Raise risks early (data volume, security parity, BI connectivity) and propose options with trade-offs.
Platform setup and governance (Unity Catalog)
  • Set up catalogs, schemas, storage credentials, external locations and volumes for landing zones and medallion layers across dev, test and prod.
  • Define the grant model with account-level groups synced from Microsoft Entra ID; configure service principals for jobs and CI/CD.
  • Work with Azure network teams on private endpoints for ADLS and Network Connectivity Configurations for serverless compute.
Ingestion and orchestration
  • Build manifest-gated, file-arrival-triggered Databricks Workflows with task dependencies, task values, condition-task publish gates, queueing, retries, timeouts and notifications.
  • Implement Bronze ingestion from delimited files with checksum and row-count verification against a JSON manifest.
  • Guarantee idempotent reruns and safe handling of overlapping or late runs.
Data modelling and transformation (Silver / Gold)
  • Build PySpark logic to validate and flatten ragged parent-child hierarchies, detecting cycles, orphans, multi-parent members and non-unit weights.
  • Handle alternate hierarchies through configuration, as flattened tables or bridge tables.
  • Pivot long-form attributes and aliases into columns; build a contract-driven Gold generator that absorbs additive schema changes automatically.
  • Design wide, denormalized consumption tables, including leaf/consolidated grain flags to prevent double counting.
Data quality and reconciliation
  • Implement blocking quality checks (contract, referential integrity, hierarchy, typing, enrichment, security) and exact reconciliation to source control totals.
  • Build control tables (run log, check results, reconciliation, schema contract) and an operations dashboard.
Security
  • Implement source-system security parity: map source users to Entra identities, resolve group and member-level rights into a security lookup table, and enforce them with a Unity Catalog row-filter function.
  • Keep data and security consistent during publish; define break-glass access, ownership and audit monitoring.
  • Advise BI teams on per-user connectivity (SSO / DirectQuery) so row-level security holds end to end.
Publishing and performance
  • Implement zero-outage publishing with atomic Delta overwrites or blue/green swaps, plus rollback via time travel.
  • Tune for sub-second queries: liquid clustering, data-skipping statistics, predictive optimization, Photon, SQL Warehouse sizing and row-filter cost; run benchmark tests with real access profiles.
DevOps, testing and operations
  • Package everything as Databricks Asset Bundles with CI/CD (GitHub Actions or Azure DevOps) across dev, test and prod.
  • Own the test strategy: unit tests, contract and fixture tests, integration, SIT reconciliation, security UAT and performance tests.
  • Write the runbook (rerun, rollback, full rebuild, schema change) and hand over to production support.
Collaboration
  • Agree file contracts, manifests and control totals with source-system administrators.
  • Work with finance users to define benchmark queries and validate numbers during UAT and parallel runs.
  • Report progress, risks and decisions to stakeholders and delivery leadership.
Required skills

Area

Databricks core

Unity Catalog

Fine-grained security

Delta Lake

Performance

PySpark and SQL

Python engineering

Azure

DevOps

Data modelling

Data quality

Must have delivered at least one of:
  • A production Databricks pipeline with Unity Catalog row-level security in a regulated or finance setting.
  • A file-based, event-triggered ingestion framework with manifest or control-file validation.
  • A financial planning or EPM data integration into a lakehouse or warehouse.
Preferred skills, qualifications and attributes
Nice to have
  • Enterprise planning / EPM platforms: multi-dimensional cubes, dimensions and hierarchies, rules and consolidations, member-level security, and exporting data from them.
  • FP&A domain knowledge: management P&A, budget and forecast versions, adjustments and allocations, spend by product or project, GAAP vs non-GAAP.
  • Power BI with Databricks: DirectQuery, Entra SSO pass-through, impact of import models on row-level security.
  • Lakeflow Declarative Pipelines (DLT) and expectations, Auto Loader, file events on external locations.
  • Databricks system tables for cost and job monitoring; Lakeview dashboards and SQL alerts.
  • Experience in regulated environments (SOX controls, audit trails, change management).
Certifications (any is a plus)
  • Databricks Certified Data Engineer Professional
  • Databricks Certified Data Engineer Associate
  • Microsoft Certified: Azure Data Engineer Associate or Azure Solutions Architect Expert
Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Databricks Lead
Databricks Lead

Blumetra Solutions LLC • Hyderabad

Hybrid
INR 4.000.000 - 7.000.000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Bestal • Hyderabad

Hybrid
INR 4.000.000 - 7.000.000
Lead Data Architect (Azure Databricks)
Lead Data Architect (Azure Databricks)

Diacto Technologies • Indien

Remote
INR 4.000.000 - 7.000.000
Senior Databricks Engineer
Senior Databricks Engineer

Value Vision • Bengaluru

Vor Ort
INR 4.200.000 - 6.600.000
Senior Databrics Architect
Senior Databrics Architect

dnA Consulting • Mysuru

Vor Ort
INR 4.000.000 - 7.000.000
Sr. Databricks Engineer
Sr. Databricks Engineer

Fa Ewjt Saasfaprod1 • Pune District

Hybrid
INR 3.500.000 - 6.000.000
Sr. Databricks Engineer
Sr. Databricks Engineer

ExlService Holdings, Inc. • Bengaluru

Hybrid
INR 4.500.000 - 7.500.000
P1 - GP - Senior Data Engineer / Data Architect
P1 - GP - Senior Data Engineer / Data Architect

Atlas Systems • Bengaluru

Vor Ort
INR 4.200.000 - 6.400.000
Sr. Databricks Engineer
Sr. Databricks Engineer

ExlService Holdings, Inc. • Pune District

Hybrid
INR 3.500.000 - 6.500.000
Senior Data Engineer
Senior Data Engineer

CoffeeBeans • Bengaluru

Vor Ort
INR 1.800.000 - 2.400.000