Senior Databrics Architect

dnA Consulting; Mysuru; Vor Ort; INR 4.000.000 - 7.000.000; Vollzeit

Zusammenfassung

dnA Consulting seeks a Senior Databricks Architect to own the target lakehouse on Azure Databricks, defining medallion layers, storage with ADLS Gen2, and integration with Data Factory. You will design canonical data models for finance, pricing, and operations and write ADRs to document platform choices.

You will implement Unity Catalog, fine-grained access control, data masking, and lineage while guiding DevOps with CI/CD.

Qualifikationen

• 12-15 years in data engineering and data architecture.
• 4+ years hands-on Azure Databricks in production.
• Deep experience with Unity Catalog: catalog design, access control, lineage, governance.
• Expert in PySpark, Spark SQL, and Delta Lake at scale.
• Strong data modeling skills for enterprise reporting and analytics.
• Experience with Azure Data Factory, ADLS Gen2, and Entra ID integration.
• Proven record migrating workloads to lakehouse.
• CI/CD for data with Azure DevOps or GitHub.

Aufgaben

• Define and own target lakehouse architecture on Azure Databricks (medallion layers, ADLS Gen2 storage, ADF integration).
• Design canonical and dimensional data models for finance, pricing, operations.
• Produce ADRs documenting platform choices for reviewable, reproducible outcomes.
• Design ingestion patterns for diverse sources (AS400/DB2, Oracle, SQL Server, SaaS apps).
• Lead and mentor engineers; run design reviews; ensure CI/CD with environment parity.

Kenntnisse

Azure Databricks

PySpark

Spark SQL

Delta Lake

Data modeling

CI/CD for data

Leadership

Stakeholder comms

Tools

Azure Data Factory

ADLS Gen2

Unity Catalog

GitHub

Databricks Asset Bundles

Jobbeschreibung

Role & responsibilities

Architecture & Design

• Define and own the target lakehouse architecture on Azure Databricks, including medallion (bronze / silver / gold) layering, storage design on ADLS Gen2, and integration patterns with Azure Data Factory.
• Design canonical and dimensional data models for finance, pricing, and operational domains, and set modeling standards the whole team follows.
• Produce architecture decision records (ADRs) so that platform choices are documented, reviewable, and reproducible.
• Design ingestion patterns for diverse source systems including AS400 / DB2, Oracle, SQL Server, transportation management systems, and SaaS finance applications.

Data Governance with Unity Catalog

• Design and implement the Unity Catalog structure (metastores, catalogs, schemas) aligned to environments and business domains
• . Implement fine-grained access control, row- and column-level security, and data masking for sensitive data.
• Establish data ownership and stewardship, lineage, tagging, and audit practices using Unity Catalog capabilities. Forward Air | Omni Logistics
• Senior Databricks Architect Job Description
• Partner with Finance, Information Security, and business data owners to translate governance policies into enforceable platform controls.
• Define data quality rules and expectations (e.g., Delta Live Tables expectations or equivalent) and embed them into pipelines.

Hands-On Engineering

• Build and review PySpark, Spark SQL, and Delta Lake code; lead by example on code quality, testing, and documentation.
• Design orchestration using Databricks Workflows and ADF with clear dependencies, retries, and failure handling.
• Tune performance and cost: cluster policies, job compute vs. all-purpose compute, Photon, partitioning, Z-ordering / liquid clustering, and table maintenance.
• Troubleshoot complex production issues and drive root-cause fixes rather than workarounds.

Migration & Modernization

• Plan and lead the migration of workloads from legacy warehouses and semantic layers (e.g., Redshift, SSAS) onto Databricks, including validation and cutover.
• Rationalize redundant pipelines and tools into a simpler, consistent platform.

Leadership & Collaboration

• Act as the technical authority for the data platform across US and India teams; run design reviews and mentor engineers.
• Work with DevOps to ensure every pipeline is deployed through CI/CD with environment parity across Dev, QA, and Production.
• Communicate architecture choices and trade-offs clearly to technical and non-technical stakeholders.

Preferred candidate profile

• 12-15 years in data engineering and data architecture, with 4+ years of hands‑on Azure Databricks in production.
• Deep, practical experience implementing Unity Catalog: catalog design, access control, lineage, and governance policies.
• Expert in PySpark, Spark SQL, and Delta Lake, including performance tuning at scale.
• Strong data modeling skills (dimensional, data vault, or canonical) for enterprise reporting and analytics.
• Experience with Azure Data Factory, ADLS Gen2, Azure Key Vault, and Entra ID integration.
• Proven record of migrating workloads from legacy data warehouses to a lakehouse.
• Working knowledge of CI/CD for data (Azure DevOps or GitHub), Git branching, and Databricks Asset Bundles.
• Excellent written and verbal communication; comfortable presenting to senior leadership.