Lead Data Architect (Azure Databricks)

Diacto Technologies

India

Remoto

INR 4.000.000 - 7.000.000

Tempo pieno

14 giorni+

Una candidatura completa in un minuto

Curriculum e lettera di presentazione personalizzati, pronti da inviare.

Supera i filtri ATS

Descrizione del lavoro

Diacto Technologies is hiring a Senior Azure Databricks Data Engineer to design and own an enterprise lakehouse on Azure Databricks, from the first workspace to Bronze, Silver and Gold layers.

You will build a scalable Medallion architecture with batch and streaming ingestion, governance, security, and cost controls, while mentoring engineers and collaborating with data scientists and stakeholders.

Competenze

  • 7+ years in data engineering with 3+ years on Azure Databricks.
  • Proven Medallion architecture experience from Bronze to Gold.
  • Expert in PySpark, Spark SQL, Python and SQL.
  • Deep Delta Lake knowledge including ACID, time travel and MERGE.
  • Hands-on Unity Catalog, data governance and security practices.
  • Strong Azure Data Factory, ADLS Gen2 and Key Vault familiarity.
  • Experience with CI/CD and IaC using Terraform and Databricks Asset Bundles.

Mansioni

  • Design and own an end-to-end Medallion lakehouse on Azure Databricks.
  • Define layer contracts for Bronze, Silver and Gold data.
  • Implement batch and streaming ingestion using Auto Loader, Structured Streaming and Delta Live Tables.
  • Apply incremental processing, CDC, SCD patterns, schema evolution and late-arriving data handling.
  • Enforce data quality and quarantine workflows for bad records.
  • Architect scalable Azure Databricks platform with proper environment separation and governance.
  • Tune Spark/Delta performance and manage cost via FinOps.
  • Establish CI/CD pipelines and IaC for reproducible deployments.
  • Lead and mentor engineers; communicate trade-offs to stakeholders.

Conoscenze

PySpark
Spark SQL
Python
SQL
Delta Lake
Unity Catalog
Azure Data Factory
Data governance
CI/CD
Terraform
Databricks
Leadership

Strumenti

Databricks
Terraform
GitHub Actions

Descrizione del lavoro

About the role

We are hiring a Senior Azure Databricks Data Engineer to design, build and own an enterprise lakehouse on Azure Databricks from the first workspace to production-grade Bronze, Silver and Gold layers. You have built a Medallion architecture from the ground up, not just worked inside one. You know how to set up the platform so it scales, performs well and stays cost-efficient, with governance, security and data quality built in from day one.

Key responsibilities
Medallion architecture (Bronze / Silver / Gold)
  • Design and build the Medallion lakehouse from scratch on Azure Databricks and Delta Lake.
  • Define layer contracts: raw, append-only ingestion in Bronze; cleansed, conformed and deduplicated data in Silver; business-ready, modelled data (star schemas, data marts, aggregates) in Gold.
  • Build batch and streaming ingestion with Auto Loader, Structured Streaming, Delta Live Tables (Lakeflow Declarative Pipelines) and Azure Data Factory.
  • Implement incremental processing, CDC, SCD Type 1/2, MERGE patterns, schema evolution and late-arriving data handling.
  • Set data quality rules and expectations at each layer, with quarantine and alerting for bad records.
Platform architecture and scalability
  • Architect the Azure Databricks environment: workspaces, dev/test/prod separation, VNet injection, Private Link, ADLS Gen2 storage layout and naming standards.
  • Design for scale across data volume, number of pipelines, teams and concurrent users.
  • Define cluster policies, compute types (job, all-purpose, serverless, SQL warehouses) and orchestration with Databricks Workflows.
  • Build reusable frameworks: metadata-driven ingestion, common libraries, templates and logging.
Performance optimisation
  • Tune Spark and Delta workloads: partitioning, liquid clustering, Z-ORDER, OPTIMIZE/VACUUM, file sizing, caching, AQE and Photon.
  • Diagnose and fix skew, shuffle, spill and small-file problems using the Spark UI and query profiles.
  • Set and meet SLAs for pipeline run time and data freshness.
Cost optimisation (FinOps)
  • Right-size clusters, use autoscaling, auto-termination, spot instances and serverless where it lowers cost.
  • Enforce cost controls through cluster policies, tagging and budgets.
  • Track DBU and Azure spend with system tables and dashboards, and report savings.
Governance, security and standards
  • Implement Unity Catalog: metastore, catalogs/schemas per layer and domain, external locations, storage credentials.
  • Set up role-based access, row- and column-level security, data masking and PII handling.
  • Enable lineage, auditing, data classification and integration with Microsoft Purview.
  • Manage secrets with Azure Key Vault and identities with Entra ID service principals and managed identities.
  • Define and document engineering standards: coding guidelines, naming conventions, review process and data contracts.
DevOps and delivery
  • Build CI/CD with Azure DevOps or GitHub Actions and Databricks Asset Bundles.
  • Provision infrastructure as code with Terraform or Bicep.
  • Add unit and integration tests, monitoring and alerting for pipelines.
Leadership
  • Act as technical lead: mentor engineers, review code and design, and set best practices.
  • Work with architects, analysts, data scientists and business stakeholders to turn requirements into data products.
Required skills and experience
  • 7+ years in data engineering, including 3+ years building production workloads on Azure Databricks.
  • Proven track record of building a Medallion architecture from the ground up be ready to walk through one you designed end to end.
  • Expert in PySpark and Spark SQL; strong Python and SQL.
  • Deep knowledge of Delta Lake: ACID transactions, time travel, MERGE, CDF, OPTIMIZE, liquid clustering.
  • Hands-on with Unity Catalog and data governance: access control, lineage, auditing, PII protection.
  • Strong Azure stack: ADLS Gen2, Azure Data Factory, Key Vault, Entra ID, Event Hubs, Azure Monitor, networking basics (VNet, Private Endpoints).
  • Proven results in Spark performance tuning and cloud cost optimisation, with numbers to show for it.
  • Data modelling: dimensional modelling (Kimball), Data Vault awareness, SCD patterns.
  • CI/CD and IaC: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles, Terraform.
  • Clear communicator who can explain design trade-offs to both engineers and business stakeholders.
Nice to have
  • Certifications: Databricks Certified Data Engineer Professional, Microsoft Certified: Azure Data Engineer Associate (DP-203) or Fabric Data Engineer (DP-700), Databricks Lakehouse Platform Architect.
  • Experience migrating from on-prem (SQL Server, Hadoop, Informatica, SSIS) or Synapse to Databricks.
  • Databricks SQL, Lakehouse Monitoring, MLflow or Feature Store exposure.
  • dbt on Databricks, Great Expectations, or similar data quality tools.
  • Power BI integration and semantic model design on Gold data.
  • Experience with data mesh or domain-oriented data products.
Ottieni la revisione del curriculum gratis e riservata.

o trascina qui il file.

Similar jobs

Offerte di lavoro simili che vale la pena confrontare

Databricks Lead Job ID: 411651
Databricks Lead Job ID: 411651

Blumetra Solutions LLC • Hyderabad

Ibrido
INR 1.200.000 - 1.900.000
Databricks Lead
Databricks Lead

Blumetra Solutions LLC • Hyderabad

Ibrido
INR 4.000.000 - 7.000.000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Bestal • Hyderabad

Ibrido
INR 4.000.000 - 7.000.000
Senior Databricks Engineer
Senior Databricks Engineer

Value Vision • Bengaluru

In loco
INR 4.200.000 - 6.600.000
Senior Databrics Architect
Senior Databrics Architect

dnA Consulting • Mysuru

In loco
INR 4.000.000 - 7.000.000
Azure Databricks Engineer/Lead/ Architect
Azure Databricks Engineer/Lead/ Architect

HCL Technologies Limited • Bengaluru

In loco
INR 1.200.000 - 1.800.000
Lead I - Data Engineering
Lead I - Data Engineering

UST • Pune

In loco
INR 2.500.000 - 5.000.000
Azure Data Engineer
Azure Data Engineer

InfoBeans • Indore District, Pune District

In loco
INR 3.000.000 - 5.400.000
Databricks Engineering Manager
Databricks Engineering Manager

Unison Group • Bengaluru

In loco
INR 4.000.000 - 6.000.000
Databricks Architect
Databricks Architect

Experion Technologies • Ernakulam

In loco
INR 4.000.000 - 7.000.000