Senior Azure Databricks Lakehouse Architect (Medallion)

Diacto Technologies Pvt. Ltd.; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Diacto Technologies Pvt. Ltd. is seeking a Senior Azure Databricks Data Engineer to design and own an enterprise lakehouse. You will build Bronze-Silver-Gold layers on Azure Databricks, with scalable architecture, governance, and cost awareness baked in from day one.

You will lead development of batch and streaming ingestion, data quality rules, and security controls while partnering with stakeholders to deliver data products at scale.

Qualifications

• 7+ years in data engineering, including 3+ years on Azure Databricks.
• Proven Medallion architecture experience from scratch.
• Expert in PySpark and Spark SQL; strong Python and SQL.
• Deep knowledge of Delta Lake and data governance.
• Hands-on with Unity Catalog and data governance controls.
• Strong Azure stack skills (ADLS Gen2, Data Factory, Key Vault).
• Proven results in Spark performance tuning and cloud cost optimization.
• CI/CD and IaC with Git, Azure DevOps or GitHub Actions, Terraform.

Responsibilities

• Design and build the Medallion lakehouse on Azure Databricks and Delta Lake.
• Define layer contracts: Bronze, Silver, Gold with ingestion, cleansing, and modeling.
• Architect scalable platform with governance, security, and cost controls.
• Implement batch/streaming ingestion and performance optimisations.
• Lead DevOps, CI/CD and IaC with Terraform and Databricks Asset Bundles.

Skills

Azure Databricks

PySpark

Spark SQL

Python

SQL

Delta Lake

Unity Catalog

Data Governance

CI/CD

Cloud cost optimization

Tools

Databricks Asset Bundles

Terraform

Git

Azure DevOps

GitHub Actions

Azure Data Factory

Azure Key Vault

Entra ID

Job description

Diacto Technologies Pvt. Ltd. is seeking a Senior Azure Databricks Data Engineer to design and own an enterprise lakehouse. You will build Bronze-Silver-Gold layers on Azure Databricks, with scalable architecture, governance, and cost awareness baked in from day one.

You will lead development of batch and streaming ingestion, data quality rules, and security controls while partnering with stakeholders to deliver data products at scale.