Azure Databricks Architect — Unity Catalog & Delta Lake

Planned Systems International, Inc.; United States; Remote; USD 130,000 - 170,000; Full time

Job summary

Planned Systems International, Inc. is seeking an experienced data engineering professional to design and implement Azure Databricks landing zones, with strong governance via Unity Catalog and Delta Lake. You will drive Terraform-based provisioning, encryption, and audit logging to meet compliance needs.

You will build modern DLT pipelines and CI/CD for batch and streaming workloads, while enabling MLOps with MLflow and feature stores to accelerate team delivery.

Qualifications

• Experience with Azure Databricks landing zones and governance.
• Knowledge of Unity Catalog governance and Delta Lake medallion flows.
• Proficiency with Terraform provisioning and IaC practices.
• Experience building DLT pipelines and CI/CD for batch/streaming.
• Experience with MLOps tooling (MLflow) and feature stores.
• Understanding Entra ID SSO, SCIM, and RBAC mappings for security.

Responsibilities

• Blueprint Azure Databricks landing zones with VNet injection, Private Link, and multi-workspace strategy.
• Establish Unity Catalog governance and Delta Lake medallion flows; automate provisioning with Terraform.
• Implement encryption, audit logging, and policy-as-code for compliance.
• Build DLT pipelines and CI/CD for batch/streaming with monitoring and FinOps.
• Support MLOps with MLflow, feature store, and reusable accelerators.

Skills

Azure Databricks

Unity Catalog

Terraform

Delta Lake

MLOps

DLT pipelines

Security & Compliance

Entra ID SSO

Tools

Terraform

Unity Catalog

MLflow

Feature Store

Job description

Planned Systems International, Inc. is seeking an experienced data engineering professional to design and implement Azure Databricks landing zones, with strong governance via Unity Catalog and Delta Lake. You will drive Terraform-based provisioning, encryption, and audit logging to meet compliance needs.

You will build modern DLT pipelines and CI/CD for batch and streaming workloads, while enabling MLOps with MLflow and feature stores to accelerate team delivery.