Azure Databricks Architect

HMG AMERICA LLC; Seattle (WA); Hybrid; USD 150,000 - 210,000; Full time

Job summary

HMG AMERICA LLC seeks an experienced Azure Databricks Architect to lead design, implementation, and optimization of enterprise-scale data platforms on Azure. The role requires expertise in Azure Databricks, Delta Lake, and Lakehouse architecture to deliver secure, scalable data solutions.

Responsibilities include designing end-to-end data architecture, governance with Unity Catalog, and optimizing Spark workloads for performance and cost.

Qualifications

• 12+ years IT experience with 5+ years as a Data/Cloud Architect.
• Hands-on Azure Databricks, PySpark, Spark SQL, Python, SQL.
• Expertise in Delta Lake, Lakehouse Architecture, Medallion Architecture.
• Experience with Unity Catalog, RBAC, and data governance.
• Hands-on with Azure Data Factory and ADLS Gen2.
• CI/CD with Azure DevOps, Git, and IaC (Terraform/Bicep/ARM).
• Strong performance tuning, cost optimization, and observability.
• Excellent communication and stakeholder management.

Responsibilities

• Design and architect enterprise-scale Azure Lakehouse solutions.
• Define end-to-end data architecture, ingestion, transformation, and serving layers.
• Design secure data governance using Unity Catalog and role-based access controls.
• Optimize Spark workloads for performance and cost.
• Implement reusable frameworks, coding standards, and CI/CD pipelines.
• Work closely with business, engineering, and cloud infrastructure teams.
• Provide architectural guidance, code reviews, and technical leadership.
• Ensure scalability, reliability, security, and disaster recovery across the platform.

Skills

Azure Databricks

PySpark

Spark SQL

Python

SQL

Delta Lake

Lakehouse Architecture

Medallion Architecture

Unity Catalog

RBAC

ADF

ADLS Gen2

Git

CI/CD (Azure DevOps)

Terraform

Tools

Terraform

Job description

Location: Seattle, WA (Hybrid – 3 Days Onsite)

Duration : Full-Time

Job Summary

We are seeking an experienced Azure Databricks Architect to lead the design, implementation, and optimization of enterprise-scale data platforms on Azure. The ideal candidate will have deep expertise in Azure Databricks, Spark, Delta Lake, and modern Lakehouse architecture, with strong experience designing secure, scalable, and high-performance data solutions. Knowledge of Unity Catalog, Medallion Architecture, and Infrastructure as Code is highly preferred. These are all considered current best practices for enterprise Azure Databricks environments.

Required Skills

• 12+ years of IT experience with 5+ years as a Data/Cloud Architect.
• Strong hands-on experience with Azure Databricks.
• Expertise in PySpark, Spark SQL, Python, and SQL.
• Strong knowledge of Delta Lake, Lakehouse Architecture, and Medallion (Bronze/Silver/Gold) Architecture.
• Experience with Unity Catalog, data governance, RBAC, and data lineage.
• Hands-on experience with Azure Data Factory (ADF).
• Strong experience with Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Synapse Analytics or Microsoft Fabric is a plus.
• Experience with CI/CD using Azure DevOps, Git, and Infrastructure as Code (Terraform/Bicep/ARM).
• Strong understanding of performance tuning, cluster optimization, partitioning, caching, and cost optimization.
• Experience implementing data quality, monitoring, logging, and observability.
• Knowledge of streaming technologies (Structured Streaming/Event Hubs/Kafka) is preferred.
• Excellent communication and stakeholder management skills.

Responsibilities

• Design and architect enterprise-scale Azure Lakehouse solutions.
• Define end-to-end data architecture, ingestion, transformation, and serving layers.
• Design secure data governance using Unity Catalog and role-based access controls.
• Optimize Spark workloads for performance and cost.
• Implement reusable frameworks, coding standards, and CI/CD pipelines.
• Work closely with business, engineering, and cloud infrastructure teams.
• Provide architectural guidance, code reviews, and technical leadership.
• Ensure scalability, reliability, security, and disaster recovery across the platform.

Nice to Have

• Databricks Certified Professional/Architect certification.
• Experience with MLflow, MLOps, or AI/GenAI workloads on Databricks.