Databricks Architect

Programmers.io; Minneapolis (MN); On-site; USD 140,000 - 190,000; Full time

Job summary

Programmers.io in Minneapolis seeks an Azure Databricks Architect to design scalable Lakehouse solutions on Azure using Databricks, Delta Lake, PySpark, Spark SQL, ADLS Gen2, ADF, Synapse, and governance practices.

The role focuses on cloud data architecture, pipeline development, performance optimization, and migration from legacy systems across SAS, Hadoop, Teradata, and Oracle. Onsite work is required three days per week.

Qualifications

• Hands-on cloud data architecture and Databricks platform design.
• Experience with Delta Lake and medallion architecture.
• Proficient in PySpark, Spark SQL, and data governance.

Responsibilities

• Design and deliver scalable Lakehouse solutions on Azure with Databricks.
• Build and optimize pipelines using Databricks Workflows and Delta Live Tables.
• Work with ADLS Gen2, Synapse, PySpark, Spark SQL for enterprise data needs.
• Optimize Spark workloads through partitioning, sizing, and data skew handling.
• Lead migrations from legacy platforms like SAS, Hadoop, Oracle, and SQL Server.
• Define data modeling, governance, security, lineage, and access controls.

Skills

Databricks

Delta Lake

PySpark

Spark SQL

Data governance

Security

Data modeling

Migration

Cloud architecture

Azure Data Factory

Synapse

ADLS Gen2

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Databricks SQL

Synapse

Job description

Location: Minneapolis, MN (3 Days onsite/week)

Hire Type: Full Time

Job Summary:

The Azure Databricks Architect will design and deliver scalable Lakehouse solutions on Azure using Databricks, Delta Lake, PySpark, Spark SQL, ADLS Gen2, ADF, Synapse, and modern data governance practices.

Key Skills & Responsibilities

• The candidate should have strong hands-on experience in cloud data architecture, Databricks platform design, performance optimization, migration, data modeling, streaming, and security.
• Design Databricks Lakehouse architecture using Delta Lake and Bronze/Silver/Gold Medallion layers.
• Build pipelines using Databricks Workflows, Delta Live Tables, Auto Loader, Azure Data Factory, and Databricks SQL.
• Work with Azure Data Lake Storage Gen2, Synapse, PySpark, Spark SQL, and SQL for enterprise data solutions.
• Optimize Spark workloads through partitioning, cluster sizing, data skew handling, broadcast joins, and Photon acceleration.
• Lead migration from SAS, Hadoop, Teradata, Oracle, SQL Server, Synapse, and legacy ETL platforms.
• Define data modeling, domain architecture, batch/streaming, event-driven processing, governance, security, lineage, and access controls.