Databricks Architect

Empiric; United States; Remote; USD 124,000 - 145,000; Part time

Job summary

Empiric is seeking a Databricks Architect to own the design and delivery of a modern lakehouse platform. You will lead moves from legacy data systems to Databricks, set technical direction for data engineering and ML workloads, and mentor engineers while staying hands-on with code.

This role focuses on architecture, roadmapping, and aligning delivery teams with business priorities, while enabling scalable, secure data and ML workflows.

Qualifications

• Significant Databricks architecture experience in enterprise settings.
• Hands-on with Spark, Delta Lake and Unity Catalog.
• Proficient in Python or Scala and production-grade code review.

Responsibilities

• Design end-to-end Databricks architecture including workspace strategy and governance.
• Lead migrations from legacy systems to Databricks with risk management.
• Build and optimise large-scale Spark workloads (batch and streaming).
• Develop and guide MLOps capabilities using MLflow and model deployment pipelines.
• Collaborate with data scientists to productionise models and ensure governance.

Skills

Databricks architecture

Cloud experience

Stakeholder management

Leadership

Python/Scala coding

Tools

Databricks

Delta Lake

Unity Catalog

MLflow

Apache Spark

Terraform

Job description

Databricks Architect - Python, Spark, MLOps

Location: Remote

Contract: 6-months initial

Rate: $90 - $105 per hour

We're looking for a Databricks Architect to own the design and delivery of a modern lakehouse platform for an enterprise client. You'll lead the move off legacy data systems onto Databricks, set the technical direction for data engineering and machine learning workloads, and be the person senior stakeholders turn to when they need complex data decisions explained clearly.

This is a hands-on architecture role. You'll shape the roadmap and also stay close enough to the code to unblock engineers, review designs and prove out patterns yourself.

What you'll do

• Design and own the end-to-end Databricks architecture: workspace strategy, Unity Catalog governance, Delta Lake design, security and cost management
• Set the standards the wider data team builds against and ensure the platform scales with new use cases
• Lead migrations from legacy warehouses, Hadoop or on-premises platforms into Databricks
• Assess the current estate, plan a phased migration path, and manage risk and data validation without disrupting the business
• Design and optimise large-scale Apache Spark workloads, both batch and streaming
• Write production-grade code in Python or Scala and guide engineers on performance tuning, cluster configuration and best practice
• Build out MLOps capability, including MLflow experiment tracking, model registry and CI/CD for model deployment
• Work with data scientists to move models from notebooks into reliable, monitored production
• Act as the trusted technical voice for business and technical stakeholders
• Run workshops, translate requirements into architecture and present trade-offs to senior leadership
• Keep delivery teams, product owners and executives aligned on scope and priorities

What you'll bring

• Significant experience architecting data platforms on Databricks, ideally in an enterprise setting
• Strong knowledge of Delta Lake, Unity Catalog and lakehouse design
• Hands-on experience with at least one major cloud: Azure, AWS or GCP
• Led at least one substantial data platform migration end to end, from discovery through cutover and legacy decommissioning
• Deep Apache Spark expertise
• Strong coding ability in Python or Scala, able to write, review and optimise production code
• Practical MLOps experience, including MLflow, model lifecycle management and production ML with monitoring and governance
• Proven stakeholder management: as comfortable presenting to a CTO as pairing with an engineer
• Skilled at building trust, managing expectations and keeping complex programmes moving

Nice to have

• Databricks certification, particularly Data Engineer Professional or Machine Learning Professional
• Cloud architecture certification on Azure, AWS or GCP
• Infrastructure as code experience, such as Terraform
• Real-time streaming experience with Structured Streaming or Kafka