Senior Databricks Architect — AI & Data Platforms

Ethereum Technologies; Washington (District of Columbia); On-site; USD 140,000 - 210,000; Full time

Job summary

Ethereum Technologies is seeking a senior Databricks architect to lead the adoption, implementation, and optimization of the Databricks Lakehouse platform across data pipelines, analytics, and AI initiatives. You will collaborate with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and troubleshoot complex technical issues.

The role focuses on building end-to-end pipelines with Spark, Delta Lake, Unity Catalog, and MLflow, enforcing security and

Qualifications

• 7+ years in Data Engineering or Architecture.
• 4+ years in the Databricks ecosystem.
• Proven Unity Catalog implementation with row- and column-level security.
• Deep Apache Spark tuning and Photon engine optimization.
• Hands-on MLflow, Vector Search, and generative AI with Databricks Mosaic AI.
• Proficient in Python, Scala, SQL for distributed computing.
• IaC with Terraform to deploy Databricks resources.
• Secure network topologies (Private Link/VPC Peering, BYOK, IP allow lists).
• CI/CD with GitHub Actions/Azure DevOps/GitLab and Databricks CLI or DABs.

Responsibilities

• Lead design, implementation, and optimization of end-to-end Databricks solutions.
• Architect and maintain scalable data ingestion and orchestration pipelines.
• Identify and resolve Spark performance bottlenecks and optimize costs.
• Enforce security, governance, and Unity Catalog best practices.
• Support data science with MLflow, Feature Store, Vector Search, and Mosaic AI.
• Provide root-cause analysis and rapid production issue resolution.
• Mentor engineers and developers through training and documentation.
• Design integrations between Databricks and cloud tools, ETL, BI, and orchestration systems.
• Implement monitoring, logging, alerting, and dashboards for workloads.
• Evaluate new Databricks features and adoption roadmaps.

Skills

Databricks expertise

Python

SQL

Scala

CI/CD pipelines

Spark tuning

Vector Search

MLflow

Terraform

Education

Bachelor's degree

Tools

Terraform

Databricks CLI

Databricks Asset Bundles

Unity Catalog

Photon engine

Job description

Ethereum Technologies is seeking a senior Databricks architect to lead the adoption, implementation, and optimization of the Databricks Lakehouse platform across data pipelines, analytics, and AI initiatives. You will collaborate with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and troubleshoot complex technical issues.

The role focuses on building end-to-end pipelines with Spark, Delta Lake, Unity Catalog, and MLflow, enforcing security and