Databricks Platform Architect & AI Solutions Lead

Machine Learning Technologies, LLC; Washington (District of Columbia); On-site; USD 180,000 - 260,000; Full time

Job summary

Machine Learning Technologies, LLC in Washington, DC, seeks a senior Databricks architect to lead the adoption, implementation, and optimization of the Databricks Lakehouse across data pipelines, analytics, and AI initiatives. You will collaborate with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and resolve complex issues.

Responsibilities include building end-to-end Databricks deployments (Unity Catalog, Delta Lake, Databricks SQL,

Qualifications

• This role requires a bachelor’s degree or equivalent work experience in IT/CS/Business administration or related field.
• 7+ years in data engineering or architecture with at least 4+ years in the Databricks ecosystem.
• Proven experience implementing Unity Catalog with row- and column-level security and cross-workspace governance.

Responsibilities

• Lead design, implementation, and optimization of end-to-end Databricks solutions (Unity Catalog, Delta Lake, Databricks SQL, Workflows, Jobs, Mosaic AI).
• Architect and maintain scalable data ingestion, transformation, and orchestration pipelines using Spark, Delta Live Tables, and Auto Loader.
• Identify and resolve Spark performance bottlenecks; implement cost optimization and auto-scaling strategies.
• Establish and enforce Databricks security, governance, and Unity Catalog best practices.
• Support data science teams by enabling MLflow, Feature Store, Vector Search, and generative AI on Databricks.
• Provide RCA and rapid resolution for production workloads; mentor engineers through training and documentation.
• Design integrations between Databricks and cloud platforms, ETL tools, BI tools, catalogs, and orchestration systems.
• Implement monitoring, logging, alerting, and dashboards for Databricks workloads.
• Evaluate new Databricks features and roadmaps for adoption and business value.

Skills

Python

Scala

SQL

Apache Spark tuning

Databricks

Unity Catalog

Delta Lake

MLflow

Vector Search

Terraform

Databricks CLI

CI/CD pipelines

Education

Bachelor's degree in IT/CS/Business or related field

Tools

Terraform

Databricks

GitHub Actions

Azure DevOps

Databricks CLI

Job description

Machine Learning Technologies, LLC in Washington, DC, seeks a senior Databricks architect to lead the adoption, implementation, and optimization of the Databricks Lakehouse across data pipelines, analytics, and AI initiatives. You will collaborate with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and resolve complex issues.

Responsibilities include building end-to-end Databricks deployments (Unity Catalog, Delta Lake, Databricks SQL,