Databricks Lakehouse Architect & AI Solutions Lead

Cloud Analytics Technologies, LLC; Washington (District of Columbia); On-site; USD 150,000 - 200,000; Full time

Job summary

Cloud Analytics Technologies, LLC seeks a senior Databricks Architect to drive adoption, implementation, and optimization of the Databricks Lakehouse across data pipelines, analytics, and AI initiatives. The role collaborates with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and troubleshoot complex issues.

Responsibilities include leading end-to-end Databricks design, building ingestion pipelines with Spark and Delta Live Tables, enforcing

Qualifications

• Bachelor's degree or equivalent in IT/CS or related field.
• 7+ years in Data Engineering or Architecture, with 4+ years on Databricks.
• Experience implementing Unity Catalog with cross-workspace governance.
• Deep Apache Spark tuning and Photon engine optimization.
• Proficiency in Python, SQL, and Scala for distributed computing.
• Terraform IaC to deploy Databricks workspaces and permissions.

Responsibilities

• Lead design, implementation, and optimization of end-to-end Databricks solutions (Unity Catalog, Delta Lake, SQL, Workflows, Jobs, Mosaic AI).
• Architect and maintain scalable data ingestion and orchestration pipelines using Spark, Delta Live Tables, and Auto Loader.
• Identify and resolve Spark, cluster, and SQL performance bottlenecks; implement cost optimization and auto-scaling.
• Establish and enforce Databricks security, governance, and Unity Catalog best practices.
• Support data science and ML teams by enabling MLflow, Vector Search, and AI capabilities on Databricks.
• Provide root cause analysis and rapid resolution for production workloads; mentor engineers and analysts.

Skills

Spark tuning

Python

SQL

Databricks

Terraform

CI/CD

Education

Bachelor's degree or equivalent

Tools

Unity Catalog

Delta Lake

Auto Loader

MLflow

Vector Search

Job description

Cloud Analytics Technologies, LLC seeks a senior Databricks Architect to drive adoption, implementation, and optimization of the Databricks Lakehouse across data pipelines, analytics, and AI initiatives. The role collaborates with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and troubleshoot complex issues.

Responsibilities include leading end-to-end Databricks design, building ingestion pipelines with Spark and Delta Live Tables, enforcing