Databricks Architect for AI-Powered Data Pipelines

Robotics Technologies LLC; Washington (District of Columbia); On-site; USD 170,000 - 230,000; Full time

Job summary

Robotics Technologies LLC in Washington, DC seeks a senior Databricks Architect to own the platform strategy across data pipelines, analytics, and AI initiatives. You will collaborate with data engineers, data scientists, and infrastructure teams to design robust, scalable solutions.

You will collaborate with data engineers, data scientists, and infrastructure teams to design robust, scalable solutions, optimize Spark workloads, and implement Unity Catalog and Delta Lake.

Qualifications

• 7+ years in data engineering or architecture.
• Deep, low-level Apache Spark tuning expertise.
• Proven track record implementing Unity Catalog from scratch.
• Experience with Spark performance optimization techniques (skew, shuffles, AQE, caching).
• Hands-on with Generative AI on Databricks (MLflow, Vector Search).
• Proficiency in Python, Scala, and SQL for distributed computing.
• IaC experience with Terraform for Databricks deployments.
• CI/CD pipelines using GitHub Actions, Azure DevOps, or GitLab.

Responsibilities

• Lead design, implementation, and optimization of end-to-end Databricks solutions.
• Architect and maintain scalable data ingestion, transformation, and orchestration pipelines.
• Resolve performance bottlenecks in Spark jobs, clusters, and SQL queries; optimize costs.
• Enforce Databricks security, governance, and Unity Catalog practices.
• Support ML/AI teams with MLflow, Vector Search, and Mosaic AI capabilities.
• Provide root cause analysis for production issues as escalation point.
• Mentor engineers and developers through training and internal centers of excellence.
• Design integrations between Databricks and cloud platforms, ETL/tools, BI tools.

Skills

Leadership

Mentoring

Cross-functional collaboration

Problem solving

Education

Bachelor's degree or equivalent in IT/CS/Engineering

Tools

Databricks

Spark

Delta Lake

Unity Catalog

Delta Live Tables

Auto Loader

MLflow

Vector Search

Photon

Terraform

Databricks CLI

Databricks Asset Bundles (DABs)

GitHub Actions

Azure DevOps

GitLab

Private Link

VPC Peering

Job description

Robotics Technologies LLC in Washington, DC seeks a senior Databricks Architect to own the platform strategy across data pipelines, analytics, and AI initiatives. You will collaborate with data engineers, data scientists, and infrastructure teams to design robust, scalable solutions.

You will collaborate with data engineers, data scientists, and infrastructure teams to design robust, scalable solutions, optimize Spark workloads, and implement Unity Catalog and Delta Lake.