Databricks Lakehouse Architect

Valiant Technologies LLC; Washington (District of Columbia); On-site; USD 180,000 - 240,000; Full time

Job summary

Valiant Technologies LLC in Washington, DC is seeking a senior Databricks Architect to lead adoption, implementation, and optimization of the Databricks Lakehouse across data pipelines, analytics, and AI initiatives. You will collaborate with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and troubleshoot complex issues.

You will mentor engineers, automate pipelines with Spark, Delta Live Tables, and MLflow, enforce security and governance with

Qualifications

• Bachelor's degree in Information Technology, Computer Science, Business Administration, or a related field, or equivalent work experience.
• 7+ years in Data Engineering or Architecture, with at least 4+ years dedicated exclusively to the Databricks ecosystem.
• Proven track record implementing Unity Catalog from scratch, including row- and column-level security and cross-workspace governance.
• Deep, low-level Apache Spark tuning expertise (e.g., managing data skew, optimizing shuffles, utilizing Adaptive Query Execution, caching strategies, and Photon engine optimization).
• Hands-on experience building production-grade Generative AI or LLM applications using Databricks Mosaic AI, Vector Search, and MLflow.
• Advanced proficiency in Python, Scala, and SQL optimized specifically for distributed computing environments
• Production experience with Infrastructure as Code (IaC) tools like Terraform to deploy and manage Databricks workspaces, clusters, and permissions.
• Experience setting up secure network topologies for Databricks, including Private Link/VPC Peering , encryption keys (BYOK), and IP access lists.
• Strong background in CI/CD pipelines (e.g., GitHub Actions, Azure DevOps, GitLab) using Databricks Asset Bundles (DABs) or Databricks CLI.

Responsibilities

• Lead the design, implementation, and optimization of end-to-end solutions on Databricks, including Unity Catalog, Delta Lake, Databricks SQL, Workflows, Jobs, and Mosaic AI.
• Architect, build, and maintain scalable data ingestion, transformation, and orchestration pipelines using Databricks tools such as Spark, Delta Live Tables, and Auto Loader.
• Identify and resolve performance bottlenecks in Spark jobs, clusters, and SQL queries while implementing cost optimization and auto-scaling strategies.
• Establish and enforce Databricks best practices for security, data governance, compliance, and Unity Catalog implementation.
• Support data science and machine learning teams by implementing MLflow, Feature Store, Vector Search, and generative AI capabilities on Databricks.
• Provide root cause analysis and rapid resolution for complex technical issues as the escalation point for production workloads.
• Mentor data engineers, analysts, and developers through training sessions, documentation, and the creation of internal centers of excellence.
• Design seamless integrations between Databricks and cloud platforms, ETL tools, BI tools, data catalogs, and orchestration systems.
• Implement robust monitoring, logging, alerting, and dashboarding for Databricks workloads using native and external tools.
• Evaluate new Databricks features and roadmaps to recommend adoption strategies that drive business value.

Skills

Databricks

Spark

Unity Catalog

Python

Scala

SQL

Terraform

AWS

Azure

Education

Bachelor's degree

Tools

Databricks CLI

GitHub Actions

Azure DevOps

Job description

Valiant Technologies LLC in Washington, DC is seeking a senior Databricks Architect to lead adoption, implementation, and optimization of the Databricks Lakehouse across data pipelines, analytics, and AI initiatives. You will collaborate with data engineering, data science, analytics, and infrastructure teams to design scalable solutions and troubleshoot complex issues.

You will mentor engineers, automate pipelines with Spark, Delta Live Tables, and MLflow, enforce security and governance with