Databricks Platform Lead for Data & AI

Apex Systems; Washington (District of Columbia); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

ESPP

401K program

HSA

EAP with counseling

Certification discounts

Job summary

Everforth Apex Systems in Washington, DC is seeking an accomplished Databricks Database Manager II to lead the Databricks Lakehouse platform across data pipelines, analytics, and AI initiatives. The role requires collaborating with data engineering, data science, analytics, and infrastructure teams to design scalable solutions.

You will champion security, Unity Catalog governance, and Spark tuning, while enabling MLflow and Mosaic AI capabilities.

Qualifications

• Bachelor's degree or equivalent work experience.
• US Citizenship required.
• 7+ years in data engineering/architecture with 4+ years in the Databricks ecosystem.
• Proven Unity Catalog from scratch with row- and column-level security and cross-workspace governance.
• Deep Apache Spark tuning expertise (skew, shuffles, AQE, caching, Photon).
• Hands-on experience building production-grade Generative AI/LLM apps using Mosaic AI, Vector Search, MLflow.
• Advanced Python/Scala/SQL for distributed computing.
• Terraform IaC for deploying Databricks workspaces, clusters, permissions.
• CI/CD pipelines with GitHub Actions, Azure DevOps, GitLab (DABs/CLI).
• Security: Private Link/VPC, BYOK keys, IP allowlists.

Responsibilities

• Lead end-to-end design and optimization of Databricks solutions (Unity Catalog, Delta Lake, Databricks SQL, Workflows).
• Architect and maintain scalable data ingestion, transformation, and orchestration pipelines.
• Identify and resolve Spark performance bottlenecks and optimize costs.
• Enforce Databricks security, governance, and Unity Catalog best practices.
• Support ML/AI initiatives with MLflow, Vector Search, and Mosaic AI capabilities.
• Provide root cause analysis and rapid resolution for production workloads.
• Mentor data engineers and developers through training and centers of excellence.
• Design integrations between Databricks and cloud services, ETL/BI tools, and data catalogs.
• Implement monitoring, logging, alerts, and dashboards for Databricks workloads.
• Evaluate features and roadmaps to drive business value.

Skills

Data engineering

Databricks

Spark tuning

Python/Scala/SQL

Terraform IaC

CI/CD pipelines

Unity Catalog

Security & compliance

Generative AI

Cloud networking

Education

Bachelor's degree or equivalent

Tools

Delta Live Tables

Auto Loader

MLflow

Vector Search

Databricks CLI

Databricks Asset Bundles (DABs)

Terraform

Photon engine

Job description

Everforth Apex Systems in Washington, DC is seeking an accomplished Databricks Database Manager II to lead the Databricks Lakehouse platform across data pipelines, analytics, and AI initiatives. The role requires collaborating with data engineering, data science, analytics, and infrastructure teams to design scalable solutions.

You will champion security, Unity Catalog governance, and Spark tuning, while enabling MLflow and Mosaic AI capabilities.