Senior Databricks Architect & AI Solutions

Artifintech; Washington (District of Columbia); On-site; USD 150,000 - 210,000; Full time

Job summary

Artifintech in the United States seeks a lead Databricks Engineer who will serve as the principal technical authority for the Databricks Lakehouse platform.

You will drive adoption, implementation, and optimization across data pipelines, analytics, and AI initiatives, collaborating with data engineering, data science, analytics, and infrastructure teams to design robust, scalable solutions and troubleshoot complex issues.

Qualifications

• Bachelor's degree or equivalent work experience.
• 7+ years in Data Engineering/Architecture with 4+ years in Databricks.
• Experience implementing Unity Catalog with row- and column-level security.
• Deep Spark tuning and performance optimization.
• Experience with MLflow, Vector Search, and Mosaic AI on Databricks.
• Proficiency in Python, Scala, SQL for distributed computing.

Responsibilities

• Lead design and optimization of end-to-end Databricks solutions (Unity Catalog, Delta Lake, Databricks SQL, Workflows, Jobs, Mosaic AI).
• Build scalable data ingestion, transformation, and orchestration pipelines (Spark, Delta Live Tables, Auto Loader).
• Identify and resolve Spark performance bottlenecks; implement auto-scaling and cost optimization.
• Establish Databricks security, governance, and Unity Catalog best practices.
• Support ML teams with MLflow, Vector Search, and AI capabilities.
• Provide root cause analysis and mentor engineers; create internal centers of excellence.

Skills

Databricks

Python

SQL

Spark

Unity Catalog

Delta Lake

MLflow

Vector Search

Mosaic AI

Terraform

GitHub Actions

Azure DevOps

Databricks CLI

Education

Bachelor's degree in IT/CS or related field

Tools

Terraform

Databricks CLI

GitHub Actions

Azure DevOps

Job description

Artifintech in the United States seeks a lead Databricks Engineer who will serve as the principal technical authority for the Databricks Lakehouse platform.

You will drive adoption, implementation, and optimization across data pipelines, analytics, and AI initiatives, collaborating with data engineering, data science, analytics, and infrastructure teams to design robust, scalable solutions and troubleshoot complex issues.