Lead Databricks Architect for Data & AI Platform

Quantum Technologies LLC; Princeton (NJ); On-site; USD 93,000 - 133,000; Full time

Job summary

Quantum Technologies. LLC seeks a hands-on Lead Data & AI Architect to design and build governed, conversational self-service data solutions on Databricks. Lead architecture from source data to answers, blending a federated data platform, semantic layer, and conversational AI.

Guide a small engineering team and contribute directly to implementation. You will establish shared data definitions, publish local definitions to the hub, and implement ABAC policies, Unity Catalog metric views, and CI/CD

Qualifications

• 10–12 years of experience in data engineering and data architecture, including at least 3 years as a hands-on architect on a delivery team.
• At least 3 years of production experience with Databricks, including Unity Catalog, Delta Lake, Databricks SQL, Lakehouse Federation, and Delta Sharing.
• Experience building a governed semantic or metrics layer—such as Databricks metric views or dbt metrics—used by multiple teams.
• Experience delivering a Genie, text-to-SQL, or LLM agent solution beyond a demo, including evaluation of answer accuracy.
• Strong SQL and Python skills, PySpark experience, and familiarity with Git and CI/CD for data assets.
• Experience with data modelling, conformed dimensions, common keys across sources, and reconciling metrics against a system of record.
• Experience with access control, lineage, data sharing, and versioning definitions across multiple teams or entities.
• Strong communication skills, with the ability to explain technical decisions clearly, lead workshops, and write effectively.

Responsibilities

• Own the end-to-end architecture for conversational self-service on Databricks.
• Design a federated architecture with a central hub for shared definitions and business units or entities that retain their data and context.
• Define the semantic and context layer, including the ontology, common keys, metric views, business rules, entitlements, certification status, lineage, and knowledge sources.
• Design how local definitions and metric views are published to the hub, mapped to a common model, and reused without copying data.
• Establish governance for metric changes and certification, including ownership, versioning, impact analysis, approvals, and effective dates.
• Build Unity Catalog metric views, SQL functions for business rules, and attribute‑based access control (ABAC) policies using row filters and column masks.
• Configure Genie spaces, Genie Agents, and Genie One, including instructions, example SQL, trusted assets, and benchmark questions.
• Build Agent Bricks solutions that route questions to the appropriate agent.
• Use Lakehouse Federation, Delta Sharing, and clean rooms when data needs to remain in place.
• Manage metric views and agent configurations in Git, with CI/CD using Databricks Asset Bundles.
• Measure answer accuracy using MLflow tracing and evaluation.
• Explain architecture decisions to business and technology leaders, run workshops, and guide engineers through delivery.

Skills

Databricks

SQL

Python

PySpark

Git

CI/CD

Data modelling

Governance

Communication

Leadership

Tools

Unity Catalog

Delta Lake

Delta Sharing

Lakehouse Federation

Databricks SQL

Job description

Quantum Technologies. LLC seeks a hands-on Lead Data & AI Architect to design and build governed, conversational self-service data solutions on Databricks. Lead architecture from source data to answers, blending a federated data platform, semantic layer, and conversational AI.

Guide a small engineering team and contribute directly to implementation. You will establish shared data definitions, publish local definitions to the hub, and implement ABAC policies, Unity Catalog metric views, and CI/CD