Lead Databricks Architect for Conversational Data Platform

Valiant Technologies LLC; Princeton (NJ); On-site; USD 93,000 - 133,000; Full time

Job summary

Valiant Technologies LLC in Princeton, NJ seeks a hands-on Databricks Architect to lead end-to-end design of governed, conversational self-service data solutions on Databricks, spanning from source data to business-ready answers.

You will guide architecture with a federated data platform and central semantic layer, integrating a governance framework for metrics, lineage, and access policies, while mentoring a small engineering team and aligning with business and technology leaders.

Qualifications

• 10–12 years of experience in data engineering and data architecture.
• At least 3 years as a hands-on architect on a delivery team.
• Proficient with Databricks, Unity Catalog, Delta Lake, and data governance.

Responsibilities

• Own the end‑to‑end architecture for conversational self‑service on Databricks.
• Design a federated architecture with a central hub for shared definitions and units that retain data and context.
• Define semantic and context layers, including ontology, keys, metrics, rules, and lineage.
• Publish local definitions to the hub and map to a common model without copying data.
• Establish governance for metric changes, ownership, versioning, and approvals.
• Build Unity Catalog metric views, SQL functions for rules, and ABAC policies.
• Configure Genie spaces, agents, and integration tooling for AI workflows.
• Create Agent Bricks solutions routing questions to appropriate agents.
• Leverage Lakehouse Federation, Delta Sharing, and clean rooms as needed.
• Manage metric views and agent configs in Git with CI/CD via Databricks Asset Bundles.
• Assess answer accuracy with MLflow tracing and evaluation; present architecture decisions to leaders.

Skills

Databricks

SQL

Python

PySpark

Git

CI/CD

Unity Catalog

Delta Lake

Delta Sharing

Lakehouse Federation

Leadership

Tools

Git

CI/CD

Job description

Valiant Technologies LLC in Princeton, NJ seeks a hands-on Databricks Architect to lead end-to-end design of governed, conversational self-service data solutions on Databricks, spanning from source data to business-ready answers.

You will guide architecture with a federated data platform and central semantic layer, integrating a governance framework for metrics, lineage, and access policies, while mentoring a small engineering team and aligning with business and technology leaders.