Senior Databricks Data & AI Solution Architect - Remote

Eliassen Group; Boston (MA); Remote; USD 138,000 - 145,000; Full time

Benefits offered by this job

Medical, Dental, Vision benefits

401k with company matching

Life insurance

Job summary

Eliassen Group is seeking a Senior Solution Architect to provide technical leadership for a large-scale Data & AI modernization initiative. You will own the overall architecture across Databricks platform modernization, data engineering, governance, observability, and AI/agent enablement in a remote, contract capacity.

This hands-on role requires moving between enterprise architecture, Databricks design, governance, and problem solving, leading senior engineers and aligning with platform

Qualifications

• Deep understanding of Databricks Lakeflow connect capabilities and AWS analytics and ingestion services.
• Extensive experience addressing ingestion challenges from Lakeflow connect sources including CDC handling and reconciliation techniques.
• Expertise with data quality libraries and configuration-based frameworks that plug DQ into ingestion pipelines.
• Extensive experience designing enterprise Databricks lakehouse architectures with Spark, Delta Lake, Lakeflow, and Unity Catalog.
• Strong experience with medallion architecture and Bronze/Silver/Gold data patterns.
• Experience designing enterprise data governance, lineage, security, and access-control frameworks including ABAC and RBAC, data classification, tagging, and least-privilege security.
• Experience designing production-grade data pipelines, reusable ingestion frameworks, observability, CI/CD, deployment automation, and production support.
• Experience designing semantic layers and governed enterprise data products, and architecting AI or GenAI solutions operating against governed enterprise data, including Databricks Genie.
• Experience with MCP and agent-based architectures, GenAI governance, tracing, and evaluation preferred.
• Strong communication skills and ability to work across engineering, security, architecture, and business teams.

Responsibilities

• Lead a Databricks Well‑Architected assessment across platform architecture, observability, pipelines, consumption, data quality, security, governance, and cost.
• Define target‑state architecture and technical standards for the Data & AI platform.
• Identify architectural gaps, technical debt, platform risks, and remediation priorities.
• Establish architectural patterns supporting scalable data ingestion, transformation, analytics, AI, and agent‑based solutions.
• Guide engineering teams in implementing standards and remediation priorities.
• Establish standards for development, test, production promotion, CI/CD, monitoring, and operational support.
• Design and implement governance standards leveraging Databricks Unity Catalog.
• Define least‑privilege access models, attribute‑based access control policies, data classification, tagging, and lineage standards.
• Establish monitoring for system health, jobs, pipelines, audit activity, data quality, and platform performance.
• Implement platform approaches for cost monitoring and optimization.
• Define data architecture standards for Bronze, Silver, and Gold medallion layers.
• Establish reusable patterns for source ingestion, transformation, data quality, lineage, and deployment.
• Architect configuration‑driven onboarding frameworks using Databricks Lakeflow, Asset Bundles, YAML/control tables, and automated CI/CD.
• Guide development of conformed dimensions, shared business keys, and cross‑domain data products.
• Define the architecture for an AI‑ready Gold layer with certified, governed, documented data products.
• Design semantic and context layers using governed business definitions, metric views, KPIs, metadata, and business glossary information.
• Architect natural‑language analytics using Databricks Genie.
• Define agent‑ready interfaces using governed Unity Catalog functions, MCP endpoints, tracing, and evaluation.
• Provide architectural direction and technical mentorship to Data & AI Engineers.
• Partner with business SMEs and data owners to translate requirements into scalable technical solutions.
• Lead architecture and design reviews.
• Establish reusable engineering standards, reference architectures, runbooks, and implementation patterns.
• Participate in milestone and steering reviews.
• Deliver a secure, observable, governed, cost‑aware, AI‑ready Databricks architecture that can scale beyond a single use case.
• Produce assessment scorecards, remediation plans, platform standards, reusable data architecture patterns, and semantic and AI architecture assets.
• Enable operational readiness with monitoring, runbooks, documentation, and knowledge transfer.

Skills

Databricks Lakeflow

Lakeflow connect

AWS analytics

Data quality libraries

Medallion architecture

ABAC/RBAC

CI/CD

Governance & lineage

AI/GenAI governance

Databricks Unity Catalog

Tools

Databricks Unity Catalog

Job description

Eliassen Group is seeking a Senior Solution Architect to provide technical leadership for a large-scale Data & AI modernization initiative. You will own the overall architecture across Databricks platform modernization, data engineering, governance, observability, and AI/agent enablement in a remote, contract capacity.

This hands-on role requires moving between enterprise architecture, Databricks design, governance, and problem solving, leading senior engineers and aligning with platform