Lakehouse Data Architect for Generative AI & Databricks

BHSI; Boston (MA); On-site; USD 200,000 - 240,000; Full time

Benefits offered by this job

Health, Dental and Vision benefits

401(k) plan with company match

Paid time off

Job summary

Berkshire Hathaway Specialty Insurance (BHSI) in Boston is seeking a Principal Data Architect to own the Databricks Lakehouse architecture across data engineering, analytics, and AI/ML. This hands-on senior IC role reports to the Global Head of Data Analytics Solutions and collaborates with portfolio leaders and business stakeholders.

You will set technology direction, ensure security, performance, and cost efficiency, and drive a phased roadmap with engineering teams.

Qualifications

• 15+ years in data engineering or architecture roles.
• Databricks Lakehouse Platform experience required.
• Proficient in Python and SQL; IaC with Terraform.

Responsibilities

• Define and evolve the enterprise Lakehouse reference architecture on Databricks for batch, streaming, and AI workloads.
• Lead design and rollout of Databricks AI capabilities, including vector search and model serving.
• Implement lineage, access controls, and auditing with Unity Catalog.
• Establish CI/CD and IaC standards for data pipelines and ML models; optimize Spark performance.
• Set standards, run reviews, and mentor engineers and data scientists.

Skills

Data architecture

Databricks Lakehouse

Spark

Delta Lake

Unity Catalog

MLflow

Python

SQL

Terraform

AWS

Azure

Stakeholder influence

CI/CD for data pipelines

MLOps

Architecture leadership

Education

Databricks Certification

Tools

Databricks

Terraform

Job description

Berkshire Hathaway Specialty Insurance (BHSI) in Boston is seeking a Principal Data Architect to own the Databricks Lakehouse architecture across data engineering, analytics, and AI/ML. This hands-on senior IC role reports to the Global Head of Data Analytics Solutions and collaborates with portfolio leaders and business stakeholders.

You will set technology direction, ensure security, performance, and cost efficiency, and drive a phased roadmap with engineering teams.