Principal Data Architect

BHSI; Boston (MA); On-site; USD 200,000 - 240,000; Full time

Benefits offered by this job

Health, Dental and Vision benefits

401(k) plan with company match

Paid time off

Job summary

Berkshire Hathaway Specialty Insurance (BHSI) in Boston is seeking a Principal Data Architect to own the Databricks Lakehouse architecture across data engineering, analytics, and AI/ML. This hands-on senior IC role reports to the Global Head of Data Analytics Solutions and collaborates with portfolio leaders and business stakeholders.

You will set technology direction, ensure security, performance, and cost efficiency, and drive a phased roadmap with engineering teams.

Qualifications

• 15+ years in data engineering or architecture roles.
• Databricks Lakehouse Platform experience required.
• Proficient in Python and SQL; IaC with Terraform.

Responsibilities

• Define and evolve the enterprise Lakehouse reference architecture on Databricks for batch, streaming, and AI workloads.
• Lead design and rollout of Databricks AI capabilities, including vector search and model serving.
• Implement lineage, access controls, and auditing with Unity Catalog.
• Establish CI/CD and IaC standards for data pipelines and ML models; optimize Spark performance.
• Set standards, run reviews, and mentor engineers and data scientists.

Skills

Data architecture

Databricks Lakehouse

Spark

Delta Lake

Unity Catalog

MLflow

Python

SQL

Terraform

AWS

Azure

Stakeholder influence

CI/CD for data pipelines

MLOps

Architecture leadership

Education

Databricks Certification

Tools

Databricks

Terraform

Job description

Who are we?

A strategic and trusted insurance partner, Berkshire Hathaway Specialty Insurance (BHSI), provides a broad range of commercial property, casualty and specialty insurance coverages and outstanding service to customers and brokers around the world. Part of Berkshire Hathaway’s insurance operations, we bring our solutions to market with our stellar brand name, top-rated balance sheet, and the expertise of our global team of professionals, who exude excellent capabilities and strong character.

We are a values-based organization where respect, integrity, excellence, collaboration, and passion define who we are and how we do business.

We value diversity of backgrounds, experience, and perspectives and strive to foster an inclusive environment that enables all our team members to bring their best selves to work.

We are one team committed to building a culture where every teammate has the opportunity to contribute and be recognized. Want to be part of the team building the finest property, casualty, and specialty lines insurance company in the world?

Learn more about our unique culture and history.

Job Opportunity

BHSI is modernizing its data ecosystem, moving from a monolithic, reporting-centric infrastructure to a Lakehouse platform that supports insight-driven decisions and AI-enabled outcomes at the speed of business. We are seeking a Principal Data Architect to set the technical direction for that platform.

You will own the architecture of our Databricks-based Lakehouse across data engineering, analytics, and AI/ML, ensuring it is secure, performant, cost-efficient, and ready for Generative AI. This is a hands-on, senior individual-contributor role for a technologist and strategic thinker, not a people-management position. You will report to the Global Head of Data Analytics Solutions and partner with Enterprise Technology portfolio leaders and business stakeholders.

Responsibilities

• Platform Architecture & Strategy: Define and evolve the enterprise Lakehouse reference architecture on Databricks, supporting batch, streaming, and AI workloads at scale across AWS or Azure.
• AI & Generative AI Enablement: Lead the design and rollout of Databricks AI capabilities, including vector search, model serving, and LLM orchestration, so teams can build, fine-tune, and deploy models securely against enterprise data.
• Governance & Security by Design: Implement lineage, access controls, and auditing with Unity Catalog to meet regulatory requirements and ensure data quality across domains.
• MLOps & Engineering Excellence: Establish CI/CD and infrastructure-as-code standards for data pipelines and ML models. Optimize Spark performance, cluster configuration, and compute spend.
• Technical Leadership & Influence: Set architecture standards, run design reviews, and mentor engineers and data scientists. Translate business strategy into a phased technical roadmap and build alignment with engineering and business leaders.

Qualifications, Skills And Experience

• 15+ years in data engineering, data architecture, or software architecture, including 5+ years architecting on the Databricks Lakehouse Platform
• Deep expertise in Apache Spark, Delta Lake, Unity Catalog, and MLflow
• Proven experience designing infrastructure for both traditional ML and Generative AI workloads (RAG, fine-tuning, vector search, agentic frameworks)
• Strong proficiency in Python and SQL, infrastructure-as-code (e.g., Terraform), and software design patterns
• Hands-on architecture experience on AWS or Microsoft Azure
• Demonstrated ability to influence senior technical and business stakeholders without direct authority

Preferred Qualifications

• Databricks Professional or Expert-level certification
• Experience with advanced agentic architectures, DSPy, or graph databases
• Background in insurance, financial services, fintech, or other regulated environments
• Track record of optimizing cloud compute costs while improving pipeline throughput

BHSI Offers

• A competitive package and exciting growth opportunities for career-oriented teammates
• A dynamic, action-oriented, and thoughtful environment centered on always doing the right thing for our customers, teammates and our other stakeholders
• A purposely non-bureaucratic organization that embraces simplicity over complexity and emphasizes individual excellence in a team framework
• Benefits that support your life and well-being, which include:
• Comprehensive Health, Dental and Vision benefits
• Disability Insurance (both short-term and long-term)
• Life Insurance (for you and your family)
• Accidental Death & Dismemberment Insurance (for you and your family)
• Flexible Spending Accounts
• Health Reimbursement Account
• Employee Assistance Program
• Retirement Savings 401(k) Plan with Company Match
• Generous holiday and Paid Time Off
• Tuition Reimbursement
• Paid Parental Leave

The base salary range for this position in Boston is $200,000 to $240,000, along with annual bonus eligibility; a candidate’s salary is determined by their relevant skills and experience. We value our teammates – both their capabilities and character – as demonstrated by our amazing culture.

NOTE: Compensation will be commensurate with experience. This job description is not intended to be all-inclusive. Team Member may perform other related duties as negotiated to meet the ongoing needs of the organization.