Lead Databricks Lakehouse & AI Data Platform Architect

BJ's Wholesale Club; Marlborough (MA); Hybrid; USD 129,000 - 164,000; Full time

Job summary

BJ's Wholesale Club in Marlborough, MA is seeking a Lead Digital Data Engineer to own the design, build, and evolution of the Databricks Lakehouse. You will turn digital, fulfillment, and operational data into trusted, governed data products, and lead AI-powered analytics adoption.

You will design scalable lakehouse architectures, implement data pipelines with AWS services, govern data with Unity Catalog, and mentor a distributed engineering team in Agile delivery.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or equivalent.
• 7+ years of experience in data engineering.
• 3+ years of hands-on Databricks experience in production.
• Expert-level Apache Spark (PySpark and SQL) and Delta Lake experience.
• Experience building large-scale distributed data systems on AWS/Azure/GCP.

Responsibilities

• Design and maintain a scalable Lakehouse using Delta Lake and medallion architecture, serving as the single source of truth for analytics.
• Build batch and real-time pipelines integrating AWS services with Databricks (Structured Streaming, Auto Loader, Delta Live Tables).
• Ingest and integrate analytics and transactional data to enable governed, self-service data products.
• Lead data governance through Unity Catalog and monitor pipeline health, freshness, and cost.
• Design Genie Spaces and AI/BI dashboards with certified metrics for self-service analytics.
• Develop and productionize Databricks agents and GenAI solutions (Mosaic AI, MLflow).
• Establish CI/CD for data pipelines and automate testing with Terraform.

Skills

Apache Spark (PySpark/SQL)

Delta Lake

Lakehouse architecture

Unity Catalog

Databricks Workflows

SQL Warehouses

AWS services (S3, Kafka, DynamoDB)

Genie Spaces / GenAI dashboards

CI/CD / Terraform

Agile/Scrum leadership

Education

Bachelor's degree in Computer Science / Information Technology or equivalent

Tools

Databricks

Mosaic AI

MLflow

Kafka

DynamoDB

S3

Lambda

Job description

BJ's Wholesale Club in Marlborough, MA is seeking a Lead Digital Data Engineer to own the design, build, and evolution of the Databricks Lakehouse. You will turn digital, fulfillment, and operational data into trusted, governed data products, and lead AI-powered analytics adoption.

You will design scalable lakehouse architectures, implement data pipelines with AWS services, govern data with Unity Catalog, and mentor a distributed engineering team in Agile delivery.