Lead Digital Data Engineer (Databricks)

BJ's Wholesale Club, Inc.; Marlborough (MA); Hybrid; USD 129,000 - 164,000; Full time

Benefits offered by this job

Weekly Pay

Daily Pay

Free BJ’s Memberships

Generous Paid Time Off

Flexible and Affordable HealthBenefits

401(k) Retirement Savings Plan

Employee Stock Purchase Plan

Job summary

BJ’s Wholesale Club is seeking a Lead Digital Data Engineer to own the design, build, and evolution of our Databricks Lakehouse. You will turn digital, fulfillment, and operational data into trusted, governed data products and drive AI-powered analytics adoption across the business.

You will lead a distributed team of up to 10 engineers, collaborate with product managers and senior leaders, and shape the data and analytics roadmap while optimizing performance and cost.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or equivalent, plus 7+ years of experience in data engineering, including 3+ years of hands-on Databricks experience in production.
• Experience with Spark (PySpark/SQL), Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong ability to build and maintain data pipelines across AWS services (S3, Kafka, DynamoDB, Lambda, IAM) and cloud platforms.

Responsibilities

• Design and maintain a scalable Lakehouse architecture (Delta Lake, Bronze/Silver/Gold) as single source of truth.
• Build batch and streaming pipelines integrating AWS services with Databricks.
• Ingest and unify data from digital and transactional sources; ensure data quality and governance.
• Lead AI/BI dashboards and Databricks agents use cases (Genie Spaces, Mosaic AI, MLflow).
• Oversee CI/CD for data pipelines (Git, Terraform) and drive Agile delivery with distributed teams.

Skills

Apache Spark

PySpark

SQL

Delta Lake

Unity Catalog

Databricks Workflows

SQL Warehouses

AWS S3

Kafka

DynamoDB

Lambda

IAM

Genie Spaces

AI dashboards

Mosaic AI

MLflow

CI/CD

Git workflows

Terraform

Agile Scrum

Data governance

Databricks agents

Education

Bachelor's degree in Computer Science, Information Technology, or equivalent

Tools

Databricks

Terraform

Git

Job description

A World-Class Team BJ’s Wholesale Club is powered by more than 30,000 team members who make a real impact every day. Whether you're stocking shelves, solving problems or shaping strategy, your work helps families save on what matters most. We’re a team built on purpose and opportunity. Join us and be part of something meaningful.

Why You’ll Love Working at BJ’s

At BJ’s Wholesale Club, our team members are at the heart of everything we do. That’s why we offer a comprehensive benefits package designed to support your health, well-being and future – both on and off the job. When you grow, we grow.

• Weekly Pay: Get paid every week so that you can manage your money on your terms.
• Daily Pay: Access your earned pay before payday, giving you more flexibility when you need it. (fee may apply.)
• Free BJ’s Memberships: Enjoy a complimentary The Club Card Membership, plus a free Supplemental Membership for someone in your household.*
• Generous Paid Time Off: Take the time you need with vacation, personal, sick days, holidays, bereavement, and jury duty leave.*
• Flexible and Affordable Health Benefits: Choose from three medical plans, and access optional dental, vision, Health Savings Account (HSA), and flexible spending account options to fit your lifestyle.*
• 401(k) Retirement Savings Plan: Build your financial future with a company match (available to team members 18 and older).*
• Employee Stock Purchase Plan: Accumulate funds through after-tax payroll deductions that can be used to purchase shares of BJ’s common stock at a 15% discount.*
• *Eligibility requirements vary by position.

Role Summary

As Lead Digital Data Engineer, you will own the design, build, and evolution of BJ's Databricks Lakehouse, working closely with product managers, program managers, and business unit heads. You will turn digital, fulfillment, and operational data into trusted, governed, self-service data products, and lead our adoption of AI-powered analytics, including Genie and Databricks agents.

Key Responsibilities

• Design and maintain a scalable Lakehouse using Delta Lake and the medallion (Bronze/Silver/Gold) architecture, serving as the single source of truth for analytics following the migration from AWS Redshift.
• Build batch and real-time pipelines integrating Kafka, DynamoDB, S3, and other AWS services with Databricks (Structured Streaming, Auto Loader, Delta Live Tables).
• Ingest and integrate digital sources such as Adobe Analytics, AEP, and Google Analytics alongside transactional and operational data.
• Implement data quality validation frameworks with automated checks, alerting, and SLAs so data is accurate and clearly represented across business verticals.
• Lead data governance through Unity Catalog (access control, lineage, classification) and monitor pipeline health, freshness, and cost.
• Design reusable Genie Spaces and AI/BI dashboards with certified metrics for business self-service.
• Develop and productionize Databricks agents and AI use cases using Mosaic AI, MLflow, Model Serving, and Vector Search.
• Optimize performance and cost across clusters, SQL Warehouses, and Delta tables.
• Establish CI/CD for data pipelines (Databricks Asset Bundles, Git workflows, automated testing, Terraform).
• Own and prioritize the digital data and analytics roadmap with product leaders.
• Lead and mentor a distributed team of up to 10 engineers across onshore and offshore locations, driving Agile delivery and providing regular status and risk updates to senior leadership.

This is a hybrid role. Tuesday through Thursday are in-office days at BJ's Club Support Center in Marlborough, MA and Monday and Friday are remote days.

In accordance with the Pay Transparency requirements, the following represents a good faith estimate of the compensation range for this position. At BJ’s Wholesale Club, we carefully consider a wide range of non-discriminatory factors when determining salary. Actual salaries will vary depending on factors including but not limited to location, education, experience, and qualifications. The pay range for this position is $128,500.00 - $164,000.00

We recognize the growing role of AI tools, including ChatGPT, and value familiarity with them. That said, we want to hear from your authentic self. Your application should reflect your own skills, experiences, and insights rather than AI-generated responses.

Minimum Requirements

• Bachelor's degree in Computer Science, Information Technology, or equivalent, plus 7+ years of experience in data engineering, including 3+ years of hands‑on Databricks experience in production.
• Experience must include:
• Expert-level Apache Spark (PySpark and SQL), Delta Lake, and Lakehouse architecture Unity Catalog, Databricks Workflows, and SQL Warehouses
• Integrating AWS services: S3, Kafka (MSK/Confluent), DynamoDB, Lambda, and IAM
• Building data quality and validation frameworks
• Genie Spaces, AI/BI dashboards, and building Databricks agents or GenAI solutions (Mosaic AI, MLflow)
• Building large-scale distributed data systems on AWS, Azure, or GCP
• Working in an Agile Scrum environment and leading distributed teams
• SDLC, CI/CD, version control, and test automation for data pipelines

Who We Are

BJ’s Wholesale Club is driven by a powerful purpose: we take care of the families who depend on us. More than 8 million members rely on us for unbeatable value and convenience. Headquartered in Marlborough, Massachusetts, BJ’s is a leading membership warehouse club with over 250 clubs, nearly 200 BJ’s Gas® stations and eight distribution centers across 21 states – and growing. BJ’s offers the highest-quality fresh produce, meat, deli and bakery items as well as apparel, electronics, home goods and more – all at unbeatable value. Our members shop how and when they want, whether it’s in our clubs, online or with curbside pickup.