Principal Data Engineer

Jobtailor; Massachusetts; Hybrid; USD 140,000 - 190,000; Full time

Job summary

Jobtailor is seeking a senior data engineer to own and deliver Bronze/Silver/Gold lakehouse pipelines powering our NBA platform. You will write production SQL, Python, and Spark code, and drive data modeling for model training and batch scoring.

You will lead a small team, manage contractors, and collaborate with the Principal Decision Intelligence Engineer to define feature requirements, ensuring data quality and timely delivery.

Qualifications

• Bachelor's degree in computer science or a relevant field.
• 10+ years of data engineering experience with at least 1–2 years in a tech lead or senior lead capacity.
• Deep hands-on experience with Databricks and Delta Lake.
• Strong SQL and Python skills; comfortable with Spark at scale.
• Experience designing and owning a medallion lakehouse architecture (Bronze/Silver/Gold or equivalent).
• Proven ability to lead a small team or pod through delivery and manage contractor resources.
• Strong data quality instincts.
• Clear communicator.
• Comfort using AI productivity tools (Claude, Copilot, or similar) to accelerate development and analysis.

Responsibilities

• Pipeline delivery – design, build, and own Bronze/Silver/Gold lakehouse pipelines powering the NBA platform.
• Hands-on development – write and ship production SQL, Python, and Spark code.
• Data modeling – own the Gold layer schema for model training and batch scoring.
• Data quality – build and maintain data quality checks and pipeline alerts.
• External data integration – onboard new data sources.
• Contractor management – manage data engineering contractors within your pod.
• Cross-functional partnership – collaborate with the Principal Decision Intelligence Engineer on feature requirements.

Skills

Leadership

Communication

Team management

Data quality

Cross-functional collaboration

Mentoring

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Delta Lake

SQL

Python

Spark

Job description

Responsibilities

• Pipeline delivery – design, build, and own the Bronze/Silver/Gold lakehouse pipelines that power the NBA platform
• Hands‑on development – actively write and ship production SQL, Python, and Spark code alongside your team
• Data modeling – own the Gold layer schema that serves as the canonical member feature set for model training and batch scoring
• Data quality – build and maintain data quality checks, null/sparsity monitoring, and pipeline alerting
• External data integration – onboard new data sources
• Contractor management – manage data engineering contractors within your pod
• Cross‑functional partnership – work closely with the Principal Decision Intelligence Engineer on feature requirements

Requirements

• Bachelor's degree in computer science or a relevant field
• 10+ years of data engineering experience with at least 1–2 years in a tech lead or senior lead capacity
• Deep hands‑on experience with Databricks and Delta Lake
• Strong SQL and Python skills; comfortable with Spark at scale
• Experience designing and owning a medallion lakehouse architecture (Bronze/Silver/Gold or equivalent)
• Proven ability to lead a small team or pod through delivery and manage contractor resources
• Strong data quality instincts
• Clear communicator
• Comfort using AI productivity tools (Claude, Copilot, or similar) to accelerate development and analysis

Core Competencies

Demonstrates expertise in designing and building lakehouse pipelines, with strong capabilities in SQL, Python, and Spark for data engineering. Proven leadership in managing teams and contractors while ensuring data quality and integration.