Lead Data Engineer – Lakehouse Architect & AI Tools

Jobtailor; Massachusetts; Hybrid; USD 140,000 - 190,000; Full time

Job summary

Jobtailor is seeking a senior data engineer to own and deliver Bronze/Silver/Gold lakehouse pipelines powering our NBA platform. You will write production SQL, Python, and Spark code, and drive data modeling for model training and batch scoring.

You will lead a small team, manage contractors, and collaborate with the Principal Decision Intelligence Engineer to define feature requirements, ensuring data quality and timely delivery.

Qualifications

• Bachelor's degree in computer science or a relevant field.
• 10+ years of data engineering experience with at least 1–2 years in a tech lead or senior lead capacity.
• Deep hands-on experience with Databricks and Delta Lake.
• Strong SQL and Python skills; comfortable with Spark at scale.
• Experience designing and owning a medallion lakehouse architecture (Bronze/Silver/Gold or equivalent).
• Proven ability to lead a small team or pod through delivery and manage contractor resources.
• Strong data quality instincts.
• Clear communicator.
• Comfort using AI productivity tools (Claude, Copilot, or similar) to accelerate development and analysis.

Responsibilities

• Pipeline delivery – design, build, and own Bronze/Silver/Gold lakehouse pipelines powering the NBA platform.
• Hands-on development – write and ship production SQL, Python, and Spark code.
• Data modeling – own the Gold layer schema for model training and batch scoring.
• Data quality – build and maintain data quality checks and pipeline alerts.
• External data integration – onboard new data sources.
• Contractor management – manage data engineering contractors within your pod.
• Cross-functional partnership – collaborate with the Principal Decision Intelligence Engineer on feature requirements.

Skills

Leadership

Communication

Team management

Data quality

Cross-functional collaboration

Mentoring

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Delta Lake

SQL

Python

Spark

Job description

Jobtailor is seeking a senior data engineer to own and deliver Bronze/Silver/Gold lakehouse pipelines powering our NBA platform. You will write production SQL, Python, and Spark code, and drive data modeling for model training and batch scoring.

You will lead a small team, manage contractors, and collaborate with the Principal Decision Intelligence Engineer to define feature requirements, ensuring data quality and timely delivery.