Remote Lead Data Engineer — Spark & Medallion Architecture

Logicbroker; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Logicbroker is seeking a Lead Data Engineer to drive analytics and data platform initiatives. This hands-on technical lead will translate engineering specs into executable work, coordinate contractors, and ensure high-quality progress, partnering with an Engineering Manager on architectural direction.

The role emphasizes execution leadership, refining specs into agile workloads, and clear communication of progress and risks.

Qualifications

• Strong hands-on experience with Databricks, PySpark and Spark SQL.
• Proven experience implementing Medallion architecture.
• Experience leading delivery for small engineering teams.
• Comfortable working with partially defined requirements and refining them.
• Excellent written and verbal communication skills.

Responsibilities

• Lead a small delivery team and act as a player-coach by owning the most complex data transformations and modeling work, while reviewing and guiding contractor contributions.
• Keep work unblocked, well-scoped, and on track to ensure consistent delivery of momentum.
• Continue building and refining analytics foundations driven by Gold-layer data models.
• Design and implement Medallion architecture transformations (Bronze -> Silver -> Gold).
• Ensure Gold-layer outputs are well-modeled, performant, and consumable by an external analytics engine.
• Improve the reliability, clarity, and maintainability of existing pipelines.
• Partner with the Engineering Manager to execute a Data Warehouse Unification Plan RFC.
• Plan and deliver sprint-based execution with RFC-defined scope.
• Work across multiple Databricks instances as part of a GDPR-first data strategy.
• Translate engineering specifications into tickets and sprint plans.
• Assign work across contractors based on skill and complexity.
• Implement complex Spark and Databricks transformations as needed.
• Review pull requests and uphold quality standards.
• Communicate progress, blockers, and clarity needs clearly with stakeholders.
• Maintain delivery momentum with minimal overhead.

Skills

Databricks

PySpark

Spark SQL

Medallion architecture

Data modeling

Team leadership

Requirement refinement

Communication

Job description

Logicbroker is seeking a Lead Data Engineer to drive analytics and data platform initiatives. This hands-on technical lead will translate engineering specs into executable work, coordinate contractors, and ensure high-quality progress, partnering with an Engineering Manager on architectural direction.

The role emphasizes execution leadership, refining specs into agile workloads, and clear communication of progress and risks.