Remote Databricks Data Engineering Lead

Quantum Technologies. LLC; New York (NY); Remote; USD 84,000 - 120,000; Full time

Job summary

Quantum Technologies. LLC is seeking an experienced Databricks Data Engineering Lead for remote work from New Jersey. The role requires 8+ years in data engineering, with strong hands-on Databricks, Python, PySpark, and Delta Lake expertise.

You will guide architecture, design scalable pipelines, and mentor junior engineers across teams. Responsibilities include leading enterprise-scale data solutions, optimizing Lakehouse architectures, and delivering secure, high-performance data platforms on

Qualifications

• 8+ years of data engineering experience.
• Expertise in Databricks on AWS.
• Strong Python and PySpark skills.
• Experience with Delta Lake and Lakehouse patterns.
• Ability to lead and mentor engineers.

Responsibilities

• Lead the design, development, and implementation of enterprise-scale data engineering solutions on Databricks.
• Design and develop scalable ETL/ELT pipelines using Databricks, Python, and PySpark.
• Provide hands-on technical leadership for Databricks Genie implementations and use cases.
• Design and optimize data architectures leveraging Delta Lake and Lakehouse architectural patterns.
• Build reusable frameworks and engineering components for data ingestion, transformation, validation, and processing.
• Develop and optimize complex PySpark workloads for performance, reliability, and scalability.
• Implement data quality, governance, security, monitoring, and operational controls across data pipelines.
• Provide technical leadership through architecture reviews, design reviews, code reviews, troubleshooting, and engineering best practices.
• Mentor data engineers and provide technical guidance during development and implementation.
• Work closely with architects, business analysts, data teams, and application teams to translate requirements into technical solutions.
• Support production deployments, performance tuning, incident resolution, and continuous improvement.
• Contribute to technical documentation, solution design, deployment strategy, and engineering standards.

Skills

Technical leadership

Engineering standards

Mentoring

Architecture reviews

Tools

Databricks

Databricks Genie

Python

PySpark

Delta Lake

Lakehouse architecture

SQL

Job description

Quantum Technologies. LLC is seeking an experienced Databricks Data Engineering Lead for remote work from New Jersey. The role requires 8+ years in data engineering, with strong hands-on Databricks, Python, PySpark, and Delta Lake expertise.

You will guide architecture, design scalable pipelines, and mentor junior engineers across teams. Responsibilities include leading enterprise-scale data solutions, optimizing Lakehouse architectures, and delivering secure, high-performance data platforms on