Remote Lead, Data Platform & ML Systems

Lighthouse Life Solutions; United States; Remote; USD 140,000 - 190,000; Full time

Benefits offered by this job

Remote‑first

Health benefits

401(k)

Flexible PTO

Job summary

Lighthouse Life Solutions seeks an experienced Data Engineer/Platform Engineer to own and evolve a Databricks lakehouse on AWS. You will manage data pipelines, models, and governance across owners, ensuring compliance with data sharing rules and regulatory needs.

This role is remote-first with periodic travel and reports to the CTO. You will extend data sources, build new models, and collaborate across teams to maintain reliable, auditable data products and pipelines.

Qualifications

• Bachelor’s degree in Computer Science, Statistics, Mathematics, Engineering, or related field.
• 5+ years building production data pipelines with SQL and Python.
• Strong experience with Spark or PySpark and dbt for transformation and testing.
• Hands‑on Databricks experience including Unity Catalog, Delta Lake, and Lambda/AI features.
• Experience with AWS and infrastructure as code (Terraform or OpenTofu).
• CI/CD pipeline ownership with GitHub Actions and automated testing.
• Machine learning engineering including feature pipelines, MLflow, and model serving.
• Regulated data handling: HIPAA, PHI/PII protection, audit logs.

Responsibilities

• Operate and improve pipelines feeding data from operations to the lakehouse.
• Extend the lakehouse with new data sources and governance from day one.
• Enforce ownership, sharing, and ingestion boundaries across pipelines and models.
• Own dbt transformation layers and curated data products.
• Govern the lakehouse with Unity Catalog: access controls and lineage.
• Operate platform as code via CI/CD pipelines and cloud tooling.
• Maintain data quality: automated tests, freshness checks, anomaly detection.
• Develop document ingestion with Databricks AI/ Textract and expand over time.
• Extend ML models from design to production and monitor lifecycle with MLflow.
• Run real-time and batch inference via Databricks Model Serving.

Skills

Data pipelines

SQL

Python

Spark / PySpark

dbt

Databricks

Unity Catalog

AWS

Terraform / OpenTofu

CI/CD

MLflow

Model governance

HIPAA / PHI / PII handling

AI-assisted development

Education

Bachelor’s degree in Computer Science, Statistics, Mathematics, Engineering, or related field

Tools

Databricks

Unity Catalog

Delta Lake

Lakeflow

Databricks SQL

Databricks AI functions

GitHub Actions

MLflow

Job description

Lighthouse Life Solutions seeks an experienced Data Engineer/Platform Engineer to own and evolve a Databricks lakehouse on AWS. You will manage data pipelines, models, and governance across owners, ensuring compliance with data sharing rules and regulatory needs.

This role is remote-first with periodic travel and reports to the CTO. You will extend data sources, build new models, and collaborate across teams to maintain reliable, auditable data products and pipelines.