Lead Data Platform and ML Manager

Lighthouse Life Solutions; United States; Remote; USD 140,000 - 190,000; Full time

Benefits offered by this job

Remote‑first

Health benefits

401(k)

Flexible PTO

Job summary

Lighthouse Life Solutions seeks an experienced Data Engineer/Platform Engineer to own and evolve a Databricks lakehouse on AWS. You will manage data pipelines, models, and governance across owners, ensuring compliance with data sharing rules and regulatory needs.

This role is remote-first with periodic travel and reports to the CTO. You will extend data sources, build new models, and collaborate across teams to maintain reliable, auditable data products and pipelines.

Qualifications

• Bachelor’s degree in Computer Science, Statistics, Mathematics, Engineering, or related field.
• 5+ years building production data pipelines with SQL and Python.
• Strong experience with Spark or PySpark and dbt for transformation and testing.
• Hands‑on Databricks experience including Unity Catalog, Delta Lake, and Lambda/AI features.
• Experience with AWS and infrastructure as code (Terraform or OpenTofu).
• CI/CD pipeline ownership with GitHub Actions and automated testing.
• Machine learning engineering including feature pipelines, MLflow, and model serving.
• Regulated data handling: HIPAA, PHI/PII protection, audit logs.

Responsibilities

• Operate and improve pipelines feeding data from operations to the lakehouse.
• Extend the lakehouse with new data sources and governance from day one.
• Enforce ownership, sharing, and ingestion boundaries across pipelines and models.
• Own dbt transformation layers and curated data products.
• Govern the lakehouse with Unity Catalog: access controls and lineage.
• Operate platform as code via CI/CD pipelines and cloud tooling.
• Maintain data quality: automated tests, freshness checks, anomaly detection.
• Develop document ingestion with Databricks AI/ Textract and expand over time.
• Extend ML models from design to production and monitor lifecycle with MLflow.
• Run real-time and batch inference via Databricks Model Serving.

Skills

Data pipelines

SQL

Python

Spark / PySpark

dbt

Databricks

Unity Catalog

AWS

Terraform / OpenTofu

CI/CD

MLflow

Model governance

HIPAA / PHI / PII handling

AI-assisted development

Education

Bachelor’s degree in Computer Science, Statistics, Mathematics, Engineering, or related field

Tools

Databricks

Unity Catalog

Delta Lake

Lakeflow

Databricks SQL

Databricks AI functions

GitHub Actions

MLflow

Job description

Location: Remote (United States), with periodic travel

Reports To: Chief Technology Officer

Department: Technology

About LHL Strategies

Every year, millions of seniors lapse or surrender their life insurance policies for little or no value from their insurance company. Life settlements give them the option to sell the policy instead, for significantly more than they would otherwise receive. The money we pay helps meet retirement needs, healthcare costs, long term care expenses, or whatever else they choose.

Lighthouse Life is part of LHL Strategies, Inc., a group of companies providing vertically integrated services for asset managers who acquire and manage life policies and portfolios. We are the only integrated platform in the industry spanning policy acquisition and trading, valuation, management, and portfolio servicing.

We are a remote-first company, and this role can sit anywhere in the U.S. Our offices are in Conshohocken, Pennsylvania, just outside Philadelphia. We encourage innovation, protect work-life balance, and are driven to exceed expectations.

About the Role

Our data runs on a Databricks lakehouse on AWS. It brings together structured and unstructured data from across the business and feeds analytics and machine learning models. This role owns that platform.

The lakehouse and our first production models are already in place. You will own their ongoing operation and evolution: the pipelines that feed the models, the models themselves, and the evidence that keeps them defensible to regulators and auditors.

The lakehouse holds data on behalf of multiple owners, and each owner sets its own terms for what we may collect, how it can be used, and who it can be shared with. Respecting those boundaries is a core requirement of the platform, not an afterthought. You will make sure that every pipeline, data product, and model reflects the agreed ownership, sharing, and ingestion rules, both in the work already built and in everything we add. You will also be able to show that this is true when someone asks. We are looking for someone who treats data stewardship as an engineering discipline and conforms to and builds on the controls we have in place.

This is build and run, not build and hand off. The platform already matters, and you get to decide where it goes next: bringing in new data sources, extending the models we have, and building new ones.

How We Work

• Flagging problems early, not after they have become crises
• “How can we improve this?” over “This is how we have always done it”
• Systems thinking: automate and document, do not just execute manually
• Intellectual honesty about what a model does and does not know

If you thrive when told exactly what to do and left alone, this is not the role. If you thrive when trusted to own outcomes and keep leadership informed, we want to talk.

Roles and Responsibilities

• Operate and improve the pipelines that bring data from our operational systems and document sources into the lakehouse
• Extend the lakehouse to new internal and third-party data sources as the business grows, with quality checks and governance built in from the start
• Enforce data ownership, sharing, and ingestion boundaries across every pipeline, data product, and model, and be able to demonstrate that they hold
• Own the dbt transformation layers and the curated data products built on top of them
• Govern the lakehouse in Unity Catalog: access controls, lineage, and the handling of sensitive data as new tables and consumers arrive
• Own the platform as code, with infrastructure and workloads deployed through automated, reviewed CI/CD pipelines
• Keep the data quality framework running: automated tests, freshness checks, and anomaly detection, held to an agreed reliability target
• Own document ingestion and extraction using Databricks AI functions and Amazon Textract, and extend it to new document types over time
• Extend our existing machine learning models and design, build, and ship new ones, from problem framing and feature design through validation, governance review, and production
• Run the model lifecycle in MLflow: feature pipelines, retraining, evaluation, registry, versioning, and approval, kept auditable
• Operate batch and real-time inference through Databricks Model Serving against the latency targets the business requires
• Monitor models in production, respond to drift, and maintain the explainability, fairness testing, and documentation our model governance requires
• Build and maintain the semantic layer and Databricks AI/BI dashboards our business teams use, reconciled to authoritative sources so one question does not produce two answers
• Own cloud cost for the data and machine learning estate against an agreed budget, and keep runbooks and documentation current

Desired Skills

Hard Skills

• Data engineering. 5 or more years building production data pipelines. Strong SQL and Python, production experience with Spark or PySpark, and dbt for transformation and testing.
• Databricks. Hands‑on production experience with Unity Catalog, Delta Lake, Lakeflow, Databricks SQL, and Databricks AI functions.
• AWS and infrastructure as code. Running Databricks on AWS, with infrastructure as code in Terraform or OpenTofu and workloads as Databricks Asset Bundles.
• CI/CD. Pipeline ownership using GitHub Actions, with automated testing and controlled promotion across environments.
• Machine learning engineering. Building and shipping models end to end: feature pipelines, experiment tracking, MLflow, and both batch and real-time serving.
• Model operations. Drift detection and response, retraining strategy, champion and challenger evaluation, and explainability methods such as SHAP.
• Regulated data handling. HIPAA or equivalent. PHI and PII protection, encryption, least-privilege access, and audit logging.
• AI-assisted development. We build with AI every day, and we expect our data and AI engineer to do the same. You use tools like Claude Code, Databricks Assistant, or GitHub Copilot as part of your normal workflow, and you review their output as carefully as a teammate’s.
• Helpful, not required. Document extraction at scale. Taking over and hardening a platform built by another team.

Soft Skills

• Owner. You treat the model and the platform as yours, not as something that belongs to whoever built them.
• Honest about uncertainty. You will say when a model is not good enough yet, including when that is inconvenient.
• Writer and communicator. We are remote and asynchronous, and much of this role is documentation that a regulator or auditor will read. Excellent written and verbal communication skills are required.
• Business-curious. You will want to understand how the business works, not only how the pipeline runs. You work closely with our Lead Developer and with business, analytics, and compliance teams.
• Pragmatic. A model that ships and is monitored beats a better model that is still in a notebook.

Education

• Bachelor’s degree in Computer Science, Statistics, Mathematics, Engineering, or a related quantitative field, or equivalent practical experience. A master’s degree is helpful. Insurance, actuarial, healthcare, or credit risk modeling experience is a plus.

Compensation and Benefits

• Base salary: Competitive, commensurate with experience
• Benefits: Comprehensive health, dental, and vision; 401(k); flexible PTO
• Remote work: Remote-first, anywhere in the U.S.

Why Join LHL?

• Impact: Own the data and model platform for a growing company in an underserved market
• Innovation: Work at the intersection of finance, insurance, and machine learning
• Culture: Collaborative, entrepreneurial environment with experienced leadership