Senior Databricks Lakehouse Ingestion Engineer - Remote

Lumenalta; United States; Remote; USD 90,000 - 130,000; Full time

Benefits offered by this job

Health insurance

401K Contribution

Paid time off

Professional development fund

Remote work

Job summary

Lumenalta is seeking a Senior Databricks Engineer to own the ingestion layer for a US defense client. You will run source discovery across PLM, ERP, and MBSE sources, design ingestion patterns (Lakeflow Connect, JDBC, Auto Loader) and build end-to-end pipelines with careful handling of schema evolution.

This fully remote role requires US citizenship, offers a salary range of $90,000-$130,000, strong benefits, and a path to Databricks certifications plus working with leading teams on complex data

Qualifications

• 7+ years of production data engineering on Databricks.
• Strong Python, PySpark, and SQL.
• Fluent in Delta Lake, Auto Loader, and declarative pipeline development.
• Hands-on experience with Lakeflow Connect.
• Direct experience building JDBC-based ingestion against Oracle E-Business Suite, and familiarity with its security model, session context, and multi-org data structures.
• Experience with SQL Server as an ingestion source.
• Working fluency in manufacturing PLM and MBSE systems: Teamcenter, Cameo, and Siemens Capital.
• Databricks Data Engineer Professional certification.
• US citizenship and ability to pass a background check.
• Fluent English, both written and spoken, with strong collaboration skills in distributed teams.

Responsibilities

• Run source discovery across each system, documenting data structures, change patterns, extraction constraints, data quality, and current-state integration topology.
• Assess extraction options per source, including change data capture feasibility from PLM, JDBC access to Oracle E-Business Suite under its row-level security model, SQL Server change patterns, and file-based ingestion from S3.
• Design the ingestion pattern for each source (Lakeflow Connect, custom JDBC, Auto Loader, or federation) with the tradeoffs written down and defensible.
• Model bill-of-materials data across PLM, ERP, and MBSE sources, keyed on part number plus revision, including reconciliation into bronze and silver layers.
• Build the ingestion layer: Lakeflow Connect pipelines including CDC from Teamcenter PLM, JDBC connectors against Oracle E-Business Suite with session parameter handling, Auto Loader for incremental S3 ingestion, and SQL Server ingestion patterns.
• Handle schema evolution deliberately, with automatic handling of new columns, rescued data columns for unexpected shapes, and controlled failure on renamed columns.
• Operate nightly batch patterns with idempotent reruns and backfill support, quarantining bad source data rather than dropping it.
• Apply strong engineering practices including testing, CI/CD, and version control, and write expectations and quality checks into pipelines.
• Document for handoff, producing each source's ingestion contract and runbook so the client team can own it.

Skills

Databricks

Python

PySpark

SQL

Delta Lake

Auto Loader

CI/CD

Git workflows

Tools

Lakeflow Connect

Oracle E-Business Suite

SQL Server

Terraform

Unity Catalog

Job description

Lumenalta is seeking a Senior Databricks Engineer to own the ingestion layer for a US defense client. You will run source discovery across PLM, ERP, and MBSE sources, design ingestion patterns (Lakeflow Connect, JDBC, Auto Loader) and build end-to-end pipelines with careful handling of schema evolution.

This fully remote role requires US citizenship, offers a salary range of $90,000-$130,000, strong benefits, and a path to Databricks certifications plus working with leading teams on complex data