Remote Databricks Data Engineer – PLM/MBSE Ingestion

Lumenalta; United States; Remote; USD 110,000 - 150,000; Full time

Job summary

Lumenalta is seeking a senior data engineer to design and operate ingestion pipelines on Databricks. You will build an S3 landing zone, implement Auto Loader-based ingestion for multiple source systems, and produce Bronze/Silver tables with robust schema evolution.

The role requires deep PySpark/SQL expertise and a strong focus on data quality and lineage. Remote within the United States, requires US citizenship, and collaboration with engineering SMEs to understand exports and model elements.

Qualifications

• 6+ years in data engineering
• Deep experience with Auto Loader and incremental file ingestion from S3
• Experience parsing CSV, XML, JSON into typed tables
• Experience with Databricks declarative pipelines or Delta Live Tables
• Track record of handling bad, late, and re-delivered files
• Strong PySpark and SQL; knowledge of Delta Lake internals
• Experience with Databricks Workflows for orchestration
• Fluent English; US citizenship

Responsibilities

• Establish the S3 landing zone and ingestion framework for all file-based engineering sources
• Build Auto Loader pipelines for incremental ingestion of Cameo, Teamcenter, and Capital exports landed in S3
• Parse export formats into Bronze and Silver tables with explicit schemas
• Define incremental and batch refresh patterns across in-scope sources
• Configure schema evolution and handle renamed columns safely
• Implement data quality checks and capture lineage for pipelines
• Orchestrate pipelines with Databricks Workflows
• Collaborate with SMEs to understand export contents and cadence

Skills

Databricks

PySpark

SQL

Delta Lake

Auto Loader

Workflows

Schema evolution

S3 ingestion

Tools

Databricks

Delta Live Tables

Job description

Lumenalta is seeking a senior data engineer to design and operate ingestion pipelines on Databricks. You will build an S3 landing zone, implement Auto Loader-based ingestion for multiple source systems, and produce Bronze/Silver tables with robust schema evolution.

The role requires deep PySpark/SQL expertise and a strong focus on data quality and lineage. Remote within the United States, requires US citizenship, and collaboration with engineering SMEs to understand exports and model elements.