Remote Senior Databricks Engineer: Excel-to-Delta

Lumenalta; United States; Remote; USD 120,000 - 160,000; Full time

Benefits offered by this job

Remote work (US)

Career growth

Job summary

Lumenalta seeks a Databricks Data Engineer to design and operate ingestion for Excel and flat-file sources, turning them into governed Delta tables. You will handle schema evolution, data quality, and lineage, joining to BOM and engineering data while documenting every source for client handoff.

The role is fully remote within the United States, requiring US citizenship and collaboration with business owners to pin down definitions and rules.

Qualifications

• Production experience building ingestion pipelines on Databricks.
• Hands-on ingestion of Excel and flat-file sources into Spark/Databricks at scale.
• Experience with schema evolution, data quality, and governed data pipelines.
• Ability to align ingested data to shared keys for joins with other systems.
• Strong PySpark and SQL skills; knowledge of Delta Lake and medallion architecture.

Responsibilities

• Inventory and document Excel workbooks and file exports used by business teams.
• Build Auto Loader pipelines to ingest workbooks and flat files landed in S3 with explicit schemas.
• Parse multi-sheet and inconsistently structured files at scale and handle version drift.
• Configure schema evolution and data quality checks in pipelines.
• Conform ingested data to keying schemes for joins to BOM and engineering data.
• Capture lineage, orchestrate with Databricks Workflows, and document sources.

Skills

Databricks

PySpark

SQL

Delta Lake

Auto Loader

Excel ingestion

S3 ingestion

Data quality

Data governance

Documentation

Tools

Databricks Platform

Job description

Lumenalta seeks a Databricks Data Engineer to design and operate ingestion for Excel and flat-file sources, turning them into governed Delta tables. You will handle schema evolution, data quality, and lineage, joining to BOM and engineering data while documenting every source for client handoff.

The role is fully remote within the United States, requiring US citizenship and collaboration with business owners to pin down definitions and rules.