Senior Databricks Data Engineer - Remote Platform Builder

Lumenalta; United States; Remote; USD 90,000 - 132,000; Full time

Benefits offered by this job

Health insurance

401K Contribution

Professional development fund

15 days paid time off annually

Job summary

Lumenalta seeks a Senior Databricks Data Engineer to design and implement scalable ingestion pipelines that bring Excel workbooks and flat-file data into governed Delta tables. You will parse diverse workbook layouts, apply schema evolution, and ensure data quality and lineage across the platform.

You will collaborate with business owners to define rules, maintain idempotent load behavior, and join the data to BOM and engineering datasets, delivering reliable, scalable data assets for analytics

Qualifications

• Production experience building ingestion pipelines on Databricks.
• Hands-on ingestion of Excel and flat-file sources into Spark/Databricks at scale.
• Experience with Auto Loader for incremental ingestion from S3.
• Experience with MERGE and handling changes for versioned/re-delivered data.
• Strong PySpark and SQL with Delta Lake and medallion architecture knowledge.
• Ability to work with non-technical data owners to define business rules.

Responsibilities

• Inventory Excel workbooks and file exports, document structure and owners.
• Build Auto Loader pipelines with explicit schemas and Bronze/Silver tables.
• Parse multi-sheet workbooks with varying headers and layouts at scale.
• Treat re-delivered workbooks as versions; apply changes idempotently with MERGE.
• Configure schema evolution and data quality checks in pipelines.
• Ensure ingested data aligns with part/revision keys for joins to BOM/engineering data.
• Capture lineage, orchestrate with workflows, and document sources for handoff.

Skills

Databricks

PySpark

SQL

Delta Lake

Auto Loader

Delta Live Tables

MERGE

Data quality

Medallion architecture

Data ingestion

Job description

Lumenalta seeks a Senior Databricks Data Engineer to design and implement scalable ingestion pipelines that bring Excel workbooks and flat-file data into governed Delta tables. You will parse diverse workbook layouts, apply schema evolution, and ensure data quality and lineage across the platform.

You will collaborate with business owners to define rules, maintain idempotent load behavior, and join the data to BOM and engineering datasets, delivering reliable, scalable data assets for analytics