Senior Databricks Engineer: AWS, Delta Lake, ML Ops

Longfinch Technologies; Burrell Township (PA); On-site; USD 120,000 - 180,000; Full time

Job summary

Longfinch Technologies is seeking a highly skilled Databricks engineer with deep AWS expertise to design and implement lake house architecture, Delta Lake strategies, and scalable PySpark/Scala Spark workflows. You will own production ELT/ETL pipelines, streaming jobs, and ML Ops on Databricks, contributing to a data mesh approach and optimizing costs.

This role targets complex, high-volume data environments.

Qualifications

• Must have deep expertise in Databricks and AWS ecosystems.
• Delta Lake Lake house architecture knowledge (Lake flow, Lake Base).
• Proficient in PySpark/Scala Spark for large-scale data processing.
• Proven ELT/ETL engineering in production Databricks environments.
• Experience with streaming and Delta Live Tables (DLT).
• ML Ops on Databricks experience.
• Knowledge of Data Mesh architecture principles.
• Strong cost/performance optimization skills for cloud data platforms.
• Databricks and AWS data engineer background.

Skills

Databricks AWS

Delta Lake architecture

PySpark/Scala Spark

ELT/ETL engineering

Streaming/DLT experience

ML Ops on Databricks

Data Mesh knowledge

Cost/performance engineering

Databricks AWS data eng

Job description

Longfinch Technologies is seeking a highly skilled Databricks engineer with deep AWS expertise to design and implement lake house architecture, Delta Lake strategies, and scalable PySpark/Scala Spark workflows. You will own production ELT/ETL pipelines, streaming jobs, and ML Ops on Databricks, contributing to a data mesh approach and optimizing costs.

This role targets complex, high-volume data environments.