Senior Databricks Engineer: Lakehouse & MLOps on AWS

MACHINE LEARNING TECHNOLOGIES LLC; Columbus (OH); On-site; USD 140,000 - 180,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Columbus, OH seeks a Senior Databricks Developer to design and implement high-volume data pipelines on Databricks and AWS. You will build Delta Lake Lakehouse architectures, streaming solutions, and scalable ETL/ELT processes using PySpark and Scala Spark.

The role emphasizes real-time processing, MLOps within Databricks, and Data Mesh principles to optimize performance, cost, and reliability across enterprise data platforms.

Qualifications

• Deep knowledge of Databricks and AWS ecosystems.
• Strong experience with Delta Lake and Lakehouse Architecture.
• Hands-on PySpark and Scala Spark experience.
• Experience with Streaming and Delta Live Tables (DLT).
• MLOps implementation within Databricks.
• Knowledge of Data Mesh architecture.
• Advanced cost optimization and performance engineering.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks and AWS.
• Implement and optimize Delta Lake and Lakehouse architecture.
• Develop high-volume ETL/ELT pipelines using PySpark and Scala Spark.
• Build and manage high-volume, production‑grade data processing workflows.
• Implement real-time and streaming data pipelines using Databricks and Delta Live Tables (DLT).
• Implement and support MLOps solutions within the Databricks ecosystem.
• Apply Data Mesh architecture principles to enterprise data platforms.
• Optimize data processing workloads for performance, scalability, and cost efficiency.
• Monitor and troubleshoot production data pipelines to ensure reliability and data quality.
• Collaborate with data engineering, architecture, and ML teams to deliver enterprise data solutions.

Skills

Databricks & AWS

Delta Lake

Lakehouse Architecture

PySpark

Scala Spark

DLT (Delta Live Tables)

MLOps in Databricks

Data Mesh

Cost optimization

Production-grade ETL/ELT

Tools

Lakeflow

Lakebase

Job description

MACHINE LEARNING TECHNOLOGIES LLC in Columbus, OH seeks a Senior Databricks Developer to design and implement high-volume data pipelines on Databricks and AWS. You will build Delta Lake Lakehouse architectures, streaming solutions, and scalable ETL/ELT processes using PySpark and Scala Spark.

The role emphasizes real-time processing, MLOps within Databricks, and Data Mesh principles to optimize performance, cost, and reliability across enterprise data platforms.