Senior Databricks Data Engineer: PySpark & Cloud Pipelines

Machine Learning Technologies, LLC; Madison (WI); On-site; USD 100,000 - 150,000; Full time

Job summary

Machine Learning Technologies, LLC is seeking an experienced Data Engineer in Madison, WI to design, build, and maintain scalable data pipelines. The role focuses on Databricks, PySpark, Python, and SQL, with exposure to cloud data platforms and governance practices.

You will collaborate with cross-functional teams to optimize data processing, ensure data quality, and enable robust analytics across on-prem and cloud environments. Experience with Delta Lake and Unity Catalog is a plus.

Qualifications

• Experience building scalable data pipelines and ETL/ELT processes.
• Strong SQL optimization and data modeling skills.
• Familiarity with cloud data services and orchestration frameworks.
• Experience with data quality validation and governance.

Responsibilities

• Develop data pipelines using Databricks and PySpark.
• Build ETL/ELT workflows using Python and SQL.
• Work with Apache Spark, Delta Lake, and Unity Catalog.
• Integrate data from multiple sources and optimize processing.

Skills

Databricks

PySpark

Python

SQL

Cloud platforms

Tools

Apache Spark

Delta Lake

Unity Catalog

AWS

Azure

GCP

Job description

Machine Learning Technologies, LLC is seeking an experienced Data Engineer in Madison, WI to design, build, and maintain scalable data pipelines. The role focuses on Databricks, PySpark, Python, and SQL, with exposure to cloud data platforms and governance practices.

You will collaborate with cross-functional teams to optimize data processing, ensure data quality, and enable robust analytics across on-prem and cloud environments. Experience with Delta Lake and Unity Catalog is a plus.