Databricks Data Engineer: Build Scalable Cloud Pipelines

Cloud Analytics Technologies, LLC; Madison (WI); On-site; USD 110,000 - 150,000; Full time

Job summary

Cloud Analytics Technologies, LLC is seeking an experienced Data Engineer in Madison, WI to design and maintain scalable data pipelines using Databricks and PySpark. The role emphasizes ETL/ELT development with Python and SQL, plus expertise in Spark, Delta Lake, and Unity Catalog.

You will integrate data from multiple sources, optimize SQL performance, and implement data quality validations across cloud environments (AWS/Azure/GCP). This position is on-site in Madison, WI.

Qualifications

• Experience building data pipelines using Databricks and PySpark.
• Strong SQL and Python ETL/ELT design skills.
• Experience with Spark-based data processing and data quality validation.

Responsibilities

• Develop scalable data pipelines with Databricks and PySpark.
• Build ETL/ELT workflows using Python and SQL.
• Collaborate on Spark, Delta Lake, and Unity Catalog implementations.
• Ingest data from multiple sources and optimize data processing.

Skills

Databricks

PySpark

Python

SQL

Apache Spark

Delta Lake

Unity Catalog

Tools

AWS

Azure

GCP

Job description

Cloud Analytics Technologies, LLC is seeking an experienced Data Engineer in Madison, WI to design and maintain scalable data pipelines using Databricks and PySpark. The role emphasizes ETL/ELT development with Python and SQL, plus expertise in Spark, Delta Lake, and Unity Catalog.

You will integrate data from multiple sources, optimize SQL performance, and implement data quality validations across cloud environments (AWS/Azure/GCP). This position is on-site in Madison, WI.