Senior Data Engineer – Databricks

Quantum Technologies. LLC; Madison (WI); In loco; USD 110.000 - 140.000; Tempo pieno

Descrizione del lavoro

Quantum Technologies. LLC in Madison, Wisconsin is seeking an experienced Data Engineer with expertise in Databricks, PySpark, Python, SQL, and cloud technologies to build and maintain scalable data pipelines and ETL/ELT solutions. Develop data pipelines using Databricks and PySpark.

Build ETL/ELT workflows using Python and SQL. Work with Apache Spark, Delta Lake, and Unity Catalog. Integrate data from multiple sources while optimizing SQL performance and data quality across cloud platforms

Competenze

• Experience building and maintaining data pipelines using Spark-based technologies.
• Strong knowledge of SQL and data warehousing concepts.
• Experience with cloud data platforms and data lake architectures.

Mansioni

• Build and maintain scalable data pipelines and ETL/ELT solutions.
• Develop pipelines using Databricks and PySpark.
• Build ETL/ELT workflows using Python and SQL.
• Integrate data from multiple sources and optimize processing.
• Ensure data quality and validation across pipelines.

Conoscenze

Data pipeline design

ETL/ELT

SQL optimization

Cloud data architectures

Strumenti

Databricks

PySpark

Python

SQL

AWS

Azure

GCP

Delta Lake

Descrizione del lavoro

We are looking for an experienced Data Engineer with expertise in Databricks, PySpark, Python, SQL, and cloud technologies to build and maintain scalable data pipelines and ETL/ELT solutions.

Develop data pipelines using Databricks and PySpark.

Build ETL/ELT workflows using Python and SQL.

Work with Apache Spark, Delta Lake, and Unity Catalog.

Integrate data from multiple sources.

Optimize data processing and SQL performance.

Implement data quality and validation processes.

Work with AWS, Azure, or GCP cloud platforms.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.