Senior Data Engineer – Databricks

Valiant Technologies LLC; Madison (WI); In loco; USD 120.000 - 150.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC in Madison, WI is seeking an experienced Data Engineer to design and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.

You will work across AWS, Azure, or GCP to integrate data from multiple sources, optimize performance, and implement data quality checks within Delta Lake and Unity Catalog.

This role requires hands-on expertise in Spark ecosystems and a collaborative approach to delivering reliable data solutions for analytics and reporting.

Competenze

• Experience building data pipelines with Databricks and PySpark.
• Strong Python and SQL development skills.
• Familiarity with cloud platforms (AWS/Azure/GCP) and data warehousing.
• Experience with Apache Spark, Delta Lake, Unity Catalog.
• Ability to integrate data from multiple sources and optimize queries.

Mansioni

• Build and maintain scalable data pipelines and ETL/ELT processes.
• Optimize SQL performance and data processing workflows.
• Implement data quality and validation checks.
• Collaborate with cloud platform teams (AWS/Azure/GCP).

Strumenti

Databricks

PySpark

Python

SQL

AWS

Azure

GCP

Apache Spark

Delta Lake

Unity Catalog

Descrizione del lavoro

We are looking for an experienced Data Engineer with expertise in Databricks, PySpark, Python, SQL, and cloud technologies to build and maintain scalable data pipelines and ETL/ELT solutions.

Develop data pipelines using Databricks and PySpark.

Build ETL/ELT workflows using Python and SQL.

Work with Apache Spark, Delta Lake, and Unity Catalog.

Integrate data from multiple sources.

Optimize data processing and SQL performance.

Implement data quality and validation processes.

Work with AWS, Azure, or GCP cloud platforms.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.