Senior Data Engineer (Databricks)

Spyro Soft; Deutschland; Remote; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

Spyro Soft is seeking a Senior Data Engineer to design, build and optimize data platforms using Databricks. You will develop scalable ETL/ELT pipelines, process large datasets and improve cloud-native data engineering solutions.

You'll work with Delta Lake, Medallion Architecture, and collaborate with Architects, Data Scientists and Analytics teams to deliver reliable data products and ensure data quality across the stack.

Qualifikationen

• 5+ years of experience in Data Engineering.
• 2+ years of commercial experience with Databricks.
• Strong knowledge of PySpark and Spark SQL.
• Excellent SQL skills.
• Strong Python programming skills.
• Experience building ETL/ELT pipelines.
• Experience working with Delta Lake.
• Good understanding of Medallion Architecture.
• Experience with cloud environments (Azure, AWS or GCP).
• Experience with Git and CI/CD.
• Understanding of data modelling principles.
• Good communication skills.
• Fluent English - C1

Aufgaben

• Design, develop and maintain scalable ETL/ELT pipelines.
• Build reliable batch and streaming data processing solutions.
• Develop data pipelines using Databricks and Apache Spark.
• Optimize Spark jobs and improve platform performance.
• Work with Delta Lake and Medallion Architecture.
• Build and maintain data models for analytics and reporting.
• Ensure data quality, monitoring and observability.
• Collaborate with Architects, Data Scientists and Analytics teams.
• Participate in code reviews and technical improvements.
• Contribute to CI/CD processes and engineering best practices

Kenntnisse

PySpark

Spark SQL

SQL

Python

ETL/ELT pipelines

Delta Lake

Medallion Architecture

Cloud environments

Git & CI/CD

Data modelling

Data quality/observability

Communication

English (C1)

Tools

Databricks

Apache Spark

Airflow

dbt

Kafka

Delta Lake

Git

CI/CD

Jobbeschreibung

About the role

We are looking for a Senior Data Engineer with strong Databricks experience to build and optimize modern data platforms. You will develop scalable ETL/ELT pipelines, process large datasets and contribute to the evolution of cloud-native Data Engineering solutions.

Responsibilities:

• Design, develop and maintain scalable ETL/ELT pipelines

• Build reliable batch and streaming data processing solutions

• Develop data pipelines using Databricks and Apache Spark

• Optimize Spark jobs and improve platform performance

• Work with Delta Lake and Medallion Architecture

• Build and maintain data models for analytics and reporting

• Ensure data quality, monitoring and observability

• Collaborate with Architects, Data Scientists and Analytics teams

• Participate in code reviews and technical improvements

• Contribute to CI/CD processes and engineering best practices

Requirements:

• 5+ years of experience in Data Engineering

• 2+ years of commercial experience with Databricks

• Strong knowledge of PySpark and Spark SQL

• Excellent SQL skills

• Strong Python programming skills

• Experience building ETL/ELT pipelines

• Experience working with Delta Lake

• Good understanding of Medallion Architecture

• Experience with cloud environments (Azure, AWS or GCP)

• Experience with Git and CI/CD

• Understanding of data modelling principles

• Good communication skills

• Fluent English - C1

Nice to have:

• Unity Catalog experience

• dbt

• Kafka or Structured Streaming

• Airflow or Databricks Workflows

• Databricks certification

• Experience with performance tuning and cost optimization