Senior Data Engineer (DataBricks & Azure)

Aether Biomedical; Deutschland; Vor Ort; EUR 52.479 - 82.726; Vollzeit

Benefits dieser Stelle

26 vacation days

Health and life insurance

Flexible workplace (office/home/hybrid

Learning platforms access (O'Reilly, L

Zusammenfassung

Aether Biomedical is seeking a Senior Data Engineer to join a cross-functional team. You will design, build and maintain ETL pipelines to turn raw data into insights, using Python, PySpark, SQL, and Azure Databricks.

You’ll contribute to data modeling, performance tuning, and Delta Lake technologies in a collaborative agile environment. Responsibilities include optimizing data workflows, collaborating with stakeholders, and advancing data strategies across the organization.

Qualifikationen

• Proven experience as a Data Engineer in a professional, production environment.
• Strong programming skills in Python and solid understanding of ETL processes.
• Advanced proficiency in SQL and experience working with large datasets.
• Hands-on experience with Azure Databricks and Azure Data Factory (ADF).
• Solid understanding of data modeling, data warehousing concepts, and performance tuning.
• Strong expertise in Databricks, including Delta Tables and Delta Live Tables.
• Advanced programming skills in Python and PySpark.
• Experience with Unity Catalog.
• Proficiency in dbt (Data Build Tool).
• Solid knowledge of Medallion Architecture.

Aufgaben

• Design, build, and maintain ETL processes, transforming raw data into actionable insights.
• Manage and optimize large-scale data pipelines, ensuring data quality and integrity.
• Develop and optimize SQL queries for efficient data extraction, transformation, and analysis.
• Collaborate with cross-functional teams to understand data requirements and implement effective data solutions.
• Utilize Python and PySpark for data engineering tasks, including scripting and automation.
• Leverage Azure Databricks for advanced data processing and analytics.
• Monitor, troubleshoot, and optimize data workflows to ensure high performance and reliability.

Kenntnisse

Python

SQL

PySpark

Azure Databricks

Azure Data Factory

Delta Tables

Delta Live Tables

dbt

Unity Catalog

Tools

Azure Databricks

Azure Data Factory (ADF)

dbt

Unity Catalog

Jobbeschreibung

Data

Client is a premier home improvement company, boasting a network of over 2,000 stores across eight European countries. You'll be part of an organization that values innovation and teamwork, where you contribute to the company's success, not just a single project.

Project overview

Working in cross functional team using Agile framework to design, build and maintain home improvement solution.

Team

As a Senior Data Engineer, you'll integrate into a dynamic data engineering team led by an experienced Data Lead. Your new team is a blend of talented Data Engineers, Data Designers, meticulous QA experts, and an agile Scrum Master, all dedicated to building exceptional data solutions.

Position overview

Data Engineer will offer engineering support and deliver data insights through the collection, analysis, and summarization of development and service issues. Your expertise will help steer the direction of our data strategies and ensure the highest quality in our technology offerings.

Responsibilities

• Design, build, and maintain ETL processes, transforming raw data into actionable insights
• Manage and optimize large-scale data pipelines, ensuring data quality and integrity
• Develop and optimize SQL queries for efficient data extraction, transformation, and analysis
• Collaborate with cross-functional teams to understand data requirements and implement effective data solutions
• Utilize Python and PySpark for data engineering tasks, including scripting and automation
• Leverage Azure Databricks for advanced data processing and analytics
• Monitor, troubleshoot, and optimize data workflows to ensure high performance and reliability

Requirements

• Proven experience as a Data Engineer in a professional, production environment
• Strong programming skills in Python and solid understanding of ETL processes
• Advanced proficiency in SQL and experience working with large datasets
• Hands-on experience with Azure Databricks and Azure Data Factory (ADF)
• Solid understanding of data modeling, data warehousing concepts, and performance tuning
• Strong expertise in Databricks, including Delta Tables and Delta Live Tables
• Advanced programming skills in Python and PySpark
• Experience with Unity Catalog
• Proficiency in dbt (Data Build Tool)
• Solid knowledge of Medallion Architecture

Nice to have

• Experience with data governance and data cataloging tools
• Familiarity with CI/CD practices and version control systems (Git)
• Understanding of data security and privacy best practices
• Experience with additional cloud services or platforms

What We Offer:

• Vacation days: Up to 26 business days per year.
• 10 illness/special days off per year (fully paid, no medical papers needed) for all contract types
• Health and life insurance (Luxmed)
• MyBenefit platform with Multisport option
• Internal psychological support service
• English language classes from the first working day
• Access to external learning platforms: O'Reilly, LinkedIn Learning, Udemy, and a wide catalog of diverse internal training
• Flexible workplace: work from the office, from home, or choose a hybrid option
• Opportunities to develop as a public speaker, mentor, or technical interviewer
• Fully paid idle (bench) when not involved in a project
• 20300 - 32000 PLN/month (depends on contract)
• 20300 - 32000 PLN/month (depends on contract)