Senior Data Engineer (Azure, Databricks)

Optiveum; Wrocław; On-site; PLN 136,000 - 227,000; Full time

Job summary

Optiveum is seeking a Senior Data Engineer to join a data team in the financial services arm of a global automotive company. You will build and optimize ETL pipelines in Azure and Databricks, migrate legacy data, ensure data quality and collaborate with cross-functional teams to deliver actionable insights.

The role requires 5+ years in data engineering, strong SQL and Python/Java/Scala skills, plus Azure Databricks expertise and relevant certifications.

Qualifications

• 5+ years in data engineering or related field with ETL/cloud data tools.
• Bachelor’s degree in Computer Science, Engineering or related field; Master’s preferred.
• Microsoft Certified Azure Data Engineer and Databricks Certified Data Engineer.
• Proficiency in Python, Java or Scala.
• Hands-on experience with MS Cloud SQL Server, Azure Data Factory, Azure Databricks, PySpark, Spark SQL, ADLS, Power BI, SSIS and SSRS.
• Strong SQL skills and experience with relational databases.
• Familiarity with big data technologies and data processing frameworks.
• Experience with Agile/Scrum.
• Strong problem-solving and communication skills and a collaborative mindset.
• English at C1 level

Responsibilities

• Extract, standardize, transform and load data from multiple sources to deliver clean datasets for analysis, including legacy migrations.
• Collaborate with data modellers and business analysts to map data models and source-to-staging mappings.
• Create analysis-ready data structures and DDLs for MS Cloud SQL Server tables and views in the Pond database.
• Develop Databricks notebooks to extract data and form Shared File Formats (SFFs).
• Configure Ingestion Framework tables to load staging and DLC tables.
• Build large-scale batch and real-time data pipelines in cloud/on-prem environments; optimize performance.
• Apply CI/CD practices and automate integration flows.
• Monitor data consumption and coordinate with governance teams on the data catalogue.
• Drive improvements in data acquisition, testing and tooling to raise quality.
• Monitor and optimize SQL queries; prepare design documents from requirements.
• Work in cross-functional agile teams and contribute to estimates.

Skills

Python/Java/Scala

SQL

Agile/Scrum

English(C1)

Education

Bachelors in Computer Science/Engineering or related

Master's preferred

Tools

Azure Databricks

Azure Data Factory

MS Cloud SQL Server

Spark SQL

ADLS

Power BI

SSIS/SSRS

Job description

Senior Data Engineer (Azure, Databricks)

About the Offer

We are currently looking for a Senior Data Engineer to join an innovative data team at our client, the financial services (captive finance) arm of a global automotive company. For this role, Optiveum is the recruitment and contracting party, which means you will sign the cooperation agreement directly with us.

You will build and optimise ETL pipelines in the Azure and Databricks ecosystem, migrate data from legacy systems to next-generation platforms, ensure data quality and work with cross-functional teams to deliver actionable insights.

Key Details:

• Rate: up to 165 PLN/h net + VAT

• Work Model: B2B contract, full-time (40 h/week), 100% onsite

• Location: Wrocław

• Language: English (C1)

Responsibilities:

• Extract, combine, transform, load and cleanse data from various sources to deliver high-quality datasets for analysis, including migration from legacy systems

• Work with data modellers and business analysts to understand the data model and source-to-staging mapping

• Create analysis-ready data structures, including DDLs for MS Cloud SQL Server tables and views in the Pond database (a subset of the Data Lake)

• Design and develop Databricks notebooks that extract data from input files and join related entities into Shared File Formats (SFFs)

• Configure Ingestion Framework tables to load staging and Data Lake Core (DLC) tables

• Build large-scale batch and real-time data pipelines in cloud or on-premises environments and develop performance-efficient algorithms

• Apply CI/CD best practices and introduce automation in existing and future integration flows

• Track data consumption patterns and work with data governance and information stewards on the curated data catalogue

• Drive improvements in data acquisition, testing and tooling to raise data quality

• Monitor performance and optimise SQL queries

• Prepare detailed design documents based on business and system requirement documents

• Work in cross-functional agile teams and take part in requirement analysis and estimates

Requirements:

• 5+ years of experience in data engineering or a related field, with a strong focus on ETL and cloud data tools

• Bachelor's degree in Computer Science, Engineering or a related field (Master's preferred)

• Certifications: Microsoft Certified Azure Data Engineer and Databricks Certified Data Engineer

• Proficiency in Python, Java or Scala

• Hands-on experience with MS Cloud SQL Server, Azure Data Factory, Azure Databricks, PySpark, Spark SQL, ADLS, Power BI, SSIS and SSRS

• Strong SQL skills and experience with relational databases

• Familiarity with big data technologies and data processing frameworks

• Experience with Agile/Scrum

• Strong problem-solving and communication skills and a collaborative mindset

• English at C1 level

Nice to have:

• Experience in financial services, ideally in a captive finance company, including customer, dealer and supplier data

• Experience delivering global data marts used across regions

• Java