Data Engineer

SGI; Helsinki; On-site; EUR 70,000 - 110,000; Full time

Job summary

SGI in Helsinki seeks a Data Engineer to migrate SAS workloads to the Databricks Lakehouse Platform, translating legacy SAS data processing into PySpark and SparkR, and designing scalable pipelines with reliable Delta Lake tables.

You will collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations, and implement CI/CD with Git and Azure DevOps while validating data through automated testing and data quality checks.

Qualifications

• Experience migrating SAS workloads to modern data platforms.
• Ability to translate SAS data processing into PySpark and SparkR.
• Experience designing scalable data pipelines for large datasets.
• Strong understanding of Delta Lake performance and reliability.

Responsibilities

• Migrate SAS workloads to the Databricks Lakehouse Platform.
• Translate legacy SAS data processing into PySpark and SparkR.
• Design scalable data pipelines for reliable data workflows.
• Optimize Delta Lake tables for performance and reliability.
• Collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations.
• Implement CI/CD with Git and Azure DevOps and validate data with automated tests and data quality checks.

Skills

Databricks

PySpark

SAS basics

ETL

SQL

Job description

Assignment Overview

For our customer we are looking for a Data Engineer to migrate SAS workloads to the Databricks Lakehouse Platform. You will translate legacy SAS data processing into PySpark and SparkR, design scalable pipelines, and optimize Delta Lake tables for performance and reliability. Collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations, implement CI/CD with Git and Azure DevOps, and validate data through automated testing and data quality checks.

Key Responsibilities

• Migrate SAS workloads to the Databricks Lakehouse Platform.
• Translate legacy SAS data processing into PySpark and SparkR
• Design scalable pipelines
• Optimize Delta Lake tables for performance and reliability
• Collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations
• Implement CI/CD with Git and Azure DevOps, and validate data through automated testing and data quality checks.

Required Experience Mandatory

• Databricks

Good Experience to have

• PySpark
• SAS, Basics
• ETL
• SQL

Additional Information

Language requirements: h
Finnish mandotary
English
Start date: As soon as possible after candidate is choosen

We believe in equal opportunity for all and actively encourage applications from diverse backgrounds, experiences, and perspectives.