Databricks Data Engineer: SAS Migration & Delta Lake

SGI; Helsinki; On-site; EUR 70,000 - 110,000; Full time

Job summary

SGI in Helsinki seeks a Data Engineer to migrate SAS workloads to the Databricks Lakehouse Platform, translating legacy SAS data processing into PySpark and SparkR, and designing scalable pipelines with reliable Delta Lake tables.

You will collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations, and implement CI/CD with Git and Azure DevOps while validating data through automated testing and data quality checks.

Qualifications

• Experience migrating SAS workloads to modern data platforms.
• Ability to translate SAS data processing into PySpark and SparkR.
• Experience designing scalable data pipelines for large datasets.
• Strong understanding of Delta Lake performance and reliability.

Responsibilities

• Migrate SAS workloads to the Databricks Lakehouse Platform.
• Translate legacy SAS data processing into PySpark and SparkR.
• Design scalable data pipelines for reliable data workflows.
• Optimize Delta Lake tables for performance and reliability.
• Collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations.
• Implement CI/CD with Git and Azure DevOps and validate data with automated tests and data quality checks.

Skills

Databricks

PySpark

SAS basics

ETL

SQL

Job description

SGI in Helsinki seeks a Data Engineer to migrate SAS workloads to the Databricks Lakehouse Platform, translating legacy SAS data processing into PySpark and SparkR, and designing scalable pipelines with reliable Delta Lake tables.

You will collaborate with analysts and data scientists to ensure functional equivalence between SAS and Databricks implementations, and implement CI/CD with Git and Azure DevOps while validating data through automated testing and data quality checks.