Data Engineer- Silver Conversions and Scheduling

PLATO; Stati Uniti; Remoto; USD 140.000 - 190.000; Tempo pieno

Descrizione del lavoro

PLATO is seeking a contract role in the United States to build and operate Silver-layer data conversion pipelines in a medallion Lakehouse, turning raw Bronze data into cleansed datasets. This role focuses on dependable scheduling, data integrity and performance using Microsoft Fabric and Azure data services.

The contract runs from December 2026 to March 2027 with a strong likelihood of extension. You will configure pipelines, ensure retry mechanisms, implement logging, monitoring and unit

Competenze

• At least five years of experience with Microsoft Fabric including Data Factory and Lakehouse.
• At least five years of experience using PySpark and Delta Lake for data transformations.
• Experience building and scheduling pipelines with Fabric Data Factory or Azure Data Factory.
• Exposure to Microsoft Purview for data cataloguing and lineage.
• Experience with Azure DevOps, Git and CI/CD pipelines.

Mansioni

• Develop Silver-layer transformations with Fabric, Azure Data Factory, Spark and Delta Lake.
• Configure pipeline schedules and dependencies, including retry and failure handling.
• Use reusable PySpark libraries and metadata-driven configuration.
• Apply logging, monitoring, error handling and unit-testing practices.
• Maintain data integrity through reconciliation, checkpoints, replay and recovery.
• Tune Spark workloads for runtime and cost efficiency.

Conoscenze

Data engineering

Cloud data platforms

CI/CD practices

Strumenti

Microsoft Fabric

Azure Data Factory

Spark

Delta Lake

Azure Storage

PySpark

Spark SQL

Purview

Azure DevOps

Git

Descrizione del lavoro

Role overview

Build and operate Silver-layer data conversion pipelines in a medallion Lakehouse, turning raw Bronze data into cleansed, consistent datasets. This contract role focuses on dependable scheduling, data integrity and performance using Microsoft Fabric and Azure data services. The stated term is December 2026 to March 2027, with a good likelihood of extension.

Responsibilities

• Develop Silver-layer transformations with Microsoft Fabric, Azure Data Factory, Spark and Delta Lake.
• Configure pipeline schedules and dependencies, including retry and failure-handling behavior.
• Use reusable PySpark libraries and metadata-driven configuration to support efficient delivery.
• Apply consistent logging, monitoring, error handling and unit-testing practices.
• Maintain data integrity through reconciliation, checkpoints, replay and recovery processes.
• Tune Spark workloads for runtime and cost efficiency.

Requirements

• At least five years of experience with Microsoft Fabric, including Data Factory and Lakehouse, alongside Azure Data Factory and Azure Storage.
• At least five years of experience using PySpark, Spark SQL and Delta Lake for data transformations.
• Experience building and scheduling pipelines with Fabric Data Factory or Azure Data Factory.
• Exposure to Microsoft Purview for data cataloguing and lineage.
• Experience with Azure DevOps, Git and CI/CD pipelines.
• Active Reliability status and eligibility to obtain Secret clearance.

Benefits and work setup

Work alongside experienced technology professionals on client projects in a collaborative environment that supports knowledge-sharing, mentorship and inclusive hiring. Accommodation is available during the recruitment process.