Senior Data Platform Engineer: ELT, PySpark & Pipelines

CI&T; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

CI&T is seeking a data engineer to translate legacy Databricks notebook logic into modern ELT patterns using Python and PySpark, ensuring data contracts are preserved. You will design scalable ingestion pipelines and oversee DAG creation with YAML configurations.

You will collaborate with business Data Stewards to align dependencies, define refactoring scope, and validate migrated outputs for production systems, applying quality checks and migration validation throughout the process.

Responsibilities

• Translate legacy Databricks notebook logic into modern ELT patterns using Python and PySpark, ensuring data contract preservation and reverse-view strategies.
• Build scalable ingestion pipelines with YAML configurations and orchestrate automated DAG generation via workflow schedulers, applying quality assertions and migration validation.
• Collaborate with business Data Stewards to align dependencies, negotiate refactoring scope, and validate migrated outputs for production systems.

Tools

Python

PySpark

Databricks

YAML

Workflow schedulers

Job description

CI&T is seeking a data engineer to translate legacy Databricks notebook logic into modern ELT patterns using Python and PySpark, ensuring data contracts are preserved. You will design scalable ingestion pipelines and oversee DAG creation with YAML configurations.

You will collaborate with business Data Stewards to align dependencies, define refactoring scope, and validate migrated outputs for production systems, applying quality checks and migration validation throughout the process.