Remote Data Platform Engineer: Azure→Databricks on AWS

OpsBrasil Serviços Cloud LTDA; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

OpsBrasil Serviços Cloud LTDA is seeking a senior Data Engineer to convert a large Azure Data Factory estate into Databricks workflows on AWS. The role includes rehosting Databricks workspaces, migrating storage to S3, and rewriting ADF activities as Lambda or Step Functions tasks.

You will build PySpark transformations for production-scale data, replace Synapse with Databricks SQL Warehouse, and ensure migrated data reconciles with source systems as part of the definition of done.

Qualifications

• Production experience with Databricks: workspaces, jobs and workflows.
• Strong Spark and PySpark experience for large data volumes, including tuning.
• Production-grade Python development.
• Experience migrating Azure Data Factory pipelines, with solid understanding of the ADF activity model.
• Advanced SQL, including reading and reasoning about stored procedures.
• Professional written and spoken English.

Responsibilities

• Convert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates.
• Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.
• Rewrite ADF Web Activities as Lambda functions or Step Functions tasks; replace ADF scaling with Databricks mechanisms.
• Build and tune PySpark transformations for production data volumes.
• Replace Azure Synapse Serverless with Databricks SQL Warehouse.
• Reconcile migrated data against source systems as part of the definition of done.

Skills

Databricks

Spark

PySpark

Python

SQL

English

Tools

S3

Glue

Athena

Lambda

Step Functions

Job description

OpsBrasil Serviços Cloud LTDA is seeking a senior Data Engineer to convert a large Azure Data Factory estate into Databricks workflows on AWS. The role includes rehosting Databricks workspaces, migrating storage to S3, and rewriting ADF activities as Lambda or Step Functions tasks.

You will build PySpark transformations for production-scale data, replace Synapse with Databricks SQL Warehouse, and ensure migrated data reconciles with source systems as part of the definition of done.