Remote Databricks Data Engineer – Lakehouse Migration

Brilliant Infotech Inc.; Washington (District of Columbia); On-site; USD 130,000 - 170,000; Full time

Job summary

The Databricks Data Engineer owns the end-to-end migration strategy, target architecture design, and technical execution of moving legacy ETL workloads to the Databricks Lakehouse. He will establish migration standards, optimize PySpark pipelines, orchestrate complex data workflows, and deploy proprietary automation tools to ensure a seamless, high-performing transition from legacy systems.

Location: Remote. Duration: 12+ Months.

Qualifications

• Databricks certification is required.
• Experience migrating legacy ETL workloads to a Lakehouse is expected.
• Strong PySpark, Python and SQL skills are essential.

Responsibilities

• Architectural Strategy & Governance: design target Databricks Lakehouse architecture (Delta Lake, Photon, Unity Catalog).
• Hands-on Engineering & Optimization: refactor legacy logic into PySpark notebooks, optimize performance with Spark features.
• Build and deploy migration accelerators and robust Databricks workflows for complex DAGs.

Skills

PySpark

Python

SQL

Shell Scripting

AWS

Apache Airflow

Tools

Databricks

Delta Lake

Unity Catalog

Photon

DataStage

Airflow

Databricks Workflows

Job description

The Databricks Data Engineer owns the end-to-end migration strategy, target architecture design, and technical execution of moving legacy ETL workloads to the Databricks Lakehouse. He will establish migration standards, optimize PySpark pipelines, orchestrate complex data workflows, and deploy proprietary automation tools to ensure a seamless, high-performing transition from legacy systems.

Location: Remote. Duration: 12+ Months.