Senior Databricks Engineer - Remote Data Pipelines & ETL

Motion Recruitment; United States; Remote; USD 83,000 - 152,000; Full time

Job summary

Motion Recruitment is seeking a Senior Integration Engineer for a fully remote contract in the United States. You will lead enterprise data integration initiatives, building data pipelines with IBM DataStage, PySpark, Databricks, Python, SQL, and Delta Lake to deliver governed analytics assets.

The role focuses on scalable ETL design, production troubleshooting, and modern Databricks architecture within a large automotive manufacturing-like data environment, with an initial 3-month contract and

Qualifications

• Hands-on IBM DataStage and PySpark experience.
• Strong Python and SQL development skills.
• Experience ETL and data integration pipelines.
• Databricks experience and multi-source ingestion.
• Bronze-Silver-Gold data-layer architecture understanding.
• Data transformations, incremental processing, schema enforcement, data quality.
• Troubleshooting production pipelines; error handling and recovery.
• Familiarity with GitHub, version control, code reviews, CI/CD.

Responsibilities

• Develop ETL pipelines with IBM DataStage and PySpark.
• Build and optimize PySpark/Databricks pipelines.
• Implement Python/SQL transformations.
• Leverage Delta Lake and Spark Declarative Pipelines.
• Code reviews, CI/CD, testing and monitoring.
• Document data flows and source-to-target mappings.

Skills

IBM DataStage

PySpark

Databricks

Python

SQL

ETL pipelines

Data integration

Data quality

GitHub

CI/CD

Tools

Databricks

Sterling

GitHub

CI/CD

Job description

Motion Recruitment is seeking a Senior Integration Engineer for a fully remote contract in the United States. You will lead enterprise data integration initiatives, building data pipelines with IBM DataStage, PySpark, Databricks, Python, SQL, and Delta Lake to deliver governed analytics assets.

The role focuses on scalable ETL design, production troubleshooting, and modern Databricks architecture within a large automotive manufacturing-like data environment, with an initial 3-month contract and