Databricks Engineer / Remote / Contract

Motion Recruitment; United States; Remote; USD 83,000 - 152,000; Full time

Job summary

Motion Recruitment is seeking a Senior Integration Engineer for a fully remote contract in the United States. You will lead enterprise data integration initiatives, building data pipelines with IBM DataStage, PySpark, Databricks, Python, SQL, and Delta Lake to deliver governed analytics assets.

The role focuses on scalable ETL design, production troubleshooting, and modern Databricks architecture within a large automotive manufacturing-like data environment, with an initial 3-month contract and

Qualifications

• Hands-on IBM DataStage and PySpark experience.
• Strong Python and SQL development skills.
• Experience ETL and data integration pipelines.
• Databricks experience and multi-source ingestion.
• Bronze-Silver-Gold data-layer architecture understanding.
• Data transformations, incremental processing, schema enforcement, data quality.
• Troubleshooting production pipelines; error handling and recovery.
• Familiarity with GitHub, version control, code reviews, CI/CD.

Responsibilities

• Develop ETL pipelines with IBM DataStage and PySpark.
• Build and optimize PySpark/Databricks pipelines.
• Implement Python/SQL transformations.
• Leverage Delta Lake and Spark Declarative Pipelines.
• Code reviews, CI/CD, testing and monitoring.
• Document data flows and source-to-target mappings.

Skills

IBM DataStage

PySpark

Databricks

Python

SQL

ETL pipelines

Data integration

Data quality

GitHub

CI/CD

Tools

Databricks

Sterling

GitHub

CI/CD

Job description

We are working with a leading automotive manufacturing company seeking a Senior Integration Engineer for a fully remote contract opportunity in the United States. This role will focus on enterprise data integration and analytics using IBM DataStage, PySpark, Databricks, Python, SQL, and Delta Lake. The initial contract is three months, with renewals every three months.

This is an opportunity to work on enterprise-scale data integration initiatives, building reliable pipelines that connect applications and transform data into governed assets for reporting and analytics. The team is looking for a hands-on engineer who can develop, troubleshoot, and optimize production data pipelines while contributing to modern Databricks architecture, data quality, and automation practices. The role offers exposure to modern data engineering technologies within a large automotive manufacturing environment.

Contract Duration: 3 months, with renewals every 3 months

Required Skills & Experience

• Hands-on experience with IBM DataStage and PySpark.
• Strong Python and SQL development skills.
• Experience designing, developing, and supporting ETL and data integration pipelines.
• Experience with Databricks and data ingestion from multiple sources, including files, APIs, and databases.
• Understanding of Bronze, Silver, and Gold data-layer architecture.
• Experience with data transformations, incremental processing, schema enforcement, and data quality.
• Experience troubleshooting production data pipelines and implementing error handling and recovery.
• Familiarity with GitHub, version control, code reviews, and CI/CD practices.

Desired Skills & Experience

• Experience with Delta Lake and Spark Declarative Pipelines.
• Experience integrating data through enterprise transfer services such as Sterling.
• Familiarity with automated testing, pipeline monitoring, and quarantine patterns.
• Experience developing governed data products for reporting and analytics.
• Ability to document solution architecture, data flows, dependencies, and source-to-target mappings.
• Experience working in large enterprise or automotive manufacturing environments.

What You Will Be Doing

Tech Breakdown

• 30% IBM DataStage and ETL development
• 30% PySpark and Databricks
• 20% Python and SQL transformations
• 10% Delta Lake and Spark Declarative Pipelines
• 10% GitHub, CI/CD, testing, and monitoring

Daily Responsibilities

• 50% Hands-on development, data pipeline engineering, and troubleshooting
• 20% Data quality, testing, deployment, and production support
• 15% Architecture documentation and source-to-target mapping
• 15% Team collaboration with project and platform teams