Remote Cloud Data Engineer: Databricks & AI Pipelines

International Business Machines Corporation; Paramus (NJ); Remote; USD 120,000 - 180,000; Full time

Job summary

IBM Consulting seeks a Data Engineer embedded with client teams to design and deliver hands-on solutions on the AWS/Databricks platform. You will re-engineer legacy workloads and support migrations from MySQL to Databricks Lakebase, delivering scalable data products across cloud platforms.

Role involves architecture and engineering, building prototypes, and coaching teams while aligning with security, CI/CD, and cost controls. Travel and client collaboration are expected.

Qualifications

• Extensive experience designing and delivering data pipelines and architectures.
• Strong SQL, Python, and Databricks expertise.
• Knowledge of AWS data services and data migrations.

Responsibilities

• Work onsite or embedded with client teams to translate business problems into technical solutions.
• Design and build data pipelines and integrations with Fivetran, Databricks, and AWS.
• Lead data migration from MySQL to Databricks Lakebase, including schema redesign and cutover.
• Integrate SAP data via SAP BTP/BDC Connect and design domain-centric data products.
• Create prototypes and reference implementations; help production hardening.
• Support data and AI use cases with GCP ML/GenAI components.
• Establish CI/CD, testing, observability, security, and cost controls.
• Coach client teams and communicate status and risks to stakeholders.
• Design end-to-end data solutions across platforms; apply industry standards.
• Lead data platform architectures including cognitive components and unstructured data.

Skills

Databricks

AWS

Python

SQL

Data pipelines

CI/CD

GenAI

Terraform

Kubernetes

Tools

Databricks Lakebase

Fivetran

Job description

IBM Consulting seeks a Data Engineer embedded with client teams to design and deliver hands-on solutions on the AWS/Databricks platform. You will re-engineer legacy workloads and support migrations from MySQL to Databricks Lakebase, delivering scalable data products across cloud platforms.

Role involves architecture and engineering, building prototypes, and coaching teams while aligning with security, CI/CD, and cost controls. Travel and client collaboration are expected.