Remote Data Engineer - AWS/Databricks & GenAI Solutions

International Business Machines Corporation; Minneapolis (MN); Remote; USD 120,000 - 180,000; Full time

Job summary

IBM Consulting is seeking a Data Engineer to embed with client teams across the US to design and deliver hands-on data solutions on AWS/Databricks. You will re-engineer legacy workloads, migrate data from MySQL to Databricks Lakebase, and integrate SAP and non-SAP sources into domain-focused data products.

You will establish standards, coach teams, and ensure scalable, production-ready pipelines with CI/CD, security, and observability.

Qualifications

• Experience designing and delivering complex data solutions on cloud platforms.
• Experience migrating data workloads and re-architecting schemas for lakehouse architectures.

Responsibilities

• Work onsite or embedded with client teams to understand business problems and turn them into working technical solutions.
• Design and build data pipelines and integrations using Fivetran, Databricks, and AWS services.
• Lead technical design and delivery for migrating a Tier-1 application from MySQL to Databricks Lakebase, including schema re-design, data migration, cutover, and validation.
• Integrate SAP data through SAP BTP and BDC Connect, non-SAP data, and design domain centric data products.
• Build prototypes and reference implementations, then work with the client to harden them for production.
• Support data and AI use cases, including integration with GCP-based ML/GenAI components.
• Establish engineering standards: CI/CD, testing, observability, security, and cost control.
• Coach client and delivery team members, and communicate status, risks, and design decisions to stakeholders.
• Design end-to-end data solutions across multiple platforms, applying industry standards and best practices to meet client needs.
• Lead the development of data platform architectures, including cognitive components and unstructured data technologies, annotation, and related data and analytics.
• Manage and maintain large data repositories, ensuring scalability and performance.
• Create data solutions on cloud-native platforms (AWS, with GCP for ML/GenAI), tailored to client requirements.
• Collaborate with clients to understand their needs and deliver customized data solutions that meet their business objectives.

Skills

Databricks

AWS

SQL

Python

ETL

CI/CD

Kafka

Education

Databricks Certified Data Engineer

Tools

Databricks

AWS

MySQL

Terraform

CI/CD

Job description

IBM Consulting is seeking a Data Engineer to embed with client teams across the US to design and deliver hands-on data solutions on AWS/Databricks. You will re-engineer legacy workloads, migrate data from MySQL to Databricks Lakebase, and integrate SAP and non-SAP sources into domain-focused data products.

You will establish standards, coach teams, and ensure scalable, production-ready pipelines with CI/CD, security, and observability.