Data Platform Engineer: AWS/Databricks & Data Migrations

International Business Machines Corporation; Denver (CO); Remote; USD 110,000 - 150,000; Full time

Job summary

IBM Consulting seeks a Data Engineer embedded with client teams to design and deliver hands-on solutions on the AWS/Databricks platform. You will re-engineer legacy data workloads and support data migration from MySQL to Databricks Lakebase, collaborating with SAP data integrations and non-SAP data products.

You will design end-to-end data solutions across multiple platforms, establish engineering standards, and coach teams.

Qualifications

• Demonstrated data engineering and solution architecture experience.
• Experience migrating data workloads and re-engineering schemas.

Responsibilities

• Work onsite or embedded with client teams to understand business problems and turn them into working technical solutions.
• Design and build data pipelines and integrations using Fivetran, Databricks, and AWS services.
• Lead technical design and delivery for migrating a Tier-1 application from MySQL to Databricks Lakebase, including schema re-design, data migration, cutover, and validation.
• Integrate SAP data through SAP BTP and BDC Connect, non-SAP data, and design domain centric data products.
• Build prototypes and reference implementations, then work with the client to harden them for production.
• Support data and AI use cases, including integration with GCP-based ML/GenAI components.
• Establish engineering standards: CI/CD, testing, observability, security, and cost control.
• Coach client and delivery team members, and communicate status, risks, and design decisions to stakeholders.
• Design end-to-end data solutions across multiple platforms, applying industry standards and best practices to meet client needs.
• Lead the development of data platform architectures, including cognitive components and unstructured data technologies, annotation, and related data and analytics.
• Manage and maintain large data repositories, ensuring scalability and performance.
• Create data solutions on cloud-native platforms (AWS, with GCP for ML/GenAI), tailored to client requirements.
• Collaborate with clients to understand their needs and deliver customized data solutions that meet their business objectives.

Skills

Databricks

AWS

Python

SQL

Data architecture

CI/CD

Kafka

GCP ML/GenAI

MySQL

Tools

Databricks

AWS

MySQL

SAP BTP

SAP BDC Connect

Databricks Lakebase

Terraform

Git

Delta Lake

Apache Iceberg

Kinesis / EventBridge / SQS

Job description

IBM Consulting seeks a Data Engineer embedded with client teams to design and deliver hands-on solutions on the AWS/Databricks platform. You will re-engineer legacy data workloads and support data migration from MySQL to Databricks Lakebase, collaborating with SAP data integrations and non-SAP data products.

You will design end-to-end data solutions across multiple platforms, establish engineering standards, and coach teams.