Data Engineer

International Business Machines Corporation; Paramus (NJ); Remote; USD 120,000 - 180,000; Full time

Job summary

IBM Consulting seeks a Data Engineer embedded with client teams to design and deliver hands-on solutions on the AWS/Databricks platform. You will re-engineer legacy workloads and support migrations from MySQL to Databricks Lakebase, delivering scalable data products across cloud platforms.

Role involves architecture and engineering, building prototypes, and coaching teams while aligning with security, CI/CD, and cost controls. Travel and client collaboration are expected.

Qualifications

• Extensive experience designing and delivering data pipelines and architectures.
• Strong SQL, Python, and Databricks expertise.
• Knowledge of AWS data services and data migrations.

Responsibilities

• Work onsite or embedded with client teams to translate business problems into technical solutions.
• Design and build data pipelines and integrations with Fivetran, Databricks, and AWS.
• Lead data migration from MySQL to Databricks Lakebase, including schema redesign and cutover.
• Integrate SAP data via SAP BTP/BDC Connect and design domain-centric data products.
• Create prototypes and reference implementations; help production hardening.
• Support data and AI use cases with GCP ML/GenAI components.
• Establish CI/CD, testing, observability, security, and cost controls.
• Coach client teams and communicate status and risks to stakeholders.
• Design end-to-end data solutions across platforms; apply industry standards.
• Lead data platform architectures including cognitive components and unstructured data.

Skills

Databricks

AWS

Python

SQL

Data pipelines

CI/CD

GenAI

Terraform

Kubernetes

Tools

Databricks Lakebase

Fivetran

Job description

Introduction

A career in IBM Consulting is built on long-term client relationships and close collaboration worldwide. You'll work with leading companies across industries, helping them shape their hybrid cloud and AI journeys. With support from our strategic partners, robust IBM technology, and Red Hat, you'll have the tools to drive meaningful change and accelerate client impact. At IBM Consulting, curiosity fuels success. You'll be encouraged to challenge the norm, explore new ideas, and create innovative solutions that deliver real results. Our culture of growth and empathy focuses on your long-term career development while valuing your unique skills and experiences.

Your role and responsibilities

Data Engineer embedded with the client's teams to design and deliver hands-on solutions on the AWS/Databricks platform. The role combines architecture and engineering, and you will build and prove solutions with the client. Work includes re-engineering legacy data workloads and supporting data migration from MySQL to Databricks Lakebase.

Responsibilities

• Work onsite or embedded with client teams to understand business problems and turn them into working technical solutions.


• Design and build data pipelines and integrations using Fivetran, Databricks, and AWS services.


• Lead technical design and delivery for migrating a Tier-1 application from MySQL to Databricks Lakebase, including schema re-design, data migration, cutover, and validation.


• Integrate SAP data through SAP BTP and BDC Connect, non-SAP data, and design domain centric data products.


• Build prototypes and reference implementations, then work with the client to harden them for production.


• Support data and AI use cases, including integration with GCP-based ML/GenAI components.


• Establish engineering standards: CI/CD, testing, observability, security, and cost control.


• Coach client and delivery team members, and communicate status, risks, and design decisions to stakeholders.


• Design end-to-end data solutions across multiple platforms, applying industry standards and best practices to meet client needs.


• Lead the development of data platform architectures, including cognitive components and unstructured data technologies, annotation, and related data and analytics.


• Manage and maintain large data repositories, ensuring scalability and performance.


• Create data solutions on cloud-native platforms (AWS, with GCP for ML/GenAI), tailored to client requirements.


• Collaborate with clients to understand their needs and deliver customized data solutions that meet their business objectives.

This job can be performed from anywhere in the US

Required technical and professional expertise

• Must be able to travel 4 days per week (mandatory).


• Databricks Certified Data Engineer or Machine Learning Professional.


• Advanced hands-on expertise in data engineering and solution architecture, with demonstrated ability to design and deliver complex data solutions.


• Strong hands-on expertise with Databricks (Spark, Delta Lake, MLflow, Unity Catalog), AWS, SQL, and Python, including large-scale batch and streaming workloads.


• Experience with relational databases such as MySQL, database migration/re-engineering, Fivetran or similar ingestion/ELT technologies, and modern data-sharing patterns.


• Experience with Delta Lake, Apache Iceberg, and other open table formats, including schema evolution, optimisation, and interoperability across engines.


• Hands-on experience designing scalable data platforms and pipelines using technologies such as Kafka, Amazon EventBridge, or Amazon SQS, along with Terraform and CI/CD practices.

Preferred technical and professional experience

• Experience with Databricks Lakebase or similar managed PostgreSQL/OLTP-on-lakehouse technologies.


• Knowledge of SAP BTP, BDC Connect, SAP extraction patterns, or SAP IS-U.


• Exposure to GCP ML/GenAI services and cloud-native solution design across AWS and GCP.


• Experience with Git-based version control and branching strategies for data workloads.


• Experience in the utilities or energy industry, with relevant Databricks and/or AWS certification.

IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, caste, genetics, pregnancy, disability, neurodivergence, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.