Data Engineer - AWS/Databricks & GenAI Pipelines

International Business Machines Corporation; San Diego (CA); Remote; USD 120,000 - 180,000; Full time

Job summary

IBM Consulting is seeking a Data Engineer embedded with client teams to design and deliver hands-on data solutions on the AWS/Databricks platform. The role blends architecture and engineering, including re-engineering legacy workloads and migrating from MySQL to Databricks Lakebase.

You will design end-to-end data solutions across multiple platforms, establish engineering standards, and coach delivery teams. Travel up to four days per week is expected as needed.

Qualifications

• Databricks Certified Data Engineer or ML Professional required.
• Advanced hands-on expertise in data engineering and solution architecture.
• Strong hands-on Databricks experience (Spark, Delta Lake, MLflow, Unity Catalog), AWS, SQL and Python.
• Experience with MySQL migration/re-engineering and data sharing patterns.
• Knowledge of Delta Lake, Apache Iceberg and schema evolution.
• Hands-on design of scalable data platforms using Kafka, EventBridge, SQS, Terraform and CI/CD.

Responsibilities

• Work onsite or embedded with client teams to translate business problems into technical data solutions.
• Design and build data pipelines and integrations using Fivetran, Databricks, and AWS services.
• Lead data migration from MySQL to Databricks Lakebase with schema redesign, cutover and validation.
• Integrate SAP data via SAP BTP/BDC Connect and other data sources; design data products.
• Prototype and harden solutions for production; support AI/ML use cases with cloud glue (GCP/AWS).
• Establish engineering standards: CI/CD, testing, observability, security and cost controls.
• Coach client and delivery teams; communicate status, risks and decisions to stakeholders.
• Design end-to-end data solutions across platforms; lead architecture of data platforms and governance.
• Manage large data repositories ensuring scalability and performance; collaborate with clients on requirements.

Skills

Data engineering

Solution architecture

Client collaboration

Mentoring/coaching

Tools

Databricks

AWS

SQL

Python

Fivetran

Terraform

CI/CD

GCP

Job description

IBM Consulting is seeking a Data Engineer embedded with client teams to design and deliver hands-on data solutions on the AWS/Databricks platform. The role blends architecture and engineering, including re-engineering legacy workloads and migrating from MySQL to Databricks Lakebase.

You will design end-to-end data solutions across multiple platforms, establish engineering standards, and coach delivery teams. Travel up to four days per week is expected as needed.