Databricks & AI Data Architect

IBM; Austin (TX); Remote; USD 150,000 - 190,000; Full time

Job summary

IBM Consulting seeks a Databricks Solution Architect to define and lead end-to-end architecture for a large data modernization program. You will bring deep expertise in cloud-based engineering, Databricks, data modeling, lakehouse architecture, and AI solutions across hyperscalers.

You will drive governance, MLOps, and secure data movement while collaborating with executives and technical teams to deliver enterprise-scale outcomes.

Qualifications

• Databricks Certified Data Engineer or ML Professional, with proven enterprise-scale data, AI/ML, and Generative AI solutions.
• Advanced expertise in data/solution architecture, enterprise data migration, data modelling, lakehouse/data-product architecture, reporting, and cloud-based data engineering using Databricks and AWS.
• Hands-on experience with Spark (PySpark/Scala), Delta Lake, MLflow, Unity Catalog, and large-scale batch and streaming workloads.

Responsibilities

• Define and lead the target data and AI architecture across the AWS/Databricks data platform and GCP-based ML/GenAI services.
• Lead the data migration architecture, including migration patterns, integration & ingestion patterns, sequencing, cutover, reconciliation, and risk management.
• Own data architecture and data modeling standards across lakehouse layers and data products.
• Define data engineering patterns for ingestion (Fivetran), transformation, orchestration, CI/CD, observability, and cost control.
• Architect the reporting and analytics layer with semantic models and governed metrics.
• Design the ML/GenAI lifecycle including feature engineering, model training, serving, MLOps/LLMOps, monitoring.
• Architect secure, scalable cross-cloud data movement across AWS/Azure/GCP.
• Ensure integration of Databricks lakehouse data products and SAP data via SAP BTP/BDC Connect.
• Define governance guardrails for AI use cases.
• Provide architectural oversight on initiatives like MySQL to Databricks Lakebase migrations.

Skills

Databricks

cloud-architecture

Data engineering

Spark PySpark/Scala

Delta Lake

MLflow

Unity Catalog

GPT/GenAI governance

APIs and microservices

communication with executives

Education

Databricks Certified Data Engineer or ML Professional

Tools

AWS

Azure

GCP

Databricks Lakehouse

Fivetran

Kafka

Job description

IBM Consulting seeks a Databricks Solution Architect to define and lead end-to-end architecture for a large data modernization program. You will bring deep expertise in cloud-based engineering, Databricks, data modeling, lakehouse architecture, and AI solutions across hyperscalers.

You will drive governance, MLOps, and secure data movement while collaborating with executives and technical teams to deliver enterprise-scale outcomes.