Azure AI Architect

Prominds Business Consulting Inc; Chicago (IL); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Prominds Business Consulting Inc. seeks an Azure Databricks & Agentic AI Architect in Chicago for a long-term contract. The role focuses on building AI-powered data platforms with Lakehouse architecture, autonomous data engineering, and governance frameworks.

You will lead enterprise AI adoption, design agent orchestration, GenAI integrations, and observability while collaborating with CXOs on strategy and business value realization.

Qualifications

• 15+ years of Data & Analytics experience.
• Strong Azure Databricks expertise.
• Hands-on experience building Agentic AI platforms and AI-powered SDLC frameworks.

Responsibilities

• Design and implement AI-powered Data Engineering platforms using Azure Databricks and Lakehouse architecture.
• Lead autonomous data ingestion, transformation, quality, and lineage with AI Agents.
• Define HITL governance and enable AI-driven data lifecycle management across the pipeline.

Skills

Azure Databricks

Agentic AI

GenAI

Leadership

CXO Engagement

Tools

Azure Data Factory

Delta Lake

Unity Catalog

ADLS Gen2

Databricks Workflows

Job description

• Data ingestion
• Pipeline generation
• Cost optimization

Chicago, IL( Monthly 2-3 days onsite)

Long term contract

Azure Databricks & Agentic AI Architect

A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.

Role Summary

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.

The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Key Responsibilities

Agentic Data Engineering Leadership

• Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
• Define autonomous workflows using AI Agents for:
• Data ingestion
• Data mapping
• Schema evolution
• Data quality remediation
• Metadata Enrichment
• Pipeline optimization
• Root cause analysis
• Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.

AI-Driven Data Lifecycle (AI-DLC)

• Lead architecture for AI-enabled Data Development Lifecycle across:
• Requirement analysis
• Data modeling
• Pipeline generation
• Automated testing
• Code review
• Documentation
• Deployment
• Monitoring
• Implement AI copilots to accelerate developer productivity.
• Enable automated lineage creation and intelligent impact analysis.

Lakehouse & Data Platform Architecture

• Design scalable Lakehouse platforms using:
• Azure Databricks
• Delta Lake
• Unity Catalog
• ADLS Gen2
• Databricks Workflows
• Delta Live Tables

Enterprise GenAI Integration

• Architect RAG-based solutions using enterprise data assets.
• Design agent orchestration frameworks using:
• Azure OpenAI
• LangGraph
• Semantic Kernel
• AutoGen
• MCP-enabled architectures
• Build domain-specific AI agents supporting Data Engineering and Analytics teams.

AI Governance & Responsible AI

• Define guardrails for enterprise GenAI adoption.
• Implement:
• Prompt governance
• Observability
• Cost monitoring
• Auditability
• Explainability
• Security controls
• Establish governance models for autonomous AI agents.

AI-Powered Platform Optimization

• Design self-healing data pipelines.
• Implement AI-driven:
• Incident triage
• Failure prediction
• Capacity planning
• Cost optimization
• SLA monitoring
• Enable intelligent workload placement and model routing.

Technical Skills

• Data Platform
• Azure Databricks
• Delta Lake
• Unity Catalog
• Azure Data Factory

AI & Agentic Frameworks

• Azure OpenAI
• Knowledge Graph
• RAG Architecture
• LangChain
• LangGraph
• MCP Protocol
• Vector Databases
• AI Agent Orchestration

Data Engineering

• PySpark
• Spark SQL
• Python
• SQL
• ELT/ETL Modernization

DevOps & AI-DLC

• Azure DevOps
• GitHub Actions
• CI/CD
• MLOps
• LLMOps
• Evaluation Frameworks
• AI Testing Frameworks

Leadership Expectations

• Drive AI-First Data Engineering transformation.
• Define enterprise patterns, accelerators, and reusable AI agents.
• Mentor architects, data engineers, and AI engineers.
• Lead executive conversations on AI adoption, ROI, and transformation roadmaps.

Preferred Certifications

• Databricks Certified Data Engineer Professional
• Databricks Certified Solution Architect
• Microsoft Azure Solution Architect (AZ-305)
• Azure Data Engineer (DP-203)
• Microsoft Applied Skills – Azure OpenAI
• Generative AI / Agentic AI Certifications