Senior Data & AI Engineer: Delta Lake & Databricks on Azure

Accenture; Irving (TX); Hybrid; USD 100,000 - 190,000; Full time

Job summary

Accenture seeks an experienced Data & AI Engineer to design, build, and optimize scalable data pipelines using Azure Databricks and Delta Lake. You will work closely with analytics, data science, and cloud teams to ensure reliable data delivery for client initiatives.

This hybrid role requires three days in the office and offers growth within a global network of technology and data experts, covering data modeling, integration, AI agent frameworks, and governance.

Qualifications

• Minimum 3 years as a Data & AI Engineer or related role.
• Minimum 3 years of experience with Microsoft Azure cloud data services.
• Minimum 2 years of Databricks experience (PySpark, SQL, Delta Lake, workflows).
• Bachelor’s degree or equivalent work experience (12 years total).

Responsibilities

• Architect, develop, and maintain scalable, low‑latency data pipelines to ingest and transform diverse datasets using Azure Databricks and Spark.
• Design and optimize Lakehouse architecture with Bronze/Silver/Gold layers for governed data storage.
• Collaborate with data scientists and analysts to deliver clean, validated datasets for analytics and AI initiatives.
• Design and implement autonomous agents using Mosaic AI Agent Framework and LangChain/AutoGen.
• Build and optimize vector search indexes for accurate context retrieval.
• Process large‑scale datasets on the Databricks Intelligence Platform with Apache Spark.
• Model data to meet business needs and enhance usability.
• Continuously improve data architectures for efficiency and effectiveness.
• Design AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and observability.
• Monitor pipeline health with automated alerts and observability for rapid troubleshooting.

Skills

Data & AI Engineer

Education

Bachelor's degree or equivalent

Tools

Azure Databricks

Spark

Delta Lake

LangChain

AutoGen

Databricks Mosaic AI Agent Framework

GitHub Actions

Azure DevOps

Job description

Accenture seeks an experienced Data & AI Engineer to design, build, and optimize scalable data pipelines using Azure Databricks and Delta Lake. You will work closely with analytics, data science, and cloud teams to ensure reliable data delivery for client initiatives.

This hybrid role requires three days in the office and offers growth within a global network of technology and data experts, covering data modeling, integration, AI agent frameworks, and governance.