Data Engineer/Architect with AI Agent Automation Engineer (Databricks)

Prohires; Austin (TX); Hybrid; USD 140,000 - 210,000; Full time

Job summary

Prohires is seeking a Data Engineer/Architect specializing in AI Agent Automation for Azure Databricks in Austin, TX. This hybrid role requires 3 days onsite weekly and a strong background in designing intelligent agents to automate data workflows.

Ideal candidates have 10+ years in data or software engineering, deep expertise in PySpark, SQL, and production-grade code, and experience with LLM-based automation frameworks and CI/CD pipelines.

Qualifications

• : Must have 10+ years in data engineering or software engineering with cloud data platforms (Databricks, Snowflake, Azure).
• : Proficient in Python and SQL with production-grade code quality.
• : Experience building automation/AI-powered agents and using LLMs for data workflows.
• : Strong knowledge of Databricks notebooks, jobs, workflows, and delta lake.
• : Familiarity with CI/CD pipelines and QA automation.

Responsibilities

• Design and implement intelligent agents to automate data engineering workflows in Azure Databricks.
• Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads.
• Build agents for data quality validation, testing, and reconciliation in dev/QA environments.
• Create agents for metadata management, lineage tracking, and governance within Databricks.
• Develop autonomous CI/CD tasks including test generation, deployment validation, and rollback logic.
• Implement agents for monitoring, anomaly detection, and root-cause analysis of data pipelines.
• Establish best practices for agent development, testing, deployment, and observability in Azure Databricks.
• Document agent workflows, decision logic, and runbooks for team knowledge sharing.

Skills

Python

SQL

LLM

AI/Automation

Data engineering

Education

Bachelor's degree in CS/SE/Data Science

Master's degree preferred

Tools

Azure Databricks

LangChain

OpenAI/Azure OpenAI

Databricks Agent Framework

GitHub Actions/Azure DevOps

Job description

Hi,

Hope you are doing great!!!

Data Engineer/Architect with AI Agent Automation Engineer (Databricks)

Azure Databricks Environment*******

Austin, TX Need only Locals or very nearby places within 50Miles

Hybrid Role (3d onsite in a week)

Duration: 12 Months

Must have skills: Databricks( 5-10 Years), Azure cloud, LLM, AI

Responsibilities

• Design and implement intelligent agents (LLM-powered, rule-based, or hybrid) that automate data engineering workflows in Azure Databricks
• Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads
• Build agents that automate data quality validation, testing, and reconciliation tasks in dev and QA environments
• Create agents for metadata management, lineage tracking, and governance automation within Databricks
• Develop autonomous CI/CD pipeline tasks, including automated test case generation, deployment validation, and rollback logic
• Implement agents that perform intelligent monitoring, anomaly detection, and root-cause analysis for data pipelines
• Build prompt engineering frameworks and agent orchestration patterns tailored to data engineering use cases
• Collaborate with data engineers and architects to identify high-impact automation opportunities and prioritize agent development roadmap
• Establish best practices for agent development, testing, deployment, and observability within Azure Databricks
• Document agent workflows, decision logic, and operational runbooks for team knowledge sharing
• Evaluate emerging AI/agent frameworks and technologies; recommend adoption where applicable

Education

• Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field (Master's preferred)

Experience

• 10 years of data engineering or software engineering experience, with hands-on work in cloud data platforms (Databricks, Snowflake, Azure)
• Proven experience developing automation solutions, agents, or AI-powered tools (e.g., using LLMs, prompt engineering, or orchestration frameworks like Databricks Agent Framework (or LangChain, AutoGen)
• Strong proficiency in Python and SQL with focus on production-grade code quality
• Demonstrated experience with Azure Databricks (notebooks, jobs, workflows, DLT) or equivalent modern data platforms
• Hands‑on experience with CI/CD/CT tools (GitHub Actions, Azure DevOps Pipelines) and automation frameworks
• Experience building data quality frameworks, testing patterns, or QA automation
• Prior work in regulated or risk-sensitive environments preferred

Skills & Competencies

• Expert‑level Python programming with strong software engineering practices (testing, documentation, version control)
• Strong SQL and PySpark optimization skills
• Hands‑on experience with LLM APIs, prompt engineering, and AI agent frameworks (e.g., Databricks Agent Bricks, LangChain, OpenAI, Azure OpenAI)
• Proficiency in Azure Databricks, including Delta Lake, workflows, and Databricks Asset Bundles
• Experience designing and implementing automation workflows using orchestration tools or serverless platforms
• Strong problem‑solving and systems thinking skills; ability to decompose complex workflows into automatable components
• Excellent communication and documentation skills; ability to articulate technical concepts and agent design rationale
• Self‑directed learner with curiosity about emerging AI/automation technologies
• Ability to collaborate effectively with cross‑functional teams (data engineers, architects, QA, operations)