AI-Driven Data Engineer & Architect (Databricks) - Hybrid

Prohires; Austin (TX); Hybrid; USD 140,000 - 210,000; Full time

Job summary

Prohires is seeking a Data Engineer/Architect specializing in AI Agent Automation for Azure Databricks in Austin, TX. This hybrid role requires 3 days onsite weekly and a strong background in designing intelligent agents to automate data workflows.

Ideal candidates have 10+ years in data or software engineering, deep expertise in PySpark, SQL, and production-grade code, and experience with LLM-based automation frameworks and CI/CD pipelines.

Qualifications

• : Must have 10+ years in data engineering or software engineering with cloud data platforms (Databricks, Snowflake, Azure).
• : Proficient in Python and SQL with production-grade code quality.
• : Experience building automation/AI-powered agents and using LLMs for data workflows.
• : Strong knowledge of Databricks notebooks, jobs, workflows, and delta lake.
• : Familiarity with CI/CD pipelines and QA automation.

Responsibilities

• Design and implement intelligent agents to automate data engineering workflows in Azure Databricks.
• Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads.
• Build agents for data quality validation, testing, and reconciliation in dev/QA environments.
• Create agents for metadata management, lineage tracking, and governance within Databricks.
• Develop autonomous CI/CD tasks including test generation, deployment validation, and rollback logic.
• Implement agents for monitoring, anomaly detection, and root-cause analysis of data pipelines.
• Establish best practices for agent development, testing, deployment, and observability in Azure Databricks.
• Document agent workflows, decision logic, and runbooks for team knowledge sharing.

Skills

Python

SQL

LLM

AI/Automation

Data engineering

Education

Bachelor's degree in CS/SE/Data Science

Master's degree preferred

Tools

Azure Databricks

LangChain

OpenAI/Azure OpenAI

Databricks Agent Framework

GitHub Actions/Azure DevOps

Job description

Prohires is seeking a Data Engineer/Architect specializing in AI Agent Automation for Azure Databricks in Austin, TX. This hybrid role requires 3 days onsite weekly and a strong background in designing intelligent agents to automate data workflows.

Ideal candidates have 10+ years in data or software engineering, deep expertise in PySpark, SQL, and production-grade code, and experience with LLM-based automation frameworks and CI/CD pipelines.