Databricks AI Engineer

IT America Inc; Austin (TX); Hybrid; USD 150,000 - 190,000; Full time

Job summary

IT America Inc. seeks a Databricks AI Engineer in Austin, TX, with hybrid onsite (3 days/week) for a long-term contract. You will design and deploy AI agents to automate data engineering workflows in Azure Databricks, leveraging PySpark, Python, SQL, and LLM-based automation.

Responsibilities include building AI-powered automation for data quality, CI/CD pipelines, and governance, plus collaboration with data teams to scale AI-driven data platforms.

Qualifications

• Bachelor's degree or higher in a technical field.
• Experience with Azure Databricks and cloud data platforms.
• Proven AI/LLM-powered automation experience.

Responsibilities

• Design, develop, and deploy intelligent AI agents to automate data engineering within Azure Databricks.
• Build LLM-powered agents for code generation, review, refactoring, and optimization of PySpark/SQL workloads.
• Develop AI-driven automation for data quality validation, testing, reconciliation, and QA processes.
• Create agents for metadata management, data lineage, governance, and compliance automation.
• Implement CI/CD for AI workflows, including tests, deployment validation, and rollback.
• Monitor and diagnose data pipelines with AI-powered observability and root-cause analysis.
• Develop reusable prompt engineering frameworks and agent patterns for data engineering use cases.
• Integrate LLMs/agents with Databricks, Azure services, APIs, and enterprise workflows.

Skills

Azure Databricks

Python

PySpark

LLMs

Generative AI

AI Agents

Prompt Engineering

CI/CD

Data Automation

Data Engineering

Education

Bachelor's degree in CS/SE/Data Science

Master's degree (plus)

Tools

Databricks notebooks

Delta Lake

Databricks Asset Bundles

GitHub Actions

Azure DevOps Pipelines

LangChain

OpenAI API

Azure OpenAI

Job description

Position: Databricks AI Engineer

Location: Austin, TX (3 Days Onsite / Week)

Duration: Long-Term Contract

Environment: Azure Databricks

Note:

The ideal candidate is not simply a traditional Databricks Data Engineer. We are looking for someone who can combine: Data Engineering + Azure Databricks + Python/PySpark + Generative AI + LLMs + AI Agents + Automation

Must-Have Skills:

Azure Databricks | Azure Cloud | Python | SQL | PySpark | LLMs | Generative AI | AI Agents | Prompt Engineering | CI/CD | Data Automation

Responsibilities & Skills:

• Design, develop, and deploy intelligent AI agents that automate data engineering workflows within Azure Databricks.
• Build LLM-powered agents for code generation, code review, refactoring, optimization, and troubleshooting of Python, PySpark, and SQL workloads.
• Develop AI-driven solutions to automate data quality validation, testing, reconciliation, and QA processes.
• Build agents to support metadata management, data lineage, governance, and compliance automation within Databricks.
• Develop intelligent automation for CI/CD pipelines, including test-case generation, deployment validation, release checks, and rollback processes.
• Implement AI-powered monitoring and automation for data pipeline failures, anomalies, and root-cause analysis.
• Develop reusable prompt engineering frameworks, agent patterns, tools, and orchestration workflows for data engineering use cases.
• Integrate LLMs and AI agents with Databricks, Azure services, APIs, data platforms, and enterprise workflows.
• Work with Data Engineers, Data Architects, QA teams, and Platform teams to identify high-value automation opportunities.
• Establish standards and best practices for AI agent development, testing, deployment, observability, security, and governance.
• Build production-grade automation solutions with appropriate logging, error handling, monitoring, testing, and documentation.
• Evaluate emerging Generative AI, agentic AI, and automation frameworks and recommend technologies that provide measurable business or engineering value.
• Document agent architecture, workflows, prompts, decision logic, integration patterns, and operational runbooks.

Required Qualifications:

• Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related technical discipline. Master's degree is a plus.
• 10+ years of experience in data engineering, software engineering, cloud data platforms, or related technical roles.
• Strong hands-on experience with Azure Databricks and modern cloud data platforms.
• Proven experience developing AI/LLM-powered automation, intelligent agents, or Generative AI solutions.
• Strong programming skills in Python and SQL with experience developing production-quality applications.
• Strong hands-on experience with PySpark and distributed data processing.
• Experience with Databricks notebooks, Jobs/Workflows, Delta Lake, and Databricks Asset Bundles.
• Experience with LLM APIs, prompt engineering, agent orchestration, and AI frameworks.
• Experience with technologies/frameworks such as Azure OpenAI, OpenAI, LangChain, AutoGen, Databricks Agent Framework/Agent Bricks, or comparable agentic AI technologies.
• Hands-on experience with CI/CD and automation, including GitHub Actions and/or Azure DevOps Pipelines.
• Experience building data quality, testing, QA automation, or data validation frameworks.
• Strong understanding of software engineering principles including Git, testing, version control, documentation, code quality, and observability.