Databricks AI Engineer

Ven Soft LLC; Austin (TX); On-site; USD 96,000 - 207,000; Part time

Job summary

Ven Soft LLC seeks a senior data/AI engineer to architect and deploy AI agents that automate Databricks workflows. You will build LL-powered tools for code generation, testing, and data quality validation, integrating with Azure services and enterprise systems.

The role combines data engineering, AI agent development, and automation with emphasis on governance, observability, and security. Austin onsite with long-term contract terms.

Qualifications

• Bachelor’s degree in CS/Engineering, data science, or related field.
• 10+ years in data or software engineering or cloud data platforms.
• Hands-on with Azure Databricks and modern cloud data platforms.
• Proven AI/LLM-powered automation and intelligent agents experience.
• Strong Python and SQL, production-grade apps.

Responsibilities

• Design and deploy AI agents to automate data engineering workflows in Databricks.
• Build LL-powered agents for code generation, review, and refactoring of Python/SQL workloads.
• Develop AI-driven data quality validation, testing, and QA automation.
• Create governance, metadata management, and lineage automation in Databricks.
• Develop CI/CD automation, test generation, deployment validation, and rollback checks.
• Implement AI-powered monitoring for data pipelines and root-cause analysis.
• Document architectures, prompts, and orchestration patterns; ensure logging and observability.

Skills

AI agent design

Azure Databricks

Python

PySpark

Generative AI

LLMs

Automation

AI governance

CI/CD

Monitoring & observability

Education

Bachelor's degree in CS/ENG

Master's degree (preferred)

Tools

Databricks

Delta Lake

Databricks Asset Bundles

LangChain

Azure OpenAI

GitHub Actions

Azure DevOps

Job description

Location: Austin, TX (3 Days Onsite / Week)

Duration: Long-Term Contract

Note: The ideal candidate is not simply a traditional Databricks Data Engineer. We are looking for someone who can combine: Data Engineering + Azure Databricks + Python/PySpark + Generative AI + LLMs + AI Agents + Automation

Must-Have Skills:

Responsibilities & Skills:

• Design, develop, and deploy intelligent AI agents that automate data engineering workflows within Azure Databricks.
• Build LLM-powered agents for code generation, code review, refactoring, optimization, and troubleshooting of Python, PySpark, and SQL workloads.
• Develop AI-driven solutions to automate data quality validation, testing, reconciliation, and QA processes.
• Build agents to support metadata management, data lineage, governance, and compliance automation within Databricks.
• Develop intelligent automation for CI/CD pipelines, including test-case generation, deployment validation, release checks, and rollback processes.
• Implement AI-powered monitoring and automation for data pipeline failures, anomalies, and root-cause analysis.
• Develop reusable prompt engineering frameworks, agent patterns, tools, and orchestration workflows for data engineering use cases.
• Integrate LLMs and AI agents with Databricks, Azure services, APIs, data platforms, and enterprise workflows.
• Work with Data Engineers, Data Architects, QA teams, and Platform teams to identify high-value automation opportunities.
• Establish standards and best practices for AI agent development, testing, deployment, observability, security, and governance.
• Build production-grade automation solutions with appropriate logging, error handling, monitoring, testing, and documentation.
• Evaluate emerging Generative AI, agentic AI, and automation frameworks and recommend technologies that provide measurable business or engineering value.
• Document agent architecture, workflows, prompts, decision logic, integration patterns, and operational runbooks.

Required Qualifications:

• Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related technical discipline. Master's degree is a plus.
• 10+ years of experience in data engineering, software engineering, cloud data platforms, or related technical roles.
• Strong hands-on experience with Azure Databricks and modern cloud data platforms.
• Proven experience developing AI/LLM-powered automation, intelligent agents, or Generative AI solutions.
• Strong programming skills in Python and SQL with experience developing production-quality applications.
• Strong hands-on experience with PySpark and distributed data processing.
• Experience with Databricks notebooks, Jobs/Workflows, Delta Lake, and Databricks Asset Bundles.
• Experience with LLM APIs, prompt engineering, agent orchestration, and AI frameworks.
• Experience with technologies/frameworks such as Azure OpenAI, OpenAI, LangChain, AutoGen, Databricks Agent Framework/Agent Bricks, or comparable agentic AI technologies.
• Hands-on experience with CI/CD and automation, including GitHub Actions and/or Azure DevOps Pipelines.
• Experience building data quality, testing, QA automation, or data validation frameworks.
• Strong understanding of software engineering principles including Git, testing, version control, documentation, code quality, and observability.