Data Team Lead

Veridic Solutions LLC; United States; On-site; USD 140,000 - 210,000; Full time

Job summary

Veridic Solutions LLC is seeking an experienced AI/agentic data engineer to design and implement autonomous data platforms with multi-agent orchestration across cloud regions. You will define agent roles, workflows, and data quality enforcement at scale.

You will collaborate with data product owners and analysts to translate business requirements into agent specifications, SLAs, and performance optimization strategies, while ensuring auditing and governance capabilities across distributed

Qualifications

• Proven experience designing and implementing multi-agent systems
• Deep understanding of agent orchestration, communication protocols, and inter-agent coordination patterns
• Experience with agent-driven automation in data pipelines, ETL workflows, or similar complex distributed systems
• Knowledge of prompt engineering, agent reasoning patterns, and retrieval-augmented generation (RAG) for data-specific contexts
• Familiarity with agentic AI frameworks and LLM integration patterns for data engineering use cases

Responsibilities

• Design and architect multi-agent data platforms with autonomous ETL/ELT pipelines.
• Architect agent-driven data lake/lakehouse solutions with medallion architecture across multi-region environments.
• Develop governance frameworks leveraging Unity Catalog, ABAC, and RLS for autonomous enforcement.
• Create multi-agent orchestration patterns for scalable cross-region workflows.
• Build agent-driven data quality frameworks with autonomous validation and alerts.
• Collaborate with product owners and analysts to translate requirements into agent specs and SLAs.
• Implement logging, monitoring, and audit trails for transparency and compliance.
• Design fallback and human-in-the-loop workflows for complex scenarios.
• Stay current with agent frameworks, LLM integration patterns, and best practices in autonomous data engineering

Skills

Python

SQL

Agent orchestration

LLM integration

Asynchronous processing

Distributed data processing

Tools

Databricks

Delta Lake

Spark

Notebooks

DLT

AgentBricks

Git

CI/CD

GitOps

Unity Catalog

Job description

• Design and architect multi-agent systems that autonomously manage ETL/ELT pipelines using Databricks, Delta Lake, and Auto Loader, with specialized agents for data ingestion, validation, transformation, and quality monitoring
• Architect agent-driven data lake/lakehouse solutions with medallion architecture (Bronze/Silver/Gold layers) where autonomous agents handle promotion logic, schema evolution, and data quality enforcement across multi-region environments
• Develop intelligent agentic governance frameworks leveraging Unity Catalog, ABAC policies, and Row-Level Security (RLS), where agents autonomously detect data sensitivity levels, enforce fine-grained access control, and manage compliance across distributed systems
• Design and implement multi-agent orchestration patterns that enable agents to communicate, coordinate workflows, share state, and resolve conflicts in a scalable manner across multiple cloud regions
• Build agent-driven data quality frameworks with autonomous quality monitoring agents that detect data anomalies, enforce schema contracts, trigger remediation workflows, and generate real-time alerts
• Architect agents for autonomous data discovery, cataloging, and metadata management using Unity Catalog, enabling self-describing data ecosystems
• Collaborate with data product owners, analysts, and business stakeholders to translate requirements into agent specifications, SLAs, and performance optimization strategies for agentic systems
• Implement comprehensive logging, monitoring, and audit trails for multi-agent systems to ensure transparency, accountability, and compliance across geographically distributed deployments
• Design fallback mechanisms, error recovery patterns, and human-in-the-loop workflows for complex scenarios requiring agent intervention or escalation
• Stay current with evolving agent frameworks, LLM integration patterns, Databricks agentic capabilities, AWS services, and industry best practices in autonomous data engineering

Required Qualifications

Agentic AI & Multi-Agent Systems

• Proven experience designing and implementing multi-agent systems and agentic workflows
• Deep understanding of agent orchestration, communication protocols, and inter-agent coordination patterns
• Experience with agent-driven automation in data pipelines, ETL workflows, or similar complex distributed systems
• Knowledge of prompt engineering, agent reasoning patterns, and retrieval-augmented generation (RAG) for data-specific contexts
• Familiarity with agentic AI frameworks and LLM integration patterns for data engineering use cases

Technical Skills

• Proficiency in Python and SQL for agentic system and data pipeline development
• Hands-on experience with Databricks (Delta Lake, Spark, Notebooks, Jobs, DLT, AgentBricks) and agentic job orchestration
• Deep understanding of data lake and lakehouse architecture patterns, including medallion architecture and autonomous layer promotion
• Strong expertise in ETL/ELT design patterns, distributed data processing, and autonomous pipeline patterns
• Experience implementing ABAC (Attribute-Based Access Control) and Row-Level Security (RLS) frameworks with autonomous enforcement
• Familiarity with Unity Catalog for data governance and autonomous metadata management
• Experience with multi-region data architecture, federation, data residency requirements, and cross-region agent coordination
• Proficiency with version control (Git), CI/CD pipelines, and GitOps for agent configuration management
• Understanding of data quality frameworks, autonomous validation patterns, and self-healing pipelines
• Strong background in asynchronous processing, event-driven architectures, and message queuing for agent communication

Domain Expertise

• Working knowledge of insurance industry terminology, business processes, and data models
• Familiarity with common insurance data entities: Party, Claims, Fiduciary
• Understanding of regulatory requirements relevant to insurance and how they apply to autonomous systems

Professional Skills

• Strong problem-solving and analytical thinking abilities applied to complex system design
• Excellent communication skills to explain agentic architectures and technical concepts to non-technical stakeholders
• Collaborative mindset with experience working in cross-functional teams and translating business requirements into agent specifications
• Ability to manage multiple priorities and projects involving large-scale, autonomous systems
• Multi-agent system architecture documentation with agent roles, communication flows, and coordination patterns
• Autonomous ETL/ELT pipeline implementations with agents managing data quality and transformations
• Agent-driven governance frameworks enforcing ABAC and Row-Level Security (RLS) across multi-region deployments
• Monitoring and observability solutions for multi-agent data systems

Runbooks and operational guides for managing agentic data platforms