Senior Data Engineer (AI & Agents)

Appnovation; Austin (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Appnovation seeks a Senior Data Engineer, AI & Agents to prepare enterprise domain data for AI agents and collaborate with business stakeholders to ensure governed, high-quality data assets.

You will build domain data agents over governed datasets, perform lakehouse migrations to open formats like Apache Iceberg, and generate catalogue metadata for downstream automation and data-contract workflows using Databricks and Snowflake. Strong cross-team collaboration is essential.

Qualifications

• 5+ years of professional data engineering experience with modern warehousing or lakehouse platforms.
• Hands-on with Databricks and Snowflake for data platform work.
• Experience building domain-grounded data agents over governed datasets.
• Strong governance, masking, and data security practices.
• Excellent stakeholder-facing and collaboration skills.

Responsibilities

• Prepare enterprise domain data for AI agents and stakeholder consumption.
• Build and register domain data agents over governed tables.
• Perform lakehouse migrations and open-format conversions (e.g., Iceberg).
• Generate and curate catalogue metadata for data-contract workflows.
• Partner with business stakeholders through iterative build, test, and validation cycles.

Skills

Databricks

Snowflake

PySpark

SQL

Airflow

Education

Bachelor's or Master's in Computer Science, Information Systems, Engineering, or related field

Tools

dbt

Git

Apache Iceberg

Python

Job description

• As a Senior Data Engineer, AI & Agents, you will prepare enterprise domain data for consumption by AI agents, partnering directly with business stakeholders and subject-matter experts
• The role centers on building and registering domain-grounded data agents over governed datasets, performing lakehouse migrations, and onboarding new data domains — including commercial, finance, research and development, and real-world data — onto the enterprise data platform
• You will operate across Databricks and Snowflake, converting source tables to open formats and generating the catalogue metadata that powers downstream automation and data-contract workflows, all while ensuring high data quality, governed access, and reliable agent-based access to trusted data
• Build and register domain data agents at scale over governed tables across Databricks and Snowflake
• Perform lakehouse migrations, including converting source tables to open formats (e.g., Apache Iceberg) to enable agent-based access
• Generate and curate catalogue metadata that feeds downstream automation and data-contract workflows
• Partner with business stakeholders and subject-matter experts through iterative build, test, and validation cycles

The ideal candidate brings deep, hands-on data engineering expertise, strong governance instincts, and excellent stakeholder-facing skills5+ years of professional experience in data engineering, with significant hands-on experience across modern data warehousing and lakehouse platforms (Databricks and Snowflake preferred)Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related fieldStrong data engineering background with genuine, hands-on fluency across both Databricks and SnowflakeDemonstrated experience building data agents or query interfaces over governed datasets (e.g., Snowflake Cortex or Genie)Experience implementing data quality, observability, and lineage, and applying governance controls such as masking and row- and column-level securityExcellent stakeholder-facing skills, with a track record of translating business requirements into delivered data assetsAdvanced SQL together with Spark / PySpark, and experience with pipeline orchestration (dbt, Apache Airflow, or Databricks Workflows)Foundations: Python; YAML data contracts; Apache Iceberg; Git and CI/CDAI and agents: MCP; vector databases and embeddingsData platforms: Databricks, Snowflake (Cortex, Genie); AWS and S3Governance and catalogue: Unity Catalogue, Horizon, CollibraPipelines and modelling: SQL, PySpark, dbt, Airflow / Databricks WorkflowsTechnical Experience:Agent-Oriented Builder: You enjoy turning governed datasets into reliable, domain-grounded agents that business users can query with confidenceQuality-Focused: You are rigorous about data accuracy, lineage, and observability, ensuring high standards through validation before data reaches agents or the businessCollaborative Partner: You thrive working directly with stakeholders and subject-matter experts through iterative build, test, and validation cyclesForward-Thinking: You are interested in the "big picture" of lakehouse architecture and open formats, eager to advance agent-based access patterns and best practicesGovernance-Minded: You understand the critical nature of data security in regulated domains and proactively apply masking and row- and column-level controlsWho you are:Experience with AWS and S3, in anticipation of onboarding native cloud data sourcesFamiliarity with MCP-based data exposure and with embeddings or vector search for retrieval-augmented use casesExperience with regulated life-sciences data domains (clinical, commercial, or real-world data)