Senior AI Data Engineer: Scalable Pipelines & Governance

Dairy Farmers of America; Kansas City (KS); On-site; USD 120,000 - 180,000; Full time

Job summary

Dairy Farmers of America in Kansas City seeks a senior data engineer to design, build, and optimize scalable ETL/ELT pipelines for AI/ML model training and inference across enterprise data sources.

You will own AI data infrastructure components, ensure data quality and security, and collaborate with software engineers, data architects, and product teams to deliver reliable data solutions aligned with business goals.

Qualifications

• Undergraduate degree in Computer Science, Data Engineering, Information Systems, or related field.
• 8+ years data engineering, software development, or related experience, including work on data pipelines for AI/ML systems.
• Proficiency with data engineering languages, platforms, and orchestration tools such as Python, SQL, PySpark, Snowflake, Databricks, and Apache Airflow.
• Strong knowledge of relational and NoSQL databases, data warehouses or data lakes, integration platforms such as Azure Data Factory, and streaming technologies such as Kafka.
• Experience with at least one major cloud data ecosystem, preferably Azure, and familiarity with infrastructure-as-code and automated deployment practices.
• Certifications in cloud data engineering or in data privacy/security and experience with MLOps or AI/ML lifecycle management preferred.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines for AI/ML model training and inference across enterprise data sources.
• Ensure data accuracy, currentness, reliability, and security through cleansing, normalization, validation, automation, monitoring, and error handling.
• Troubleshoot complex integration and performance issues, remove bottlenecks, and apply techniques such as caching and distributed processing.
• Own AI data infrastructure components and ensure they are scalable, maintainable, and supported by monitoring, logging, data lineage, and quality metrics.
• Apply enterprise architecture, AI governance, privacy, security, coding, and CI/CD standards to data solutions and communicate risks to leadership.
• Partner with software engineers, data architects, domain experts, product teams, and platform teams to align data solutions with model requirements and business goals.
• Independently deliver assigned work, influence cross-team decisions, promote data engineering best practices, and evaluate technologies that improve AI data delivery.
• Other duties may be assigned as needed.

Skills

Data engineering

Python

SQL

PySpark

Snowflake

Databricks

Airflow

Kafka

Azure

CI/CD

MLOps

Education

Bachelor's degree in CS/Data Eng

Tools

Azure Data Factory

CI/CD tooling

Job description

Dairy Farmers of America in Kansas City seeks a senior data engineer to design, build, and optimize scalable ETL/ELT pipelines for AI/ML model training and inference across enterprise data sources.

You will own AI data infrastructure components, ensure data quality and security, and collaborate with software engineers, data architects, and product teams to deliver reliable data solutions aligned with business goals.