Senior Data Engineer: Scalable AI Pipelines & Governance

Dairy Challenge; Kansas City, Northern (KS, KY); Hybrid; USD 99,000 - 139,000; Full time

Job summary

Dairy Challenge is seeking an experienced Data Engineer to design, build, and optimize scalable ETL/ELT pipelines supporting AI/ML model training and inference. You will ensure data accuracy, security, and reliability through cleansing, validation, and continuous updates, while troubleshooting complex integration issues and promoting best practices.

Ideal candidates will have 8+ years in data engineering, strong Python/SQL/PySpark skills, and experience with Azure Data Factory, Kafka, Snowflake,

Qualifications

• Undergraduate degree in Computer Science, Data Engineering, Information Systems, or related field
• 8+ years data engineering, software development, or related experience including data pipelines for AI/ML systems
• Proficiency with Python, SQL, PySpark, Snowflake, Databricks, and Apache Airflow
• Strong knowledge of relational and NoSQL databases, data warehouses or data lakes, and streaming tech like Kafka
• Experience with at least one major cloud data ecosystem, preferably Azure, and familiarity with infrastructure-as-code and automated deployment practices
• Certifications in cloud data engineering or data privacy/security and experience with MLOpspreferred

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines for AI/ML model training and inference
• Ensure data quality, accuracy, and security through cleansing, normalization, validation, automation, and monitoring
• Troubleshoot complex integration and performance issues and remove bottlenecks
• Own key AI data infrastructure components with monitoring, logging, data lineage, and quality metrics
• Apply enterprise architecture, AI governance, privacy, security, and CI/CD standards to data solutions
• Collaborate with software engineers, data architects, domain experts, product and platform teams to align data solutions with model requirements and business goals
• Deliver assigned work independently, influence cross-team decisions, promote data engineering best practices
• Note: duties may vary and other responsibilities may be assigned

Skills

Python

SQL

PySpark

Snowflake

Databricks

Apache Airflow

Kafka

Azure

Education

Undergraduate degree in Computer Science, Data Engineering, Information Systems, or related field

Tools

Azure Data Factory

CI/CD

IaC (infrastructure-as-code)

Job description

Dairy Challenge is seeking an experienced Data Engineer to design, build, and optimize scalable ETL/ELT pipelines supporting AI/ML model training and inference. You will ensure data accuracy, security, and reliability through cleansing, validation, and continuous updates, while troubleshooting complex integration issues and promoting best practices.

Ideal candidates will have 8+ years in data engineering, strong Python/SQL/PySpark skills, and experience with Azure Data Factory, Kafka, Snowflake,