Senior Data Engineer, AI Pipelines & Data Quality

Dairy-Farmers-of-America,-Inc.; Kansas City (KS); On-site; USD 99,000 - 139,000; Full time

Job summary

Dairy-Farmers-of-America, Inc. is seeking a senior data engineer to design, build, and optimize scalable ETL/ELT pipelines for AI/ML data needs.

You will ingest, transform, and integrate structured and unstructured enterprise data for model training and inference, ensuring accuracy, reliability, and security. Required: 8+ years in data engineering or related fields, proficiency with Python, SQL, PySpark, Snowflake, Databricks, Airflow, and Azure Data Factory; experience with cloud data

Qualifications

• Undergraduate degree in Computer Science, Data Engineering, Information Systems, or related field.
• 8+ years data engineering, software development, or related experience, including data pipelines for AI/ML systems.
• Proficiency with Python, SQL, PySpark, Snowflake, Databricks, and Apache Airflow.
• Strong knowledge of relational and NoSQL databases, data warehouses or data lakes, Azure Data Factory, and Kafka.
• Experience with at least one major cloud data ecosystem, preferably Azure, and familiarity with infrastructure-as-code and automated deployment practices.
• Certifications in cloud data engineering or data privacy/security and experience with MLOps or AI/ML lifecycle management preferred.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines that ingest, transform, and integrate structured and unstructured enterprise data for AI/ML model training and inference
• Ensure data is accurate, current, reliable, and secure through cleansing, normalization, validation, automation, continuous updates, monitoring, and error handling
• Troubleshoot complex integration and performance issues, remove bottlenecks, and apply appropriate techniques such as caching, distributed processing, and specialized ETL patterns
• Own key AI data infrastructure components and ensure they are scalable, maintainable, and supported by effective monitoring, logging, documentation, data lineage, and quality metrics
• Apply enterprise architecture, AI governance, privacy, security, coding, and CI/CD standards to data solutions and communicate risks or limitations to leadership
• Partner with Software engineers, data architects, domain experts, product teams, and platform teams to align data solutions with model requirements, business goals, and enterprise standards
• Independently deliver assigned work, influence cross-team decisions, promote data engineering best practices, and evaluate technologies that improve AI data delivery
• The requirements herein describe the general nature and level of work performed by the employee but are not a complete list of responsibilities, duties, and skills required. Other duties may be assigned as needed

Skills

Python

SQL

PySpark

Snowflake

Databricks

Apache Airflow

Azure Data Factory

Kafka

Cloud data platforms (Azure)

Data governance

Education

Bachelor's degree in Computer Science, Data Engineering, Information Systems, or related field

Job description

Dairy-Farmers-of-America, Inc. is seeking a senior data engineer to design, build, and optimize scalable ETL/ELT pipelines for AI/ML data needs.

You will ingest, transform, and integrate structured and unstructured enterprise data for model training and inference, ensuring accuracy, reliability, and security. Required: 8+ years in data engineering or related fields, proficiency with Python, SQL, PySpark, Snowflake, Databricks, Airflow, and Azure Data Factory; experience with cloud data