Senior Data Engineer - AI Data Pipelines & Infra

Dairy Farmers of America, Inc.; Kansas City, Northern (KS, KY); Hybrid; USD 99,000 - 139,000; Full time

Job summary

Dairy Farmers of America, Inc. is seeking an experienced data engineer to design and optimize scalable ETL/ELT pipelines for AI/ML model training and inference.

You will ensure data accuracy, security, and governance while collaborating with software engineers, data architects, and product teams. Responsibilities include troubleshooting complex data integration issues, owning AI data infrastructure components, and promoting best practices.

Qualifications

• Undergraduate degree in Computer Science, Data Engineering, Information Systems, or related field.
• 8+ years data engineering, software development, or related experience, including work on data pipelines for AI/ML systems.
• Proficiency with Python, SQL, PySpark, Snowflake, Databricks, and Apache Airflow.
• Strong knowledge of relational and NoSQL databases, data warehouses or data lakes, integration platforms such as Azure Data Factory, and streaming technologies such as Kafka.
• Experience with at least one major cloud data ecosystem, preferably Azure, and familiarity with infrastructure-as-code and automated deployment practices.
• Certifications in cloud data engineering or in data privacy/security and experience with MLOps or AI/ML lifecycle management preferred.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines for AI/ML data needs.
• Ensure data accuracy, cleanliness, and security with monitoring and error handling.
• Troubleshoot integration and performance issues and remove bottlenecks.
• Own data infrastructure components and ensure they are scalable and well-documented.
• Communicate risks and align data solutions with enterprise standards across teams.
• Deliver assigned work independently and promote data engineering best practices.

Skills

ETL/ELT design

Cross-team collaboration

Problem solving

English proficiency

Education

Undergraduate degree in CS/Data Eng/IS

Tools

Python

SQL

PySpark

Snowflake

Databricks

Apache Airflow

Azure Data Factory

Kafka

Job description

Dairy Farmers of America, Inc. is seeking an experienced data engineer to design and optimize scalable ETL/ELT pipelines for AI/ML model training and inference.

You will ensure data accuracy, security, and governance while collaborating with software engineers, data architects, and product teams. Responsibilities include troubleshooting complex data integration issues, owning AI data infrastructure components, and promoting best practices.