Data Engineer - Airflow Pipelines & Cloud (Hybrid, 6Mo)

Agile Resources, Inc.; Maryland Heights (MO); Hybrid; USD 69,000 - 94,000; Full time

Job summary

Agile Resources, Inc. is seeking an experienced Data Engineer for a 6-month contract in St. Louis, MO. The role is hybrid—4 days onsite and 1 day remote—focusing on building and maintaining production data pipelines using Python, SQL, and Apache Airflow.

You will design ETL/ELT processes, optimize data delivery, troubleshoot production issues, and support cloud-based data platforms. Strong Docker/Kubernetes, CI/CD, and Agile collaboration are required to succeed.

Qualifications

• 4–7 years of professional Data Engineering experience.
• Strong proficiency with Python and advanced SQL.
• Hands-on experience with Apache Airflow, including development, scheduling, monitoring, and troubleshooting of DAGs.

Responsibilities

• Author, debug, and optimize complex data engineering scripts using Python and advanced SQL.
• Design, develop, schedule, and monitor complex Apache Airflow DAGs and data pipelines.
• Build and maintain ETL/ELT processes, data transformations, and data integration workflows.
• Design and manage SQL-based databases, including schema development, query performance tuning, and data warehousing.
• Improve existing pipelines by resolving recurring failures, eliminating bottlenecks, and optimizing resource utilization.
• Install, configure, and maintain data pipeline utilities, custom Airflow providers, and database connectors.
• Support cloud-based data platform infrastructure and data lake/data warehouse integration efforts.
• Work with containerization technologies such as Docker and Kubernetes.
• Identify opportunities to automate, innovate, and scale data delivery across the organization.
• Perform root cause analysis and incident management for data quality issues, pipeline/DAG failures, and database outages.
• Create and maintain technical documentation covering data architecture, DAG workflows, and operational troubleshooting procedures.
• Collaborate with engineering and technical teams using Agile development methodologies.
• Support CI/CD deployment pipelines for data engineering code and workflows.

Skills

Python

Advanced SQL

Airflow

ETL/ELT

Data Warehousing

Cloud Data Platforms

Docker/Kubernetes

CI/CD

Agile

Tools

Docker

Kubernetes

Job description

Agile Resources, Inc. is seeking an experienced Data Engineer for a 6-month contract in St. Louis, MO. The role is hybrid—4 days onsite and 1 day remote—focusing on building and maintaining production data pipelines using Python, SQL, and Apache Airflow.

You will design ETL/ELT processes, optimize data delivery, troubleshoot production issues, and support cloud-based data platforms. Strong Docker/Kubernetes, CI/CD, and Agile collaboration are required to succeed.