Senior Data Engineer: Data Lakes & AI Pipelines (Remote)

AgileEngine; Chicago (IL); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Professional growth

Competitive pay (USD)

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer in Chicago to design scalable data lakes, warehouses, and lakehouse architectures for a thematic research platform handling large volumes of financial data daily.

You will implement Python ETL/ELT pipelines, orchestrate workflows with Airflow, and integrate with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. This role blends hands-on engineering with technical consulting for business goals.

Qualifications

• Bachelor’s degree in computer science/engineering or other technical field is required.
• Minimum 5+ years of Python experience with hands-on coding.
• 5+ years of data processing and analytics libraries (Pandas, Polars, PySpark, DuckDB).
• 2+ years of Big Data tech experience (Spark, Snowflake).
• Expert in Airflow or similar workflow orchestration tools.
• Strong knowledge of Medallion Architecture, columnar formats, SQL/NoSQL, and lakehouse concepts.
• Experience with 3rd party APIs for data ingestion and cloud platforms (AWS, GCP, Snowflake).
• Excellent cross-functional collaboration and stakeholder communication.
• Upper-intermediate English language ability.

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT pipelines with Airflow and Python.
• Create ingestion workflows from diverse 3rd party APIs and data sources.
• Optimize Parquet/Avro/ORC formats for high-performance retrieval.
• Collaborate with AI/ML initiatives and analytics teams.
• Collect requirements from leadership and translate into architectures.
• Work with Terraform to provision AWS and on-prem infrastructure.

Job description

AgileEngine is seeking a Senior Data Engineer in Chicago to design scalable data lakes, warehouses, and lakehouse architectures for a thematic research platform handling large volumes of financial data daily.

You will implement Python ETL/ELT pipelines, orchestrate workflows with Airflow, and integrate with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. This role blends hands-on engineering with technical consulting for business goals.