Senior Data Engineer - Data Lakes, AI Pipelines & Cloud

AgileEngine; Miami (OK); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a thematic research platform handling large volumes of financial data daily. You will implement Python ETL/ELT pipelines, orchestrate workflows with Airflow, and work with Snowflake, Spark, and AWS for high-performance data infrastructure.

The role blends hands-on engineering with technical consulting, translating business goals into data architecture roadmaps and collaborating with

Qualifications

• 5+ years of Python experience in data engineering.
• 5+ years of data processing and analytics libraries (Pandas, Polars, PySpark, DuckDB).
• 2+ years of Big Data technologies (Spark, Snowflake).
• Expert-level pipeline orchestration using Airflow or similar.
• Deep understanding of SQL, NoSQL, and Lakehouse architectures.
• Proven ability to work with 3rd party APIs for data ingestion.
• Strong communication and cross-functional collaboration with leadership.

Responsibilities

• Design and implement Python data engineering solutions.
• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT processes at scale using Python and Airflow.
• Create ingestion workflows from diverse third-party APIs and data sources.
• Optimize file formats (Parquet, Avro, ORC) and storage for performance.
• Collaborate with AI teams to accelerate analytics and ML initiatives.
• Translate business goals into data architecture roadmaps and technical plans.
• Work with Terraform to build AWS and on-prem infrastructure.

Skills

Python

Pandas

Polars

PySpark

DuckDB

Big Data

Airflow

SQL

NoSQL

Lakehouse

ML/AI workflows

Education

Bachelor's degree in computer science/engineering or equivalent

Tools

Airflow

Spark

Snowflake

AWS

GCP

Terraform

Job description

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a thematic research platform handling large volumes of financial data daily. You will implement Python ETL/ELT pipelines, orchestrate workflows with Airflow, and work with Snowflake, Spark, and AWS for high-performance data infrastructure.

The role blends hands-on engineering with technical consulting, translating business goals into data architecture roadmaps and collaborating with