Senior Data Engineer: Scalable Lakes & AI Pipelines Remote

AgileEngine; Boca Raton (FL); Hybrid; USD 140,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive USD-based pay

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a thematic research platform processing large financial datasets daily.

You will implement Python ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role blends engineering with technical consulting to align data roadmaps with business goals.

Qualifications

• Authorized to work in the US (visa sponsorship not provided).
• Bachelor’s degree in CS/engineering or equivalent experience.
• 5+ years of Python experience.
• 5+ years of data processing with Pandas/Polars/PySpark/DuckDB.
• 2+ years with Spark/Snowflake.
• Expert-level Airflow or similar workflow orchestration.
• Medallion Architecture, columnar formats, SQL/NoSQL tech.
• Experience with 3rd party API data ingestion.
• Cloud proficiency: AWS/GCP/Snowflake and SQL optimization.
• Strong cross-functional collaboration and communication.
• Data pipeline integrity and latency troubleshooting.
• Self-starter mindset and interest in AI tooling.

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop ETL/ELT processes with Python and Airflow.
• Ingest data from diverse 3rd party APIs and sources.
• Optimize file formats (Parquet/Avro/ORC) for performance.
• Work with AI tools to accelerate analytics.
• Translate business goals into technical roadmaps.
• Collaborate on AWS/on-prem infrastructure using Terraform.

Skills

Python

Pandas

Polars

PySpark

DuckDB

Spark

Airflow

SQL

NoSQL

Data architectures

AI tools

English (Advanced)

Education

Bachelor’s degree in CS/engineering or equivalent

Tools

Airflow

Snowflake

Spark

DuckDB

Pandas

Polars

PySpark

Terraform

AWS

GCP

Job description

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a thematic research platform processing large financial datasets daily.

You will implement Python ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role blends engineering with technical consulting to align data roadmaps with business goals.