Senior Data Engineer - Remote & Flexible Data Lakes

AgileEngine; Jacksonville (FL); Hybrid; USD 120,000 - 190,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouse architectures for processing large volumes of financial data daily. You will implement Python ETL/ELT pipelines, orchestrate workflows with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

This role blends hands-on engineering with technical consulting. You will translate business goals into data architecture roadmaps, collaborate with

Qualifications

• 5+ years of experience with Python
• 5+ years of experience with data processing and analytics libraries
• 2+ years of experience with Big Data technologies (Spark, Snowflake)
• Expert-level knowledge of pipeline orchestration using Airflow
• Strong understanding of Medallion Architecture and columnar formats
• Experience with 3rd party APIs for data ingestion
• Proficiency with cloud platforms (AWS, GCP) and SQL optimization
• Excellent cross-functional collaboration and communication

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Lakehouses.
• Develop ETL/ELT pipelines using Python and Airflow.
• Create ingestion workflows from diverse APIs and data sources.
• Optimize data formats for high-performance retrieval.
• Collaborate with AI teams on analytics and ML initiatives.
• Translate business goals into technical roadmaps and architectures.
• Work with Terraform to build AWS/on-prem infrastructure.

Skills

Python

Pandas

PySpark

DuckDB

Airflow

SQL

NoSQL

Data Lakes

Lakehouse

AWS

Education

Bachelor’s degree in CS/Engineering or related field

Tools

Snowflake

Spark

Terraform

Job description

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouse architectures for processing large volumes of financial data daily. You will implement Python ETL/ELT pipelines, orchestrate workflows with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

This role blends hands-on engineering with technical consulting. You will translate business goals into data architecture roadmaps, collaborate with