Senior Data Engineer — Remote, Flexible Hours, Data Lakes

AgileEngine; Port Charlotte (FL); Hybrid; USD 140,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is an Inc. 5000 company delivering award-winning software for Fortune 500 brands and startups across 17+ industries. We foster a people-first culture with Best Place to Work awards.

We are seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses for a thematic research platform handling large financial datasets daily. You will build Python ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS.

Qualifications

• Bachelor’s degree in CS/Engineering or equivalent experience.
• 5+ years of Python development experience.
• 5+ years data processing with Pandas/Polars/PySpark/DuckDB.
• 2+ years Big Data tech (Spark, Snowflake).
• Expert-level orchestration with Airflow.
• Strong understanding of Medallion Architecture and SQL/NoSQL.
• Proven API data ingestion experience.
• Cloud proficiency: AWS/GCP and SQL optimization.
• Excellent cross-functional collaboration and communication.
• English proficiency at an upper-intermediate level.

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Lakehouses.
• Create ETL/ELT processes at scale using Python and Airflow.
• Develop ingestion workflows from diverse 3rd party APIs.
• Manage and optimize Parquet/Avro/ORC formats for performance.
• Collaborate with AI teams to support analytics and ML initiatives.
• Translate requirements into technical roadmaps with stakeholders.
• Work with Terraform to provision AWS/on-prem infrastructure.

Skills

Python

Pandas

Polars

PySpark

Airflow

SQL

Communication

Team collaboration

Self-starter

Education

Bachelor’s degree in CS/Engineering or equivalent

Tools

Snowflake

Spark

AWS

Terraform

GCP

OpenSearch

Job description

AgileEngine is an Inc. 5000 company delivering award-winning software for Fortune 500 brands and startups across 17+ industries. We foster a people-first culture with Best Place to Work awards.

We are seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses for a thematic research platform handling large financial datasets daily. You will build Python ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS.