Senior Data Engineer: Data Lakes & Pipelines (Remote)

AgileEngine; Tampa (FL); Hybrid; USD 130,000 - 185,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses, processing large volumes of financial data daily. You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

The role combines hands-on engineering with technical consulting responsibilities, translating business goals into

Qualifications

• Bachelor’s degree in computer science/engineering or other technical field, or equivalent experience.
• 5+ years of experience with Python (strong, hands-on Python experience is a must).
• 5+ years of experience with data processing, manipulation, and analytics libraries like Pandas, Polars, PySpark or DuckDB.
• 2+ years of experience with Big Data technologies (Spark, Snowflake).
• Expert-level knowledge of pipeline orchestration using Airflow or similar industry-standard tools.
• Deep understanding of Medallion Architecture, columnar file formats, and diverse database technologies (SQL, NoSQL, and Lakehouse architectures).
• Proven ability to work with 3rd party APIs for complex data ingestion tasks.
• Proficiency with modern Cloud platforms (AWS, GCP, Snowflake) and advanced SQL optimization.
• Strong communication skills for collaboration across cross-functional teams.
• Upper-intermediate English level.
• Authorized to work for ANY employer in the US; no visa sponsorship available.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Design and implement robust ETL/ELT processes at scale using Python and Airflow.
• Develop ingestion workflows from diverse 3rd party APIs and data sources.
• Manage and optimize Parquet, Avro, ORC and columnar storage for high performance.
• Work with AI development tools to support ML and analytics initiatives.
• Gather requirements from leadership and translate into technical roadmaps.
• Work with Terraform and other tools to build AWS and on-prem infrastructure.

Skills

Python

Pandas

PySpark

DuckDB

Airflow

SQL

NoSQL

Snowflake

Spark

AWS

Medallion Architecture

API ingestion

GCP

OpenSearch

Elasticsearch

English (Upper-intermediate)

Education

Bachelor’s degree in computer science/engineering or equivalent experience

Tools

OpenSearch

Elasticsearch

Job description

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses, processing large volumes of financial data daily. You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

The role combines hands-on engineering with technical consulting responsibilities, translating business goals into