Senior Data Engineer - Scalable Data Lakes & AI/ML (Remote/Flex)

AgileEngine; Jersey City (NJ); Hybrid; USD 140,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a thematic research platform handling large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and ingest from third-party APIs while leveraging Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role blends hands-on engineering with technical consulting.

Qualifications

• Must have 5+ years of Python experience.
• 5+ years of data processing with Pandas, Polars, PySpark or DuckDB.
• 2+ years with Big Data tech (Spark, Snowflake).
• Expert-level with pipeline orchestration using Airflow.
• Deep understanding of Medallion Architecture and columnar formats.
• Proven ability to ingest data from 3rd party APIs.

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Lakehouses.
• Develop ETL/ELT processes using Python and Airflow.
• Create ingestion workflows from diverse APIs and data sources.
• Optimize file formats and storage for high performance.
• Collaborate with stakeholders to translate requirements into roadmaps.
• Work with Terraform to build AWS and on-prem infrastructure.

Skills

Python

Airflow

Communication

English (upper-intermediate)

Education

Bachelor’s degree in computer science/engineering or related field

Tools

Spark

Snowflake

AWS

Pandas/Polars/PySpark/DuckDB

Medallion Architecture

Job description

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a thematic research platform handling large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and ingest from third-party APIs while leveraging Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role blends hands-on engineering with technical consulting.