Senior Data Engineer: Scale Data Lakes & AI Pipelines

AgileEngine, LLC.; Jersey City (NJ); Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine, LLC. in the United States is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses for a high‑volume financial research platform.

You will implement Python ETL/ELT pipelines and orchestrate workflows with Airflow, leveraging Snowflake, Spark, and AWS to deliver high‑performance data infrastructure. You will combine hands‑on engineering with technical consulting to translate business goals into data architecture roadmaps and collaborate with

Qualifications

• 5+ years of Python development and data processing.
• Strong experience with data processing libraries (Pandas, Polars, PySpark, DuckDB).
• 2+ years with Spark and Snowflake.
• Expert-level Airflow for orchestration.
• Experience with AWS/GCP and SQL optimization.
• Ability to translate business goals into data roadmaps.
• Authorized to work in the US without sponsorship.
• Excellent collaboration with cross-functional teams.

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT pipelines with Airflow.
• Create ingestion workflows from diverse 3rd party APIs.
• Manage file formats (Parquet, Avro, ORC) for performance.
• Collaborate with AI teams to support ML initiatives.
• Gather requirements and translate them into technical roadmaps.
• Use Terraform to provision AWS and on‑prem infrastructure.

Skills

Python

Pandas

Polars

PySpark

DuckDB

Spark

Snowflake

Airflow

SQL

NoSQL

Lakehouse

AWS

GCP

Medallion Architecture

API data ingestion

Terraform

ETL/ELT design

Data architecture

Communication

Education

Bachelor’s degree in computer science/engineering or related field

Tools

Terraform

Snowflake

AWS

GCP

Spark

Parquet/ORC

Job description

AgileEngine, LLC. in the United States is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses for a high‑volume financial research platform.

You will implement Python ETL/ELT pipelines and orchestrate workflows with Airflow, leveraging Snowflake, Spark, and AWS to deliver high‑performance data infrastructure. You will combine hands‑on engineering with technical consulting to translate business goals into data architecture roadmaps and collaborate with