Senior Data Engineer — Build Data Lakes & AI Pipelines (Remote)

AgileEngine; Boston (MA); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation (USD-based)

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouses supporting a thematic research platform that processes large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role blends hands-on engineering with technical consulting.

Qualifications

• Bachelor’s degree in computer science, engineering or equivalent experience.
• 5+ years of Python experience and hands-on Python development.
• 5+ years of data processing using Pandas/Polars/PySpark/DuckDB.
• 2+ years with Big Data tech (Spark, Snowflake).
• Expert-level workflow orchestration with Airflow or equivalent.
• Deep understanding of Medallion Architecture and columnar formats; SQL/NoSQL and lakehouse tech.
• Experience integrating 3rd party APIs for data ingestion.
• Proficiency with cloud platforms (AWS, GCP, Snowflake) and SQL optimization.
• Strong communication and collaboration across cross-functional teams.
• Self-starter mindset, commitment to data integrity and pipeline performance.
• Upper-intermediate English required.

Responsibilities

• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Implement Python-based ETL/ELT pipelines and Airflow workflows.
• Develop ingestion workflows from diverse third‑party APIs and data sources.
• Optimize storage formats (Parquet/Avro/ORC) for performance.
• Collaborate with stakeholders to translate business goals into data roadmaps.
• Work with Terraform to provision AWS and on-prem infrastructure.
• Support AI/ML initiatives and analytics through robust data platforms.

Skills

Python programming

Data processing

Soft skills

English proficiency

Education

Bachelor’s degree in CS/Engineering

Tools

Pandas

Polars

PySpark

DuckDB

Airflow

Snowflake

Spark

AWS

GCP

Terraform

Job description

AgileEngine is seeking a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouses supporting a thematic research platform that processes large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role blends hands-on engineering with technical consulting.