Senior Data Platform Engineer — Remote & Flexible

AgileEngine; Richmond (VT); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive USD-based pay

Exciting projects

Flextime

Job summary

AgileEngine, a leading software developer for Fortune 500 brands and startups, seeks a Senior Data Engineer. You will design scalable data lakes, warehouses, and lakehouse architectures to support a thematic research platform processing large financial data daily, building Python ETL/ELT pipelines and Airflow workflows.

You will ingest data from third-party APIs, work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure, and partner with stakeholders to translate goals

Qualifications

• Authorized to work for any employer in the US.
• Bachelor's degree in computer science/engineering or equivalent experience.
• 5+ years of Python hands-on experience.
• 5+ years of data processing, manipulation, and analytics with libraries like Pandas, Polars, PySpark or DuckDB.
• 2+ years with Big Data technologies (Spark, Snowflake).
• Expert-level knowledge of Airflow or similar tools.
• Deep understanding of Medallion Architecture and columnar data formats; experience with SQL/NoSQL and lakehouse architectures.
• Proven ability to ingest data from 3rd party APIs and work with cloud platforms (AWS, GCP, Snowflake).
• Strong communication and collaboration skills; upper-intermediate English.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Implement robust ETL/ELT processes at scale using Python and Airflow.
• Develop ingestion workflows from diverse 3rd party APIs and data sources.
• Manage and optimize Parquet/Avro/ORC formats and columnar storage for fast data retrieval.
• Collaborate with AI/ML teams to accelerate analytics initiatives.
• Translate business needs into technical roadmaps and architectures.
• Work with Terraform to provision AWS/on-prem infrastructure.

Skills

Python

Pandas

Polars

PySpark

DuckDB

Airflow

SQL

NoSQL

Medallion Architecture

Data Lakes

Data Warehouses

ETL/ELT

OpenAI/AI tooling

Education

Bachelor's degree in CS/Engineering

Tools

Snowflake

Spark

AWS

GCP

Job description

AgileEngine, a leading software developer for Fortune 500 brands and startups, seeks a Senior Data Engineer. You will design scalable data lakes, warehouses, and lakehouse architectures to support a thematic research platform processing large financial data daily, building Python ETL/ELT pipelines and Airflow workflows.

You will ingest data from third-party APIs, work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure, and partner with stakeholders to translate goals