Senior Data Engineer — Remote & Flexible Hours

AgileEngine; Atlanta (GA); Hybrid; USD 140,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine, an Inc. 5000 company, seeks a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures supporting a research platform that processes large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

Qualifications

• Bachelor's degree or equivalent experience in a technical field.
• 5+ years of Python development and hands-on experience.
• 5+ years of data processing with Pandas, Polars, PySpark or DuckDB.
• 2+ years of Big Data tech (Spark, Snowflake).
• Expert-level workflow orchestration with Airflow or similar tool.
• Strong understanding of Medallion Architecture and data lakehouse concepts.
• Experience with APIs for data ingestion and cloud platforms (AWS/GCP).
• Excellent collaboration and communication across teams.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Create robust ETL/ELT pipelines with Airflow.
• Develop ingestion workflows from diverse APIs and data sources.
• Optimize file formats (Parquet/Avro/ORC) for performance.
• Collaborate with stakeholders to translate requirements into data roadmaps.
• Work with Terraform to manage AWS/on-prem infrastructure.

Skills

Python

Pandas

Polars

PySpark

DuckDB

Airflow

SQL

NoSQL

Cloud AWS

GCP

Snowflake

Spark

Medallion Architecture

Terraform

OpenSearch

Elasticsearch

Education

Bachelor's degree in computer science/engineering or related field

Tools

Airflow

Terraform

Snowflake

Job description

AgileEngine, an Inc. 5000 company, seeks a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures supporting a research platform that processes large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.