Senior Data Engineer - Remote, Flexible Hours, Data Lakes

AgileEngine; Dallas (TX); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine seeks a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a high-volume financial data platform. You will implement Python ETL/ELT pipelines, orchestrate with Airflow, and ingest data from third-party APIs, leveraging Snowflake, Spark, and AWS to deliver robust data infrastructure.

You will translate business goals into data roadmaps, work across teams, and help ensure data integrity and performance for advanced analytics and AI initiatives.

Qualifications

• 5+ years of experience with Python and hands-on development.
• 5+ years of experience with data processing, manipulation, and analytics libraries.
• 2+ years with Big Data technologies (Spark, Snowflake).
• Expert-level knowledge of pipeline orchestration using Airflow or similar tools.
• Deep understanding of Medallion Architecture, columnar formats, and diverse DB technologies.
• Proven ability to work with 3rd party APIs for data ingestion tasks.
• Proficiency with AWS, GCP, Snowflake and advanced SQL optimization.
• Strong cross-functional collaboration and requirements gathering.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT processes at scale using Python and Airflow.
• Create ingestion workflows from diverse 3rd party APIs.
• Optimize file formats (Parquet/Avro/ORC) for high-performance retrieval.
• Collaborate with AI/ML teams to accelerate analytics initiatives.
• Translate business goals into data-roadmap requirements and timelines.
• Build AWS on-prem infrastructure with Terraform and related tools.

Skills

Python

Communication

English proficiency

API requirements gathering

Leadership

Education

Bachelor's degree in CS/Engineering or related field

Tools

Pandas

Polars

PySpark

DuckDB

Spark

Snowflake

Airflow

Parquet

Avro

ORC

AWS

GCP

Job description

AgileEngine seeks a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses supporting a high-volume financial data platform. You will implement Python ETL/ELT pipelines, orchestrate with Airflow, and ingest data from third-party APIs, leveraging Snowflake, Spark, and AWS to deliver robust data infrastructure.

You will translate business goals into data roadmaps, work across teams, and help ensure data integrity and performance for advanced analytics and AI initiatives.