Senior Data Engineer — Flexible, High-Impact Data Lakes

AgileEngine, LLC.; Austin (TX); Hybrid; USD 140,000 - 180,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine, LLC. is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouse architectures for a high-volume financial data platform.

You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and integrate with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role combines hands-on engineering with technical consulting, translating business goals into data architecture roadmaps while collaborating across

Qualifications

• Bachelor’s degree in computer science/engineering or equivalent experience.
• 5+ years of Python experience with hands-on coding.
• 5+ years of data processing, analytics libraries like Pandas, Polars, PySpark or DuckDB.
• 2+ years with Big Data tech (Spark, Snowflake).
• Expert-level pipeline orchestration with Airflow or similar tools.
• Deep understanding of Medallion Architecture and various DBs (SQL/NoSQL/Lakehouse).
• Experience with 3rd party APIs for data ingestion.
• Cloud proficiency (AWS, GCP, Snowflake) and SQL optimization.
• Strong cross-functional collaboration and requirements gathering.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT processes at scale using Python and Airflow.
• Create ingestion workflows from diverse 3rd party APIs and data sources.
• Optimize file formats (Parquet/Avro/ORC) and columnar storage for performance.
• Collaborate with AI/ML teams and translate business goals into roadmaps.
• Work with Terraform and other tools to build AWS/on-prem infrastructure.

Skills

Python

Pandas

Polars

PySpark

DuckDB

Spark

Airflow

SQL

NoSQL

Snowflake

AWS

GCP

Data Lakes

Data Warehouses

Lakehouse

Medallion Architecture

APIs

Terraform

Education

Bachelor’s degree in computer science/engineering

Tools

Terraform

Snowflake

AWS

GCP

Spark

Airflow

Job description

AgileEngine, LLC. is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouse architectures for a high-volume financial data platform.

You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and integrate with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role combines hands-on engineering with technical consulting, translating business goals into data architecture roadmaps while collaborating across