Senior Data Engineer

AgileEngine, LLC; United States; On-site; USD 140,000 - 200,000; Full time

Benefits offered by this job

100% remote work with flexible hours

Job summary

AgileEngine, LLC is seeking a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures for a financial data platform. You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

Role combines hands-on engineering with technical consulting, translating business goals into data architecture roadmaps and collaborating across

Qualifications

• 5+ years of experience with Python and hands-on Python development.
• 5+ years of experience with data processing, manipulation, and analytics libraries such as Pandas, Polars, PySpark or DuckDB.
• 2+ years of experience with Big Data technologies (Spark, Snowflake).
• Expert-level knowledge of pipeline orchestration using Airflow or similar tools.
• Strong understanding of Medallion Architecture and columnar formats (Parquet, ORC).
• Proven ability to integrate with 3rd party APIs and diverse data sources.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT processes at scale using Python and Airflow.
• Create ingestion workflows from diverse 3rd party APIs and data sources.
• Optimize file formats and storage for high-performance data retrieval.
• Collaborate with stakeholders to translate business goals into data roadmaps.
• Contribute to AWS and on-prem infrastructure using Terraform and related tools.

Skills

Python

Data engineering

Airflow

SQL

Cloud platforms

Big data

Education

Bachelor's degree in CS/Engineering

Tools

Airflow

Snowflake

Spark

Pandas

Polars

PySpark

DuckDB

Job description

We are looking for a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures supporting a thematic research platform that processes large volumes of financial data daily. You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. The role combines hands‑on engineering with technical consulting responsibilities, translating business goals into data architecture roadmaps.

What you will do

• Design and implement Python Data Engineering solutions;
• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses;
• Design and implement robust ETL/ELT processes at scale using Python, incorporating modern pipeline orchestration tools like Airflow;
• Develop sophisticated ingestion workflows from diverse 3rd party APIs and data sources;
• Manage and optimize various file formats (Parquet, Avro, ORC) and columnar storage to ensure high-performance data retrieval;
• Work with AI development tools to support and accelerate ongoing development, machine learning initiatives and advanced analytics;
• Act as a technical consultant for stakeholders and leadership to gather requirements, understand business goals, and translate them into technical roadmaps;
• Work with Terraform and other tools to build AWS and on-prem infrastructure.

Must haves

• You must be authorized to work for ANY employer in the US (e.g., Green card holders, TN visa holders, GC EAD, H4 EAD, U4U with EAD), as we are unable to sponsor or take over employment visa sponsorship at this time;
• Bachelor’s degree in computer science/engineering or other technical field, or equivalent experience;
• 5+ years of experience with Python (strong, hands‑on Python experience is a must);
• 5+ years of experience with data processing, manipulation, and analytics libraries like Pandas, Polars, PySpark or DuckDB;
• 2+ years of experience with Big Data technologies (Spark, Snowflake);
• Expert‑level knowledge of pipeline orchestration using Airflow or similar industry-standard tools;
• Deep understanding of Medallion Architecture, columnar file formats, and diverse database technologies (SQL, NoSQL, and Lakehouse architectures);
• Proven ability to work with 3rd party APIs for complex data ingestion tasks;
• Proficiency with modern Cloud platforms (AWS, GCP, Snowflake) and advanced SQL optimization;
• Exceptional soft skills with a proven ability to gather requirements from leadership and collaborate effectively across cross‑functional teams;
• Excellence in optimizing complex data pipelines and troubleshooting data latency or consistency issues in massive datasets;
• A self‑starter mindset, regularly investigating more efficient data architectures and AI development tools to improve pipeline performance;
• Taking pride in data integrity and the accuracy of the end‑to‑end pipelines and architectures you build;
• Strong communication skills for seamless global collaboration with stakeholders and distributed teams;

Nice to haves

• Familiarity with the fintech industry, understanding of financial data, regulatory requirements, and business processes specific to the domain;
• Documentation skills to document data pipelines, architecture designs, and best practices for knowledge sharing and future reference;
• AWS Sagemaker Studio, Jupyter for analyze data;
• Terraform;
• Scala.
• Growth without limits: build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
• Competitive compensation: get recognition that reflects your skills and impact, with regular performance and compensation reviews
• Flexibility: work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
• Meaningful, modern projects: build impactful products using modern technologies alongside global teams and leading brands
• Collaborative culture: join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
• Well-being & support: access local well-being programs and people‑focused support tailored to your location

We designed our hiring process to be fast, transparent, and convenient — so you always know what to expect and can move through the steps without unnecessary delays.

Have any questions?

What is the work format andschedule?

What level of English isrequired?

Does AgileEngine provide workequipment?

What opportunities for professional growth do youoffer?

What does a typical team look like, and what tools do youuse?