Remote Senior Data Engineer: AWS Data Pipelines & AI

AgileEngine, LLC.; Centrosur; Presencial; COP 373.750.000 - 560.625.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

Remote work with flexible hours

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine, LLC. is seeking a Senior Data Engineer to own ingestion, orchestration, and the Single Source of Truth for new product modules.

Build containerized data pipelines on AWS, integrate with AI APIs, and manage data lifecycle from extraction through UAT, operating autonomously during the initial build phase. You will develop core pipelines to ingest high-volume metrics from multiple APIs into AWS S3/Glue/Athena, write Python workers on EKS, and collaborate with DevOps and Bedrock layers

Formación

• 5+ years of heavy Data Engineering or Backend experience focused on data pipelines and infrastructure.
• Expert-level Python skills, specifically in building containerized microservices and workers (ECS/EKS).
• Deep experience with AWS data infrastructure, specifically S3, AWS Glue, and Amazon Athena.
• Proven hands-on experience integrating AI APIs, specifically AWS Bedrock, Claude, or Gemini, and handling external data firehoses.
• High comfort level working in a Mac/Linux terminal environment (Bash, Makefiles).
• Strong communication and interview skills; able to defend technical choices and pass a rigorous interview.
• Extreme pragmatism and patience with cross-team dependencies (DevOps access provisioning).
• Proficiency with AI-assisted development tools and agentic coding workflows to accelerate engineering productivity.

Responsabilidades

• Construct the core ingestion pipelines to pull high-volume metrics from YouTube Data API and other sources into AWS S3/Glue/Athena SSoT.
• Write and maintain containerized Python workers (RQ/Celery with Redis) on EKS for data processing.
• Collaborate with Bedrock LLM layer and internal SDKs to ensure data flows into AI orchestration/ scoring layer.
• Extend existing Python SDKs to cover new services like ContentCreatorInsightsService.
• Own UAT for the data layer, ensuring high quality and catching edge cases with no dedicated QA team.
• Work with DevOps/platform teams for CI/CD (GitHub Actions, ArgoCD) and IaC (Terraform) per internal standards.

Conocimientos

Python

Data pipelines

ECS/EKS

AWS data infra

S3

AWS Glue

Athena

Containerized microservices

CI/CD

Unix/Linux shell

Herramientas

Terraform

GitHub Actions

ArgoCD

Descripción del empleo

AgileEngine, LLC. is seeking a Senior Data Engineer to own ingestion, orchestration, and the Single Source of Truth for new product modules.

Build containerized data pipelines on AWS, integrate with AI APIs, and manage data lifecycle from extraction through UAT, operating autonomously during the initial build phase. You will develop core pipelines to ingest high-volume metrics from multiple APIs into AWS S3/Glue/Athena, write Python workers on EKS, and collaborate with DevOps and Bedrock layers