Senior Data Engineer: AI Pipelines on AWS (Remote)

AgileEngine, LLC.; Sur; Presencial; COP 373.750.000 - 560.625.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth budget

Competitive compensation

Remote work

Modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to own ingestion, pipeline orchestration, and the Single Source of Truth architecture for new product modules. You will build containerized data pipelines on AWS, integrate directly with AI APIs, and manage the data lifecycle from extraction through user acceptance testing.

This role requires strong autonomy as it operates without dedicated QA support during the initial build phase, and you will collaborate with DevOps for CI/CD and IaC.

Formación

• 5+ years of heavy Data Engineering or Backend experience focused on data pipelines and infrastructure.
• Expert-level Python skills, building containerized microservices and workers (ECS/EKS).
• Deep experience with AWS data infrastructure: S3, AWS Glue, Amazon Athena.
• Hands-on experience integrating AI APIs (AWS Bedrock, Claude, Gemini) and external data firehoses.
• Mac/Linux terminal experience (Bash, Makefiles).
• Strong communication and interview skills; able to defend technical choices under pressure.
• Extreme pragmatism and patience with cross-team dependencies (DevOps provisioning).
• Experience with AI-assisted development tools and agentic coding workflows.

Responsabilidades

• Construct core ingestion pipelines to pull high-volume metrics from multiple data sources into AWS S3/Glue/Athena SSoT.
• Write and maintain containerized Python workers (RQ/Celery with Redis) on EKS for data processing.
• Work with Bedrock LLM layer and internal SDKs to ensure data flows into the AI orchestration layer.
• Extend Python SDKs to cover new services like ContentCreatorInsightsService.
• Own User Acceptance Testing (UAT) for the data layer without dedicated QA.
• Collaborate with DevOps/platform teams for CI/CD (GitHub Actions, ArgoCD) and IaC (Terraform).

Conocimientos

Data engineering

Python

AWS data infrastructure

CI/CD

Herramientas

ECS

EKS

Redis

Terraform

GitHub Actions

ArgoCD

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to own ingestion, pipeline orchestration, and the Single Source of Truth architecture for new product modules. You will build containerized data pipelines on AWS, integrate directly with AI APIs, and manage the data lifecycle from extraction through user acceptance testing.

This role requires strong autonomy as it operates without dedicated QA support during the initial build phase, and you will collaborate with DevOps for CI/CD and IaC.