Senior Data Engineer: AI Data Pipelines (Remote)

AgileEngine; Medellín; Presencial; COP 120.000.000 - 180.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote-friendly with flexible hours

Meaningful, modern projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine seeks a Senior Data Engineer to own ingestion, pipelines, and the Single Source of Truth for new product modules. Build containerized data pipelines on AWS, integrate with AI APIs, and manage the data lifecycle from extraction through UAT with high autonomy.

You'll work with Bedrock LLM and internal SDKs, extend Python tooling, and collaborate with DevOps for CI/CD and IaC, ensuring robust data flows across services.

Formación

• 5+ years of heavy Data Engineering or Backend experience focused on data pipelines and infrastructure.
• Expert-level Python skills, building containerized microservices and workers (ECS/EKS).
• Deep experience with AWS data infrastructure: S3, Glue, Athena.
• Hands-on experience integrating AI APIs (AWS Bedrock, Claude, Gemini).
• Comfort with Mac/Linux terminal (Bash, Makefiles).
• Strong communication and interview skills for a rigorous technical interview.
• Pragmatism handling cross-team dependencies (DevOps access/provisioning).
• Proficiency with AI-assisted development tools and agentic coding workflows.
• Upper-intermediate English level.

Responsabilidades

• Construct core ingestion pipelines to pull high-volume metrics from YouTube Data API, YouTube Creator API, Google Trends, and other sources into AWS S3/Glue/Athena SSoT.
• Write and maintain containerized Python workers (RQ/Celery with Redis) on EKS for data processing.
• Work with Bedrock LLM layer and internal SDKs to ensure data flows into AI orchestration/scoring layer.
• Extend Python SDKs to cover new services like ContentCreatorInsightsService.
• Own User Acceptance Testing (UAT) for the data layer, ensuring quality without dedicated QA.
• Collaborate with DevOps for CI/CD (GitHub Actions, ArgoCD) and IaC (Terraform).

Conocimientos

Python

Data pipelines

AWS

CI/CD

Linux terminal

Communication skills

English (Upper-Intermediate)

ECS/EKS

Terraform

GitHub Actions

ArgoCD

Herramientas

AWS S3

AWS Glue

Amazon Athena

Bedrock API

Python SDKs

Descripción del empleo

AgileEngine seeks a Senior Data Engineer to own ingestion, pipelines, and the Single Source of Truth for new product modules. Build containerized data pipelines on AWS, integrate with AI APIs, and manage the data lifecycle from extraction through UAT with high autonomy.

You'll work with Bedrock LLM and internal SDKs, extend Python tooling, and collaborate with DevOps for CI/CD and IaC, ensuring robust data flows across services.