Remote LATAM Data Engineer: Spark, Kafka & Airflow

Redcloud Consulting, Inc.; Bogotá; On-site; COP 341,753,000 - 434,958,000; Full time

Job summary

RedCloud Consulting is seeking a Data Engineer to support client operations from LATAM, remote with working hours overlapping US PST and CST.

You will build and maintain scalable batch and streaming pipelines, optimize Spark and Scala workloads, monitor production data flows, and review AI/LLM-generated code for production readiness. You will collaborate with stakeholders on data modeling, quality controls, and observability to ensure reliable, scalable solutions.

Qualifications

• Strong Scala / Spark experience.
• Experience with batch data processing and Spark optimization.
• Streaming experience with Kafka, Flink, or Kafka Streams.
• Airflow workflow orchestration experience.
• Data modeling, schema design, and source-to-target mapping.
• Data quality, validation, and production troubleshooting.
• Strong software engineering practices: testing, deployment, versioning.
• Ability to review and validate AI/LLM-generated code.
• Flink experience preferred.
• NoSQL experience (Scylla/Cassandra/DynamoDB).
• Experience with SLAs, SLOs, SLIs, observability.
• Experience with AI-assisted tools (Claude, Copilot, Cursor).

Responsibilities

• Build and maintain scalable batch and streaming pipelines.
• Optimize Spark and Scala workloads for performance and cost.
• Monitor production pipelines and troubleshoot data quality defects.
• Collaborate with stakeholders to define requirements and data mappings.
• Orchestrate workflows using Airflow or similar tools.
• Process real-time data with Kafka, Flink, or Kafka Streams.
• Support data modeling, schema design, and quality controls.
• Apply strong engineering practices across testing, deployment, and environments.
• Review AI/LLM-generated code for production readiness.
• Support observability, SLAs, SLOs, and platform reliability.

Skills

Scala / Spark

Batch processing

Streaming (Kafka / Flink)

Airflow

Data modeling

Data quality

Software fundamentals

AI code review

Spec-led dev tools

NoSQL databases

Clickstream data

Observability / SLAs

AI tooling (Claude / Copilot)

Tools

Airflow

Kafka

Flink

Spark

Scala

GitHub Copilot

Claude

Cursor

Scylla/Cassandra/DynamoDB

Job description

RedCloud Consulting is seeking a Data Engineer to support client operations from LATAM, remote with working hours overlapping US PST and CST.

You will build and maintain scalable batch and streaming pipelines, optimize Spark and Scala workloads, monitor production data flows, and review AI/LLM-generated code for production readiness. You will collaborate with stakeholders on data modeling, quality controls, and observability to ensure reliable, scalable solutions.