Data Engineer

RedCloud Consulting; Bogotá ciudad; On-site; COP 279,520,000 - 403,752,000; Full time

Job summary

RedCloud seeks a Data Engineer to support immediate client operations. Location: LATAM, Remote with working hours overlap to US PST/CST. The role focuses on large-scale batch and streaming systems, Spark optimization, and production data pipelines, with review of AI/LLM-generated code.

As part of the client’s customer data team, you will work with Airflow, Kafka, and Scala to ensure reliable data solutions, quality controls, and observability across deployments.

Qualifications

• Strong Scala / Spark experience and batch processing expertise.
• Experience with streaming technologies: Kafka, Flink, or Kafka Streams.
• Proficiency with Airflow for workflow orchestration.
• Data modeling, schema design, and source-to-target mapping.
• Data quality, validation, and production troubleshooting.
• Ability to review and validate AI/LLM-generated code for production readiness.
• Familiarity with SLAs, SLOs, SLIs and observability concepts.
• Experience with NoSQL databases (Scylla, Cassandra, DynamoDB).
• Exposure to clickstream or transactional data environments.

Responsibilities

• Build and maintain scalable batch and streaming data pipelines.
• Optimize Spark and Scala workloads for performance and cost.
• Monitor production pipelines, troubleshoot issues, and fix data quality defects.
• Collaborate with stakeholders to define data requirements and mappings.
• Orchestrate workflows using Airflow or similar tools.
• Process real-time data using Kafka or Flink technologies.
• Support data modeling, validation, and quality controls.
• Review AI/LLM-generated code for production readiness.
• Support observability, SLAs, and platform reliability.

Skills

Scala / Spark experience

Batch data processing

Streaming with Kafka / Flink

Airflow workflow orchestration

Data modeling & schema design

Data quality & production troubleshoot

Review AI / LLM-generated code

NoSQL (Scylla/Cassandra/DynamoDB)

Observability & SLAs/SLOs

AI-assisted engineering tools

Flink experience

Clickstream / transactional data

Spec-led development tools

Tools

Airflow

Kafka

Flink

Spark

Scala

Job description

RedCloud seeks a Data Engineer to support immediate client operations.

Location: Latin America (LATAM), Remote (must have working hours overlap with US PST and CST time zones).

Overview

The Data Engineer will play a key role on our client’s customer data team as someone who can work across large-scale batch and streaming systems, optimize Spark workloads, support production data pipelines, and review AI/LLM-generated code with good engineering judgment.

Responsibilities Include But Are Not Limited To

• Build and maintain scalable batch and streaming data pipelines.
• Optimize Spark and Scala workloads for performance, reliability, and cost efficiency.
• Monitor production pipelines, troubleshoot issues, and resolve data quality defects.
• Partner with stakeholders to define requirements, mappings, and dependable data solutions.
• Orchestrate workflows using Airflow or similar tools.
• Process real-time data using Kafka, Flink, Kafka Streams, or similar technologies.
• Support data modeling, schema design, validation, and quality controls.
• Apply strong engineering practices across testing, versioning, deployment, and environments.
• Review and validate AI/LLM-generated code for production readiness.
• Support observability, SLAs, SLOs, and ongoing platform reliability.

Required knowledge, skills, and abilities

• Strong Scala / Spark experience.
• Batch data processing and Spark job optimization.
• Streaming experience with Kafka Streams, Flink, Kafka, or similar.
• Airflow workflow orchestration.
• Data modeling, schema design, and source-to-target mapping.
• Data quality, validation, and production troubleshooting.
• Strong software engineering fundamentals: testing, deployment discipline, versioning, and environment management.
• Ability to review and validate AI/LLM-generated code.
• Awareness of frameworks and tools like spec-led development, spec-kit, agent-skills etc. preferred.
• Flink experience preferred.
• Large-scale NoSQL experience, especially Scylla, Cassandra, DynamoDB, or similar preferred.
• Clickstream, customer identity, loyalty, booking, commerce, or transactional data experience preferred.
• Experience with SLAs, SLOs, SLIs, data quality thresholds, or observability preferred.
• Experience with Claude, GitHub Copilot, Cursor, or other AI-assisted engineering tools preferred.

Benefits and bonus information can be found at https://www.redcloudconsulting.com/careers

RedCloud requires employees have the legal ability to work in the country hired during their employment period. During onboarding, proof of eligibility to work in the applicable country will be requested. RedCloud does not provide visa sponsorship.

About Us

RedCloud Consulting is a global business and technology consulting firm supporting clients across North America, Europe, Asia-Pacific, and the Middle East. For more than two decades, RedCloud has partnered with organizations ranging from high-growth mid-market companies to global enterprises, helping them solve complex business and operational challenges—leveraging modern data platforms, artificial intelligence, and secure digital foundations to drive smarter decisions and scalable outcomes.

Anchored in a foundation of integrity-based consulting, RedCloud brings together senior-level experts who work side by side with client teams to design and implement durable, high-impact solutions. Our approach emphasizes trust, accountability, and long-term value—balancing stability, growth, and innovation to help organizations operate securely and confidently in an increasingly complex global environment.

RedCloud delivers a comprehensive range of services across its core practices: Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. As our clients expand globally, so do we—providing consistent, expert support across regions while remaining agile, collaborative, and deeply invested in client success.

Recognized for both performance and culture, RedCloud has been named #1 Best Company to Work For in Washington (Mid-Sized Businesses) by Seattle Business Magazine, a “Fastest Growing Company” multiple times on the Inc. 500/5000 list, and recognized by Consulting Magazine as a Leader in Technology Consulting.