Remote Senior Data Platform Engineer — Real-Time Pipelines

Intellias; Colombia; Presencial; COP 120.000.000 - 190.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Indefinite contract with IntelliasCol"

Health insurance

Workation up to 90 day outsideColombia

100% work from home + internet subsidy

100% English environment

Descripción de la vacante

Intellias in Colombia seeks an senior data/ software engineer to design and build scalable data platforms and real‑time/batch pipelines. You will own end-to-end data processing, collaborate with cross‑functional teams, and ensure robust, production‑ready backend services and APIs.

Ideal candidates bring 8+ years in data engineering, deep Spark/PySpark expertise, and strong Python proficiency. This role offers an indefinite contract, full remote work, English-language environment, and generous

Formación

• 8+ years of professional software and data engineering experience.
• 3+ years working with large-scale distributed data systems.
• Designing and building scalable data platforms, pipelines, and distributed processing systems.
• Strong hands-on proficiency in Python, with PySpark, Pandas, and FastAPI.
• Experience with Apache Spark/PySpark and distributed processing at scale.
• Designing real-time and batch data pipelines with Kafka, Spark, or similar platforms.
• Understanding data architecture, modeling, schema evolution, data quality, and heterogeneous data integration.
• Experience with workflow orchestration tools such as Prefect, Apache Airflow.
• Building production backend services and APIs supporting data/platform capabilities.
• Hands-on experience with SQL and NoSQL (e.g., MongoDB).
• Understanding distributed systems: scalability, reliability, fault tolerance, performance.
• Designing observability, logging, monitoring, and alerting for distributed data systems.
• Cloud-native engineering: Docker, Kubernetes, CI/CD, production deployment.

Responsabilidades

• Design and implement solutions for large-scale, ambiguous data engineering and distributed-systems challenges.
• Architect, build, and evolve real-time and batch data infrastructure for operational, analytical, and AI/ML workloads.
• Design and scale event-driven data pipelines using Kafka, Spark/PySpark, and modern data-processing platforms.
• Build backend services and APIs using Python, FastAPI, and cloud-native technologies.
• Define scalable data ingestion, transformation, modeling, validation, storage, and delivery approaches.
• Establish and champion technical standards, architectural patterns, and reusable platform components.
• Serve as a cross-team technical authority for data systems and platform architecture.
• Improve reliability, scalability, performance, and cost efficiency across data infra.
• Implement robust observability, logging, monitoring, and alerting across systems.
• Design solutions with security, privacy, and compliance built in from the start.
• Improve CI/CD, automated testing, and production deployment practices.
• Collaborate with Product/ML/Platform/Infra to build shared data capabilities.
• Provide technical mentorship via architecture reviews and knowledge sharing.
• Evaluate new data/AI technologies and adopt where improvements are shown.
• Remain hands-on with production engineering and validation.

Conocimientos

Python

PySpark

Spark

Kafka

Airflow

FastAPI

MongoDB

Docker

Kubernetes

Data pipelines

Data modeling

Cloud

Herramientas

Docker

Kubernetes

GitHub Copilot

Datadog

Prometheus

OpenTelemetry

DuckDB

Descripción del empleo

Intellias in Colombia seeks an senior data/ software engineer to design and build scalable data platforms and real‑time/batch pipelines. You will own end-to-end data processing, collaborate with cross‑functional teams, and ensure robust, production‑ready backend services and APIs.

Ideal candidates bring 8+ years in data engineering, deep Spark/PySpark expertise, and strong Python proficiency. This role offers an indefinite contract, full remote work, English-language environment, and generous