Remote Data Engineer - Spark, AWS EMR & AI Pipelines

Upwork; Bogotá ciudad; Presencial; COP 279.520.000 - 372.694.000; Jornada completa

Descripción de la vacante

Upwork is seeking an experienced Software/Data Engineer to design and deliver scalable data processing systems and AI-enabled workflows. This contract role sits at the intersection of software engineering and data engineering, with a strong focus on Spark, cloud-based distributed processing, production reliability, and data preparation for analytics and machine learning use cases.

You will design and develop scalable data processing solutions, build batch and distributed pipelines, and

Formación

• 4+ years of software engineering or data engineering experience.
• Strong experience with Spark and distributed data processing.
• Experience with Amazon EMR or similar cloud-based data processing platforms.
• Proficiency in Java, Python, or a related programming language.
• Exposure to AI or machine learning workflows, model integration, or data preparation for intelligent systems.
• Strong understanding of scalable data architecture and performance optimization.
• Strong debugging and collaboration skills.
• Comfortable delivering in evolving, data-intensive environments.
• Ability to bridge software engineering and data engineering responsibilities.
• Strong execution focus with practical architecture judgment.

Responsabilidades

• Design and develop scalable data processing solutions using Spark and Amazon EMR or comparable cloud-based data processing platforms.
• Build and maintain batch and distributed data pipelines.
• Develop software components for data transformation, feature preparation, and AI or machine learning workflow integration.
• Collaborate with engineering, AI, and product teams to operationalize data-driven and model-enabled use cases.
• Optimize data pipeline performance, cost efficiency, scalability, and production reliability.
• Troubleshoot data and application issues across development and production environments.
• Contribute to architecture discussions, technical documentation, and engineering standards.
• Ensure solutions align with data quality, governance, and security expectations.

Conocimientos

Spark

Distributed processing

Amazon EMR

Java

Python

AI workflows

Data architecture

Debugging

Collaboration

Data security

Herramientas

Airflow

Kafka

AWS

Descripción del empleo

Upwork is seeking an experienced Software/Data Engineer to design and deliver scalable data processing systems and AI-enabled workflows. This contract role sits at the intersection of software engineering and data engineering, with a strong focus on Spark, cloud-based distributed processing, production reliability, and data preparation for analytics and machine learning use cases.

You will design and develop scalable data processing solutions, build batch and distributed pipelines, and