Senior Big Data Engineer - AWS & Databricks Data Pipelines

SoftServe Inc.; Colombia; On-site; COP 90,000,000 - 180,000,000; Full time

Job summary

SoftServe is seeking a Senior Data Engineer to design and build scalable data pipelines in AWS. You will own end-to-end data processing, from ingestion to analytics, partnering with AI and product teams to enable analytics and AI-driven use cases.

You'll work with Python, SQL, Apache Spark, Databricks, and AWS services like Glue, Athena, and Aurora, plus PostgreSQL/MySQL. Strong CI/CD and IaC experience with Terraform is a plus, as is GenAI/MLOps exposure.

Qualifications

• 5+ years of professional experience in Big Data or Data Engineering
• Strong Python and SQL proficiency
• Hands-on with Apache Spark, Databricks, and AWS data services
• Experience with relational databases (PostgreSQL, MySQL) and CI/CD
• Knowledge of IaC (Terraform) and data engineering best practices

Responsibilities

• Design, develop, and maintain scalable batch and distributed data processing pipelines
• Build and optimize data solutions focusing on performance, reliability, scalability, and cost
• Collaborate with engineering, AI, and product teams to support analytics and AI use cases
• Develop data solutions using Python, SQL, Spark, Databricks and AWS Glue/Athena/Aurora
• Work with PostgreSQL and MySQL to support data workflows
• Develop and maintain workflow orchestration with Airflow and contribute to CI/CD and IaC efforts using Terraform

Skills

Big Data

Data Engineering

Python

SQL

English

Apache Spark

Databricks

Airflow

Kafka

GenAI

MLOps

Java

Tools

Apache Spark

Databricks

AWS Glue

Amazon Athena

Amazon Aurora

PostgreSQL

MySQL

Apache Airflow

Apache Kafka

Terraform

Job description

SoftServe is seeking a Senior Data Engineer to design and build scalable data pipelines in AWS. You will own end-to-end data processing, from ingestion to analytics, partnering with AI and product teams to enable analytics and AI-driven use cases.

You'll work with Python, SQL, Apache Spark, Databricks, and AWS services like Glue, Athena, and Aurora, plus PostgreSQL/MySQL. Strong CI/CD and IaC experience with Terraform is a plus, as is GenAI/MLOps exposure.