Senior Big Data Software Engineer (AWS + Databricks)

SoftServe Inc.; Colombia; In loco; COP 90.000.000 - 180.000.000; Tempo pieno

Descrizione del lavoro

SoftServe is seeking a Senior Data Engineer to design and build scalable data pipelines in AWS. You will own end-to-end data processing, from ingestion to analytics, partnering with AI and product teams to enable analytics and AI-driven use cases.

You'll work with Python, SQL, Apache Spark, Databricks, and AWS services like Glue, Athena, and Aurora, plus PostgreSQL/MySQL. Strong CI/CD and IaC experience with Terraform is a plus, as is GenAI/MLOps exposure.

Competenze

• 5+ years of professional experience in Big Data or Data Engineering
• Strong Python and SQL proficiency
• Hands-on with Apache Spark, Databricks, and AWS data services
• Experience with relational databases (PostgreSQL, MySQL) and CI/CD
• Knowledge of IaC (Terraform) and data engineering best practices

Mansioni

• Design, develop, and maintain scalable batch and distributed data processing pipelines
• Build and optimize data solutions focusing on performance, reliability, scalability, and cost
• Collaborate with engineering, AI, and product teams to support analytics and AI use cases
• Develop data solutions using Python, SQL, Spark, Databricks and AWS Glue/Athena/Aurora
• Work with PostgreSQL and MySQL to support data workflows
• Develop and maintain workflow orchestration with Airflow and contribute to CI/CD and IaC efforts using Terraform

Conoscenze

Big Data

Data Engineering

Python

SQL

English

Apache Spark

Databricks

Airflow

Kafka

GenAI

MLOps

Java

Strumenti

Apache Spark

Databricks

AWS Glue

Amazon Athena

Amazon Aurora

PostgreSQL

MySQL

Apache Airflow

Apache Kafka

Terraform

Descrizione del lavoro

About The Role

In this role you will design and develop scalable data processing solutions within AWS environments, working with large-scale and distributed data. You will take ownership of reliable data pipelines and contribute to data platform development, collaborating with engineering, AI, and product teams to support analytics, automation, and AI-enabled solutions

About The Role

In this role you will design and develop scalable data processing solutions within AWS environments, working with large-scale and distributed data. You will take ownership of reliable data pipelines and contribute to data platform development, collaborating with engineering, AI, and product teams to support analytics, automation, and AI-enabled solutions

Responsibilities

• Design, develop, and maintain scalable batch and distributed data processing pipelines to support reliable data processing and integration
• Build and optimize data processing solutions, taking ownership of performance, reliability, scalability, and cost efficiency across data pipelines
• Collaborate with engineering, AI, and product teams to understand requirements and support analytics, automation, and AI-enabled use cases
• Develop data solutions using Python, SQL, Apache Spark, Databricks, and AWS services such as AWS Glue, Amazon Athena, and Amazon Aurora
• Work with relational databases such as PostgreSQL and MySQL to support data processing and integration workflows
• Develop and maintain workflow orchestration using technologies such as Apache Airflow and contribute to reliable data processing workflows
• Contribute to CI/CD, Infrastructure as Code, testing, troubleshooting, and production support activities using Terraform or similar technologies
• Participate in architecture discussions and contribute to the continuous improvement of data platforms and engineering practices

Requirements

• Strong experience in Big Data, Data Engineering, or related fields, with 5+ years of professional experience
• Strong proficiency in Python and SQL, with Java experience considered a plus
• Strong hands-on experience with Apache Spark, distributed data processing, and Databricks
• Solid experience developing data solutions on AWS using services such as AWS Glue, Amazon Athena, and Amazon Aurora
• Practical experience working with relational databases such as PostgreSQL and MySQL
• Familiarity with workflow orchestration technologies such as Apache Airflow and streaming technologies such as Apache Kafka
• Solid experience with CI/CD, version control, and Infrastructure as Code using Terraform or similar technologies
• Familiarity with GenAI, MLOps, machine learning workflows, or AI platform integrations is considered a plus
• Upper-intermediate or higher English proficiency for collaboration across teams

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.