Senior Big Data Engineer - AWS Spark, 3-Month US Rotation

SoftServe; Chile; On-site; CLP 42,000,000 - 66,000,000; Full time

Job summary

SoftServe in Chile is seeking a senior Data Engineer to support a cloud-based data platform on AWS. You will develop and optimize data ingestion and processing pipelines using Python, PySpark, Spark, and AWS Glue, ensuring reliability and data quality across regulated healthcare data.

You will collaborate with a distributed engineering team, implement CI/CD, containerize workloads with Docker, and participate in data security and governance. A three-month US business trip is required in Q1 2027.

Qualifications

• 5+ years of Big Data or Data Engineering experience, including production data platforms.
• Strong proficiency in Python, PySpark, and Apache Spark for large-scale data processing.
• Hands-on experience with AWS data services and data pipelines.

Responsibilities

• Support, maintain, and enhance an AWS-based data platform and ingestion pipelines.
• Develop and optimize data processing workflows using Python, PySpark, Spark, and AWS Glue.
• Monitor and troubleshoot data pipelines for reliability, performance, and quality.
• Work with AWS services such as S3, Aurora PostgreSQL, Athena, Lambda, SQS, SNS, EventBridge.
• Maintain and improve workflow orchestration with Apache Airflow.
• Support deployment processes using Docker, Terraform, and CI/CD practices.
• Apply software engineering best practices including automated testing and code quality checks.
• Collaborate with remote teams to resolve issues and improve platform stability.
• Support data security, governance, and compliance in a regulated healthcare environment.

Skills

Python

PySpark

Apache Spark

AWS

Airflow

Docker

Terraform

CI/CD

GitHub Actions

SQL

Data quality

Testing

HIPAA/HITRUST

Tools

Docker

Terraform

GitHub Actions

Job description

SoftServe in Chile is seeking a senior Data Engineer to support a cloud-based data platform on AWS. You will develop and optimize data ingestion and processing pipelines using Python, PySpark, Spark, and AWS Glue, ensuring reliability and data quality across regulated healthcare data.

You will collaborate with a distributed engineering team, implement CI/CD, containerize workloads with Docker, and participate in data security and governance. A three-month US business trip is required in Q1 2027.