Senior Data Engineer: AWS Spark for Health Data

Source Meridian; Medellín; In loco; COP 110.000.000 - 170.000.000; Tempo pieno

Vantaggi offerti da questo lavoro

Permanent contract

Learning and continuous growth

Health and well-being benefits

Competitive salary based on experience

Descrizione del lavoro

Source Meridian is seeking a Senior Data Engineer to build and operate an AWS-native data platform processing healthcare claims data and tokenized identifiers in Medellín. You will design Spark-based pipelines on Parquet on S3, queried via Athena, with a focus on data quality and privacy.

You will collaborate with cross-functional teams, optimize Spark performance and cost, contribute to dbt models, and implement robust ETL/ELT processes in a healthcare environment.

Competenze

• 5+ years of professional experience in Data Engineering.
• Strong experience with Apache Spark (PySpark or Scala) including joins, intersections, partitioning, and performance tuning.
• Strong hands-on experience with AWS data stack (S3, Athena, Glue, Lake Formation concepts).
• Experience building and operating pipelines using Airflow (DAGs, scheduling, dependencies, backfills).
• Excellent SQL skills and solid data modeling fundamentals.

Mansioni

• Build and maintain Spark pipelines to process large-scale Parquet datasets on S3.
• Process and deliver healthcare claims datasets with accurate identity mapping and data integrity.
• Orchestrate data pipelines using Airflow and AWS-native tools.
• Develop reliable ETL/ELT processes with retries, idempotency, monitoring, and reprocessing.
• Optimize performance and cost across Spark jobs, S3 layout, and Athena queries.
• Contribute to dbt models (transformations, documentation, data quality).
• Collaborate with cross-functional stakeholders with a focus on privacy and secure data handling.

Conoscenze

Apache Spark

PySpark/Scala

SQL

Airflow

AWS Data Stack

Data Modeling

Client-facing

English (Advanced)

Strumenti

dbt

AWS Glue

Amazon Athena

Amazon S3

Airflow

Descrizione del lavoro

Source Meridian is seeking a Senior Data Engineer to build and operate an AWS-native data platform processing healthcare claims data and tokenized identifiers in Medellín. You will design Spark-based pipelines on Parquet on S3, queried via Athena, with a focus on data quality and privacy.

You will collaborate with cross-functional teams, optimize Spark performance and cost, contribute to dbt models, and implement robust ETL/ELT processes in a healthcare environment.