Data engineer

Insight Global; United States; Remote; USD 101,000 - 134,000; Full time

Job summary

Insight Global ищет Data Engineer для R&D AI организации глобальной фармацевтической компании. Ваша работа будет заключаться в создании и расширении внутренней AWS‑базированной платформы для обработки больших наборов данных и построения конвейеров, позволяющих ingest, processing и хранение сложных данных.

Команда небольшая и отвечает за масштабируемые пайплайны данных, взаимодействие с внутренними и внешними источниками.

Qualifications

• Опыт разработки дата-инженерии и конвейеров обработки данных.
• Глубокие знания Python и техник обработки данных.
• Опыт Databricks и data lake архитектур.
• Умение проектировать схемы и модели данных для табличных и нетабличных форматов.
• Опыт работы с AWS S3; знание Airflow и Iceberg; плюсом биоинформатика/секвенирование.

Responsibilities

• Разрабатывать и поддерживать масштабируемые конвейеры данных.
• Проектировать схемы, модели и архитектуру хранилища данных.
• Интегрировать источники данных и обеспечивать качество данных.

Skills

Python

Tools

Databricks

AWS S3

Apache Airflow

Apache Iceberg

Pandas

DuckDB

Job description

Описание:

Insight Global are seeking a Data Engineer to join the R&D AI organisation of a leading global pharmaceutical company. This position is fully remote in Europe for a contract through June 2027. The Data Engineer will join a small engineering team responsible for building and expanding an internal AWS-based platform for the storage and processing of large-scale, non-GxP omics datasets. You will help develop scalable data pipelines that enable scientific and technical teams to ingest, process, store and consume complex datasets from internal and external sources.

Задачи:

working with large structured and unstructured scientific datasets developing data processing pipelines capturing metadata designing appropriate data models supporting the continued development of the platform's data capabilities

Требования:

Strong professional experience in Data Engineering. Strong Python skills. Experience with Databricks and data lake architectures. Experience building and maintaining scalable data processing pipelines. Experience handling large structured and unstructured datasets and files. Experience designing schemas and data models for both tabular and non-tabular data. Experience with data processing tools such as Pandas, DuckDB or similar technologies. Strong AWS experience, particularly with technologies such as Amazon S3. Experience with Apache Airflow, Apache Iceberg or similar data engineering technologies. Будет плюсом: Educational or professional experience in biology, bioinformatics, biotechnology or another scientific discipline; Experience working with omics or sequencing data; Experience with Nextflow and/or Seqera; Experience with TileDB; Experience with WEKA storage; Understanding of data archiving strategies; Experience supporting scientific or bioinformatics users and workflows

Условия:

fully remote in Europe contract through June 2027