Senior Data Engineer - Real-Time Big Data, Remote

Npv; United States; Remoto; USD 90.000 - 150.000; Tempo pieno

Vantaggi offerti da questo lavoro

Remote work

Flexible schedule

Descrizione del lavoro

PulsePoint is seeking a Data Engineer to join a fast-growing health/adtech technology company. The role processes 20TB+ daily data with near-real-time speed using Python and Spark, supporting reports, budget updates, and optimization engines against tight SLAs.

We operate remotely with a high engineering bar and ownership culture. Expect to work on Kafka, Hadoop, Presto, Hive, Spark Streaming, and cloud migrations across a distributed production system.

Competenze

• BA/BS degree in Computer Science or a related field.

Mansioni

• Install, maintain, and monitor Kafka, Hadoop, Presto, and RDBMS systems.
• Ingest, validate, and process internal and third-party data.
• Create, maintain, and monitor data flows in Hive, SQL, and Presto for consistency, accuracy, and lag time.
• Maintain and enhance the framework for jobs (primarily aggregate jobs in Hive).
• Build Kafka consumers using Spark Streaming for near-real-time aggregation.
• Train developers and analysts on data tools.
• Evaluate, select, and implement new tools.
• Handle backups, retention, high availability, and capacity planning.
• Review and approve DDL for databases, Hive framework jobs, and Spark Streaming to ensure standards are met.
• Participate in 24x7 on-call rotation for production support.
• Stack Airflow, Docker, Graphite/Beacon, Hive, Impala, Kafka, Kubernetes, Presto, Spark Streaming, SQL Server, Sqoop.

Conoscenze

Python

Spark

Spark Streaming

SQL

Linux

Kafka

Hadoop

Docker

Kubernetes

Airflow

Scala/Java

Formazione

BA/BS in Computer Science or related field

Strumenti

Hive

Presto

RDBMS

SQL Server

Sqoop

Git

Descrizione del lavoro

PulsePoint is seeking a Data Engineer to join a fast-growing health/adtech technology company. The role processes 20TB+ daily data with near-real-time speed using Python and Spark, supporting reports, budget updates, and optimization engines against tight SLAs.

We operate remotely with a high engineering bar and ownership culture. Expect to work on Kafka, Hadoop, Presto, Hive, Spark Streaming, and cloud migrations across a distributed production system.