Security Data Platform Engineer (Hybrid, GCP)

Rivago infotech inc; Charlotte (NC); Hybrid; USD 120,000 - 190,000; Full time

Job summary

Rivago infotech inc is seeking an experienced Data / Data Platform Engineer to design, develop, and optimize high-throughput pipelines (batch and streaming) for vulnerability findings. You will build an extensible framework to onboard new scanning models and assist with migrating on-prem pipelines to Google Cloud Platform.

Responsibilities include designing robust pipelines with PySpark, Hadoop, Hive, and Kafka, implementing data quality and deduplication, and delivering clean data to downstream

Qualifications

• Proficiency in Python and Go (Golang).
• Hands-on experience with PySpark, Hadoop, Hive, and Kafka for batch and streaming pipelines.
• Experience with Google Cloud Platform (BigQuery, Data Lake) and cloud data architectures.

Responsibilities

• Design and implement batch and real-time streaming pipelines using PySpark, Hadoop, Hive, and Kafka.
• Implement data quality, enrichment, and deduplication across multiple scanning feeds.
• Ingest and process data in Google Cloud Platform Data Lake and BigQuery environments.

Skills

Python

Go (Golang)

PySpark

Hadoop

Hive

Kafka

BigQuery

GCP Data Lake

REST APIs

Messaging systems

ITSM platforms

Tools

REST APIs

Messaging systems

ITSM platforms

Job description

Rivago infotech inc is seeking an experienced Data / Data Platform Engineer to design, develop, and optimize high-throughput pipelines (batch and streaming) for vulnerability findings. You will build an extensible framework to onboard new scanning models and assist with migrating on-prem pipelines to Google Cloud Platform.

Responsibilities include designing robust pipelines with PySpark, Hadoop, Hive, and Kafka, implementing data quality and deduplication, and delivering clean data to downstream