Remote Big Data Engineer - Threat Intelligence Pipelines

Lumen; Richmond (VA); Remote; USD 145,000 - 194,000; Full time

Job summary

Lumen is seeking a Big Data Developer to design and optimize large-scale data solutions that help detect and disrupt malicious threats across the Internet. You will work with Spark, Kafka, Hadoop, Hive, and other technologies to build scalable data pipelines and real-time validation systems.

You will collaborate with cross‑functional teams, mentor junior engineers, and explore PoC opportunities to adopt new technologies, advancing AI‑ready connectivity and cybersecurity capabilities in a remote

Qualifications

• Master’s or Bachelor’s degree in Computer Science, Engineering, Mathematics, or related field, or equivalent work experience.
• Experience in developing data ingestion, processing and analytical pipelines with big data technologies (Spark, Kafka, Hadoop, Hive).
• Proficient in programming languages such as Python, Java, Scala, or SQL.
• Experience with DevOps practices and tools.
• Strong analytical, problem‑solving, and attention to detail.
• Strong communication, leadership, and teamwork skills.
• Eagerness to learn new skills and technologies.

Responsibilities

• Lead the design, development, and implementation of big data solutions to identify and disrupt malicious threats on the Internet.
• Architect and build scalable data pipelines using Spark, Kafka, Hadoop, Hive, and related technologies.
• Collaborate with cross‑functional teams to understand data requirements and deliver data solutions.
• Extract, transform, and load data from relational databases, APIs, JSON, XML, CSV, and more.
• Ensure data quality, integrity, and security across all data processes.
• Develop scalable systems enabling real‑time validation of identified resources.
• Mentor junior developers and foster continuous learning.
• Evaluate PoC opportunities to adopt new technologies improving threat detection and data processing.

Skills

Analytical skills

Communication

Leadership

Teamwork

Willingness to learn

Problem solving

Education

Master’s or Bachelor’s degree in CS/Engineering/Math or related field

Tools

Spark

Kafka

Hadoop

Hive

Python

Java

Scala

SQL

DevOps practices & tools

Job description

Lumen is seeking a Big Data Developer to design and optimize large-scale data solutions that help detect and disrupt malicious threats across the Internet. You will work with Spark, Kafka, Hadoop, Hive, and other technologies to build scalable data pipelines and real-time validation systems.

You will collaborate with cross‑functional teams, mentor junior engineers, and explore PoC opportunities to adopt new technologies, advancing AI‑ready connectivity and cybersecurity capabilities in a remote