Senior Data Pipeline Engineer - AI & Security Feeds

NewGen Technologies; Herndon (VA); Hybrid; USD 95,000 - 140,000; Full time

Job summary

NewGen Technologies is seeking a Software Developer/Systems Engineer to design and maintain enterprise data feeds powering partner networks and security platforms. You will integrate data from numerous vendor APIs, build robust ETL pipelines, and leverage AI for complex data extraction and normalization tasks.

Responsibilities include designing parsers for diverse formats, orchestrating automated data pipelines, ensuring reliability with monitoring and optimization, and adapting quickly to API

Qualifications

• 5+ years building production data pipelines with Python.
• Proven experience consuming REST or similar APIs at scale.
• Familiarity with tools such as Nmap, Rapid7, or similar tools.

Responsibilities

• Design and maintain enterprise data feeds from 10+ network and security vendor APIs.
• Build intelligent parsers that handle diverse data formats.
• Orchestrate automated pipelines for regular data download, transformation and loading.
• Ensure reliability and performance through monitoring, error handling, and optimization.
• Adapt to change by quickly resolving API updates, data format changes, and vendor issues.
• Integrate AI capabilities for data extraction, classification, anomaly detection, and auto-healing.
• Perform technical evaluations of new tools for potential use and positive impact on project processes.

Skills

Python

REST APIs

AI integration

ETL pipelines

Data normalization

Education

TS/SCI Clearance

Tools

Nmap

Rapid7

Job description

NewGen Technologies is seeking a Software Developer/Systems Engineer to design and maintain enterprise data feeds powering partner networks and security platforms. You will integrate data from numerous vendor APIs, build robust ETL pipelines, and leverage AI for complex data extraction and normalization tasks.

Responsibilities include designing parsers for diverse formats, orchestrating automated data pipelines, ensuring reliability with monitoring and optimization, and adapting quickly to API