Senior Data Engineer – High-Volume Pipelines & Provenance

NIRA, Inc.; Washington (District of Columbia); On-site; USD 150,000 - 190,000; Full time

Job summary

NIRA, Inc. is seeking a Senior Data Engineer to support the U.S. DOJ Digital Evidence Review Platform program in Washington, DC.

You will design, develop, test, operate, and optimize high-volume data ingestion and processing pipelines for digital evidence and metadata while preserving provenance, auditability, and chain-of-custody information. The role requires extensive experience in data engineering, cloud platforms, and CI/CD with a focus on secure, auditable data processing across

Qualifications

• Bachelors degree in engineering, mathematics or science as a baseline.
• 15+ years of related data engineering, integration, architecture, or large-scale data processing experience.
• Masters or PhD substitutions described for years of experience.
• Extensive experience building enterprise data ingestion, transformation and processing pipelines.
• Experience with structured and unstructured data from multiple sources and formats.
• Experience with data modeling, metadata, schema design, data lineage, and audit logging.
• Experience developing in Python, Java, Scala, SQL, or comparable data-engineering languages.
• Experience with cloud-based data platforms, distributed processing, object storage, databases, search platforms, or analytics environments.
• Experience supporting CI/CD, automated testing, configuration management, and production debugging.
• S. citizenship and ability to meet DOJ residency and security requirements.
• Ability to obtain and maintain required security credentials.

Responsibilities

• Design, develop, test, operate, and optimize high-volume ingestion and processing pipelines.
• Ingest, normalize, enrich, validate, store, search, export, archive, and restore data.
• Process forensic and metadata sources including documents, multimedia, geolocation, and logs.
• Implement validation, hashing, error handling, retry, reconciliation, and data-quality controls.
• Preserve source identifiers, lineage, provenance, and chain-of-custody events.
• Support OCR, transcription, translation, entity extraction, and metadata enrichment workflows.
• Develop schemas and canonical data models for open, portable exports.
• Support search, analytics, and AI/ML-enabled processing with human oversight and auditability.
• Automate deployment and testing via CI/CD and configuration-management practices.
• Tune pipeline performance and troubleshoot across dev, test, UAT, prod, and DR environments.
• Maintain technical documentation and collaborate with architects, cloud integrators, cybersecurity, and product specialists.

Skills

Python

Java

Scala

SQL

data-engineering languages

Education

Bachelors degree in engineering, mathematics, or science

Masters degree may substitute for two years of required experience

PhD may substitute for four years

Tools

Spark

Databricks

Kafka

Airflow

cloud-native data services

Job description

NIRA, Inc. is seeking a Senior Data Engineer to support the U.S. DOJ Digital Evidence Review Platform program in Washington, DC.

You will design, develop, test, operate, and optimize high-volume data ingestion and processing pipelines for digital evidence and metadata while preserving provenance, auditability, and chain-of-custody information. The role requires extensive experience in data engineering, cloud platforms, and CI/CD with a focus on secure, auditable data processing across