TS/SCI Spark Engineer—High-Performance Data Pipelines

Absolute Business Solutions Corp (ABSC); Springfield (VA); On-site; USD 150,000 - 275,000; Full time

Benefits offered by this job

Generous PTO

401k with match

Tuition assistance

Wellness allowance

Education funds

Volunteer time off

Charitable match

Referral program

LOV awards

Job summary

Absolute Business Solutions Corp (ABSC) seeks a Secret or TS/SCI-cleared Apache Spark Developer to build large-scale data pipelines in a containerized, Kubernetes-based environment. This on-site role supports NGAs Data Modernization Services (DMS) with emphasis on performance tuning in air-gapped systems.

The candidate will design Spark pipelines (PySpark/Scala), optimize processing, and integrate with AI/ML workflows while working across multiple locations including Herndon, VA and Springfield,

Qualifications

• Bachelor's degree plus 5 years experience in data engineering or Spark development.
• Strong hands-on experience with Apache Spark or Spark/Scala, Python (PySpark).
• Experience with Parquet and Delta Lake and distributed data systems.
• Familiarity with Docker and Kubernetes; basic to intermediate experience.
• Experience with object storage (e.g., S3) and performance tuning.

Responsibilities

• Design, develop, and maintain Spark pipelines (batch and streaming) using PySpark/Scala.
• Process large datasets with Delta Lake/Parquet in data lake architectures.
• Optimize Spark jobs for performance: partitioning, shuffle, memory tuning.
• Implement Structured Streaming for near real-time processing.
• Develop Spark apps in Docker containers and run on Kubernetes.
• Integrate pipelines with analytics platforms and AI/ML workflows.
• Troubleshoot pipeline failures and ensure reliability in secure environments.

Skills

Apache Spark

PySpark

Python

Kubernetes

Docker

Delta Lake

Parquet

Shell scripting

S3 storage

CI/CD (Jenkins)

Education

Bachelor's degree

Tools

Git

Jenkins

Terraform

Iceberg

Job description

Absolute Business Solutions Corp (ABSC) seeks a Secret or TS/SCI-cleared Apache Spark Developer to build large-scale data pipelines in a containerized, Kubernetes-based environment. This on-site role supports NGAs Data Modernization Services (DMS) with emphasis on performance tuning in air-gapped systems.

The candidate will design Spark pipelines (PySpark/Scala), optimize processing, and integrate with AI/ML workflows while working across multiple locations including Herndon, VA and Springfield,