TS/SCI Spark Engineer for Mission-Critical Data Pipelines

Absolute Business Solutions Corp (ABSC); Aurora (CO); On-site; USD 150,000 - 275,000; Full time

Benefits offered by this job

Generous PTO

401k with match

Tuition Assistance

Health & Wellness allowance

Education and training funds

Volunteer Time Off

Referral Program

Living Our Values awards

Job summary

Absolute Business Solutions Corp (ABSC) is seeking a Secret or TS/SCI-cleared Apache Spark Developer to build and optimize large-scale data pipelines in a secure, air-gapped environment.

You will design and implement Spark (PySpark/Scala) pipelines, transform data with Delta Lake/Parquet, and deploy in containerized Kubernetes environments. Collaboration across analytics, AI/ML, and engineering teams is required.

Qualifications

• TS/SCI eligibility with ability to obtain/maintain counterintelligence polygraph
• Bachelor's degree plus 5 years data engineering or Spark development (or equivalent experience)
• Hands-on experience with Apache Spark, PySpark, and data processing at scale
• Experience with Parquet and Delta Lake and distributed data systems
• Familiarity with Docker/containerization and Kubernetes
• Experience with object storage (S3 or equivalent) and cloud environments (AWS/GCP/Azure)
• Strong troubleshooting and performance tuning skills
• Proficiency in Bash scripting

Responsibilities

• Design, develop, and maintain Apache Spark pipelines (batch and streaming) using PySpark and/or Scala
• Process and transform large-scale datasets using Delta Lake/Parquet in modern data lake architectures
• Optimize Spark jobs for performance including partitioning, shuffle, memory tuning, and storage efficiency
• Implement Structured Streaming pipelines for near real-time data processing
• Develop and deploy Spark applications within Docker containers and Kubernetes clusters
• Execute workloads in distributed Kubernetes environments to enable scalable processing
• Integrate Spark pipelines with analytics platforms, AI/ML workflows, and downstream systems
• Support data ingestion and storage in object-based systems and operate in air-gapped, secure environments

Skills

Apache Spark

PySpark

Scala

Kubernetes

Docker

S3

AWS

GCP

Azure

Bash

Education

Bachelor's degree + 5 years data engineering/Spark

Tools

Delta Lake

Parquet

Git

Jenkins

Job description

Absolute Business Solutions Corp (ABSC) is seeking a Secret or TS/SCI-cleared Apache Spark Developer to build and optimize large-scale data pipelines in a secure, air-gapped environment.

You will design and implement Spark (PySpark/Scala) pipelines, transform data with Delta Lake/Parquet, and deploy in containerized Kubernetes environments. Collaboration across analytics, AI/ML, and engineering teams is required.