Senior Data Engineer – Remote, Secret-Cleared Lakehouse Lead

VivSoft Technologies LLC; United States; Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Medical, Dental, Vision (employer-paid

Life Insurance

PTO (flex/combined)

401K with employer match

Dev training reimbursement

Job summary

VivSoft Technologies LLC seeks a Senior Data Engineer to join a USAF program, building a secure data platform and scalable lakehouse on AWS using Spark, Iceberg, and cloud-native tools. You will mentor engineers and drive data governance, quality, and cost controls.

You will lead design decisions, integrate with Grafana dashboards, and collaborate with Data Scientists, Cloud Engineers, and stakeholders across the organization.

Qualifications

• Active Secret Clearance required.
• 8+ years of experience in Data Engineering or large-scale data platform development.
• Expertise in Apache Spark, including performance tuning, partitioning, memory optimization, and handling data skew.
• Strong Python or Scala and advanced SQL development.
• Experience with Apache Iceberg or Delta Lake.
• Strong understanding of distributed query engines (Athena, Trino, Spark SQL).
• Hands-on AWS experience (S3, Glue, EMR, Athena, EC2, SQS).
• Experience with Apache Airflow for orchestration.
• Terraform and IaC proficiency; Kubernetes experience.
• Experience developing dashboards and data products (Grafana).
• Knowledge of data quality, governance, lineage, and access control.

Responsibilities

• Architect, build, and manage a cloud-based lakehouse on AWS using Iceberg, S3, and Glue Catalog.
• Develop and optimize Spark data pipelines on EMR and Kubernetes.
• Design event-driven ingestion using S3 events and SQS.
• Optimize queries across Athena, Trino, and Spark SQL.
• Orchestrate ETL pipelines with Apache Airflow.
• Manage infrastructure with Terraform and IaC practices.
• Develop, publish, monitor, and maintain data products and dashboards.
• Implement data quality, governance, lineage, and access controls.

Skills

Secret Clearance

Data engineering

Python/Scala

SQL

Spark tuning

Distributed queries

Technical leadership

Stakeholder comms

Mentoring

Cloud data platforms

Data governance

Dashboards

Tools

Apache Spark

Apache Iceberg

Delta Lake

AWS S3

AWS Glue

AWS EMR

Athena

Trino

Spark SQL

Airflow

Terraform

Kubernetes

Grafana

dbt

Helm

GitOps

SQS

Job description

VivSoft Technologies LLC seeks a Senior Data Engineer to join a USAF program, building a secure data platform and scalable lakehouse on AWS using Spark, Iceberg, and cloud-native tools. You will mentor engineers and drive data governance, quality, and cost controls.

You will lead design decisions, integrate with Grafana dashboards, and collaborate with Data Scientists, Cloud Engineers, and stakeholders across the organization.