Senior Data Engineer: Lakehouse Architect (Remote, Secret)

VivSoft Technologies LLC; United States; Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Comprehensive Medical, Dental, Vision

Life Insurance

PTO + Holidays

401K with match

Professional Development Budget

Job summary

VivSoft Technologies LLC is seeking a Senior Data Engineer to design and operate a scalable, secure data platform on AWS. You will lead lakehouse architecture using Iceberg, Spark, and cloud-native tools, enabling advanced analytics and AI workloads while mentoring engineering teams.

The role requires 8+ years in data engineering, strong Spark and Python/Scala skills, and a Secret Clearance. You will work on DoD-focused data initiatives with a remote-friendly setup.

Qualifications

• Active Secret Clearance required.
• 8+ years of professional experience in data engineering or large-scale data platform development.
• Expertise in Apache Spark, including performance tuning and memory optimization.
• Strong proficiency in Python or Scala and advanced SQL development.
• Experience with open table formats such as Apache Iceberg or Delta Lake.
• Strong understanding of distributed query engines (Athena, Trino, Spark SQL).
• Hands-on experience with AWS services (S3, Glue, EMR, Athena, EC2, SQS).
• Experience with Airflow for workflow orchestration.
• Proficiency with Terraform and IaC, Kubernetes environments.
• Experience developing dashboards and data products (Grafana).
• Data quality, governance, lineage, access control knowledge.
• Excellent technical leadership and mentoring skills.

Responsibilities

• Architect, build, and manage a cloud-based lakehouse with Iceberg, S3, and Glue Catalog.
• Develop and optimize Spark data pipelines on EMR and Kubernetes.
• Design event-driven ingestion using S3 events and SQS.
• Optimize queries across Athena, Trino, and Spark SQL environments.
• Orchestrate data workflows with Apache Airflow.
• Manage infrastructure deployment and automation with Terraform.
• Develop, publish, monitor, and maintain data products and dashboards.
• Implement data quality, governance, lineage, and access controls.
• Create architectural designs and runbooks.
• Provide technical leadership and mentorship to junior engineers.

Skills

Secret Clearance

8+ Years Exp

Spark Performance

Python/Scala

Advanced SQL

Iceberg/Delta

AWS Data Stack

Airflow

IaC Terraform

Kubernetes

Dashboards Grafana

Data Governance

Leadership/Mentoring

Tools

Apache Iceberg

Terraform

Airflow

Kubernetes

Grafana

Trino

Athena

EMR

S3

Glue

Job description

VivSoft Technologies LLC is seeking a Senior Data Engineer to design and operate a scalable, secure data platform on AWS. You will lead lakehouse architecture using Iceberg, Spark, and cloud-native tools, enabling advanced analytics and AI workloads while mentoring engineering teams.

The role requires 8+ years in data engineering, strong Spark and Python/Scala skills, and a Secret Clearance. You will work on DoD-focused data initiatives with a remote-friendly setup.