Senior Data Engineer

VivSoft; Northern (KY); Remote; USD 160,000 - 180,000; Full time

Benefits offered by this job

Comprehensive Medical benefits

Life Insurance

Paid Time Off

401K Retirement Plan with employer 1:1

Professional Development Training

Job summary

VivSoft is seeking a Senior Data Engineer to lead design and operation of a scalable data platform for USAF programs. You will architect modern lakehouse on AWS using Iceberg and Spark, enabling AI/ML analytics and operational reporting.

Ideal candidates have 8+ years in data engineering, strong Python/Scala, and deep experience with AWS services, Airflow, Terraform, and governance. You will mentor engineers and collaborate with data science and cloud teams.

Qualifications

• Active Secret Clearance required.
• 8+ years in data engineering or large-scale data platforms.
• Strong Spark performance tuning and memory optimization.
• Proficient in Python or Scala and advanced SQL.
• Experience with Iceberg (or Delta Lake).
• Familiar with Athena, Trino, Spark SQL engines.
• AWS services experience (S3, Glue, EMR, SQS, etc.).
• Airflow for workflow orchestration and Terraform IaC.
• Kubernetes experience and data visualization (Grafana).
• Data governance, lineage, and access controls understanding.
• Strong leadership, mentoring, and stakeholder communication.

Responsibilities

• Architect cloud-based lakehouse on Iceberg, S3, and Glue Catalog.
• Develop and optimize Spark data pipelines on EMR and Kubernetes.
• Design event-driven ingestion using S3 events and SQS.
• Optimize queries across Athena, Trino, and Spark SQL.
• Orchestrate ETL with Airflow and manage IaC with Terraform.
• Develop, publish, monitor, and maintain data products and dashboards.
• Implement data quality, governance, lineage, and access controls.
• Create technical docs, architectural designs, and runbooks.

Skills

Secret Clearance

8+ years

Apache Spark

Python/Scala

SQL

Iceberg/Delta Lake

Athena/Trino/Spark SQL

AWS

Airflow

Terraform IaC

Kubernetes

Grafana dashboards

Data governance

Leadership mentoring

Tools

Terraform

Kubernetes

Trino

dbt

Grafana

Airflow

Job description

Job Title: SeniorData Engineer

Location: Remote

Position Type: Full-Time

Clearance Required: Secret Clearance

About the company

At VivSoft, we aim to solve complex federal problems using emerging and open technologies in a collaborative and rewarding environment. VivSoft is a diverse team of strategists, engineers, designers, and creators experienced in building high-performance, effective software, with a focus on impactful organisational design and software delivery dynamics. We build secure Software Factories based on DoD reference designs and NIST Frameworks for Cloud and DevSecOps. These factories deliver AI/ML Applications, Data Science Platforms, Blockchain and Microservices for DoD, Healthcare and Civilian Agencies

Job Summary

We are seeking a Senior Data Engineer to support a United States Air Force (USAF) program responsible for building and operating a modern, scalable, and secure data platform. This role will lead the design and optimization of an enterprise lakehouse architecture on AWS, leveraging Apache Iceberg, Apache Spark, and cloud-native technologies to enable advanced analytics, AI/ML initiatives, and operational reporting. The ideal candidate will possess deep expertise in large-scale data platforms, distributed processing, data governance, and cloud infrastructure while providing technical leadership and mentoring across engineering teams.

Key Responsibilities

• Architect, build, and manage a cloud-based lakehouse environment using Apache Iceberg, AWS S3, and AWS Glue Catalog.
• Develop and optimize Apache Spark data pipelines on EMR and Kubernetes environments.
• Design and implement event-driven data ingestion solutions using S3 events and Amazon SQS.
• Optimize query performance across Athena, Trino, and Spark SQLenvironments.
• Orchestrate data workflows and ETL pipelines using Apache Airflow.
• Manage infrastructure deployment and automation using Terraform.
• Develop, publish, monitor, and maintain data products and analytical dashboards.
• Implement data quality, governance, lineage, access controls, and cost management best practices.
• Create technical documentation, architectural designs, and operational runbooks.
• Provide technical leadership and mentorship to junior engineering team members.

Required Skills

• Must possess an active Secret Clearance
• 8+ years of professional experience in Data Engineering or large-scale data platform development.
• Expertise in Apache Spark, including performance tuning, partitioning, memory optimization, and handling data skew.
• Strong proficiency in Python or Scala and advanced SQL development.
• Experience with open table formats such as Apache Iceberg (preferred) or Delta Lake.
• Strong understanding of distributed query engines, including Athena, Trino, and Spark SQL.
• Hands-on experience with AWS services, including S3, Glue, EMR, Athena, EC2, SQS, and event-driven architectures.
• Experience with Apache Airflow for workflow orchestration.
• Proficiency with Terraform and Infrastructure as Code (IaC).
• Experience working with Kubernetes environments.
• Experience developing dashboards and data products using Grafana or similar visualization platforms.
• Strong understanding of data quality, data governance, lineage, and access control frameworks.
• Excellent technical leadership, mentoring, and stakeholder communication skills.
• Ability to translate business, operational, and analytics requirements into scalable data platform solutions.
• Strong collaboration skills with Data Scientists, AI Engineers, Cloud Engineers, and business stakeholders.
• Proven ability to lead technical discussions and mentor junior engineers.
• Excellent written and verbal communication skills.
• Strong attention to detail regarding data quality, governance, lineage, security, and operational reliability.

Preferred Skills

• Experience operating and maintaining Apache Iceberg tables at enterprise scale.
• Experience with Trino administration and performance optimization.
• Knowledge of dbt or similar data transformation frameworks.
• Experience with Helm and GitOps deployment methodologies.
• Experience evaluating and implementing modern data visualization platforms beyond Grafana.
• Familiarity with data catalog, metadata management, lineage, and governance tools.
• AWS Data Analytics Certification and/or Certified Kubernetes Administrator (CKA).
• Prior DoD, USAF, or Federal Government data platform experience.
• Experience supporting AI/ML, data science, or advanced analytics workloads in cloud environments.

Benefits

• Comprehensive Medical, Dental, and Vision Plans (Healthcare benefits are 100% employer-paid for employees only)
• Life Insurance
• Paid Time Off (Flexible/Combined PTO, Bereavement Leave, 11 Company Paid Holidays)
• 401K Retirement Plan with employer match
• Professional Development Training Reimbursement

Salary Range: $160K to $180K per Annually