Data Engineer

Deltek; United States; Remote; USD 140,000 - 230,000; Full time

Job summary

Deltek is seeking aSenior Data Platform Architect to architect and evolve a company-wide medallion lakehouse (Bronze/Silver/Gold) on AWS S3. You will own schema design, partitioning, compaction, and retention, enabling efficient analytics across 18+ applications and ML workloads.

You will lead scalable Glue ETL pipelines, manage CDC ingestion across many sources, and govern data access with security and governance controls in a SOC2-compliant environment.

Qualifications

• 10+ years of software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.

Responsibilities

• Architect and evolve the full medallion lakehouse (Bronze/Silver/Gold) on AWS S3 with Apache Iceberg, owning schema, partitioning, compaction, and retention policies.

Skills

AWS data services

PySpark

Python

SQL proficiency

Data governance

Lakehouse design

ETL engineering

CDC ingestion

Terraform

Iceberg/Delta/Hudi

SageMaker

Education

Bachelor's degree in CS/Engineering

Tools

dbt

Apache Airflow

Trino/Presto

Kubernetes (EKS)

Fivetran

Iceberg REST catalog

Job description

Key Responsibilities :

• Architect and evolve the full medallion lakehouse Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.
• Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.
• Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.
• Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.
• Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.
• Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.
• Partner with data science teams on SageMaker data supply: feature engineering pipelines, training dataset preparation, and model registry integration.
• Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.
• Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs. Delta Lake vs. Hudi), tooling, and cost optimization strategies.
• Establish and enforce data engineering best practices: code reviews, CI/CD for pipeline code, IaC (Terraform / CloudFormation), and incident response runbooks.
• Mentor and level up junior and mid-level data engineers; define team standards for pipeline design, testing, and documentation.

Required Qualifications

• 10+ years of software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.
• Deep, hands‑on expertise with AWS data services: S3, Glue (PySpark ETL), Athena, Step Functions, Lambda, EventBridge, Lake Formation, SageMaker, and CloudWatch.
• Production experience with Apache Iceberg (or Delta Lake / Hudi) table formats — compaction, snapshot management, schema evolution, and time travel.
• Strong PySpark and Python skills; ability to write, review, and optimize distributed data processing jobs at scale.
• Hands‑on experience with CDC‑based ingestion platforms (Fivetran, Debezium, or equivalent) across heterogeneous source systems.
• Proven experience designing and implementing medallion (Bronze/Silver/Gold) or equivalent multi‑hop lakehouse architectures.
• Experience with data pipeline orchestration: AWS Step Functions, Apache Airflow, or equivalent; event‑driven pipeline design patterns.
• Strong SQL skills; experience with Athena, Trino, Presto, or equivalent query engines for large‑scale analytical workloads.
• Familiarity with data governance tooling: catalog management (Glue Catalog, Apache Polaris/Iceberg REST), data lineage, access controls, and audit frameworks.
• Experience with Infrastructure as Code (Terraform or CloudFormation) for data platform provisioning and drift management.
• Solid understanding of dimensional modeling, schema design (star/snowflake), and data normalization for BI and analytics workloads.
• Bachelor's degree in Computer Science, Engineering, or a related field; or equivalent professional experience.

Preferred Qualifications

• Experience operating Trino or PrestoDB on Kubernetes (EKS); tuning for sub‑second query latency and multi‑tenant workloads.
• Familiarity with streaming platforms (Kafka, Kinesis, or Pub/Sub) and real‑time lakehouse patterns.
• Experience with Apache Polaris or other Iceberg REST catalog implementations.
• Exposure to SageMaker MLOps pipelines, Model Registry, and feature store patterns for ML data supply.
• Experience with dbt (data build tool) for SQL‑layer transformation and documentation in lakehouse environments.
• Government contracting or ERP domain knowledge (Costpoint, Deltek, Oracle, or similar enterprise platforms) is a strong plus.
• AWS certifications: Data Engineer Associate, Solutions Architect Professional, or equivalent.

What You Will Build

You will be a founding architect of a strategic, cross‑product data platform that serves 18+ enterprise applications and their analytics, ML, and AI workloads. The decisions you make on schema, storage format, query layer, governance, and orchestration will shape the data foundation of the company for years. This is a high‑impact, high‑ownership role with direct visibility to senior leadership.