Data Engineer

Deltek; Bengaluru; Vor Ort; INR 4.000.000 - 7.000.000; Vollzeit

Zusammenfassung

Deltek is hiring a senior data architect to design and lead a medallion lakehouse on AWS, overseeing schema design, partitioning, and governance. You will build scalable Glue ETL pipelines for Bronze-to-Silver and Silver-to-Gold, and drive CDC ingestion across 18+ source systems.

You will also orchestrate end-to-end pipelines with Step Functions, monitor data quality with Lambda-based checks, and guide a team of engineers in adopting best practices, IaC, and cost optimization.

Qualifikationen

• 10+ years in software or data engineering, with architect/tech lead exposure.
• Hands-on AWS data services: S3, Glue, Athena, Step Functions, Lambda, Lake Formation, SageMaker, CloudWatch.
• Production experience with Iceberg/Delta Lake/Hudi table formats, including schema evolution and time travel.
• Strong PySpark and Python skills for large-scale data processing.
• CDC ingestion experience across heterogeneous sources (Fivetran, Debezium, etc.).
• Experience with medallion lakehouse architectures (Bronze/Silver/Gold).
• Proficient in data governance tooling: catalog, lineage, access controls, audits.
• IaC for provisioning (Terraform/CloudFormation).
• Dimensional modeling and BI-friendly schema design (star/snowflake).
• Bachelor-level degree in CS/Engineering or equivalent work experience.

Aufgaben

• Architect and evolve the full medallion lakehouse on AWS S3 with Apache Iceberg; own schema design and data partitioning policies.
• Design and implement scalable Glue ETL pipelines for Bronze-to-Silver and Silver-to-Gold transformations; incorporate dbt for SQL-layer transformations.
• Manage CDC ingestion and schema evolution across 18+ source products; ensure connector health and reliability.
• Develop and maintain AWS Step Functions and EventBridge workflows for end-to-end pipelines; implement Lambda-based quality monitors.
• Govern Glue Catalog and Lake Formation policies; enforce security controls and audit logging to meet SOC2/regulatory requirements.
• Plan and optimize query layers using Athena/Trino; evaluate deployment options (K8s or EKS) for low-latency analytics.
• Collaborate with data science teams on SageMaker data supply: feature engineering, training dataset prep, model registry integration.
• Implement real-time streaming with Kafka/Kinesis for sub-13-minute latency.
• Lead modernization efforts: evaluate Iceberg vs. Delta Lake vs. Hudi; drive cost optimization.
• Mentor junior/mid engineers; establish coding reviews, CI/CD, and incident response runbooks.

Kenntnisse

Leadership in architecture

PySpark

Python

SQL

Data modeling

Mentoring

Ausbildung

Bachelor's degree in Computer Science or related field

Tools

AWS

Glue

S3

Athena

Step Functions

Lambda

Lake Formation

SageMaker

Airflow

Trino/Presto

Terraform

CloudFormation

Iceberg/Delta/Hudi

Jobbeschreibung

• Architect and evolve the full medallion lakehouse — Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.
• Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.
• Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.
• Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.
• Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.
• Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.
• Partner with data science teams on SageMaker data supply: feature engineering pipelines, training dataset preparation, and model registry integration.
• Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.
• Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs. Delta Lake vs. Hudi), tooling, and cost optimization strategies.
• Establish and enforce data engineering best practices: code reviews, CI/CD for pipeline code, IaC (Terraform / CloudFormation), and incident response runbooks.
• Mentor and level up junior and mid-level data engineers; define team standards for pipeline design, testing, and documentation.

Required Qualifications

• 10+ years of software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.
• Deep, hands-on expertise with AWS data services: S3, Glue (PySpark ETL), Athena, Step Functions, Lambda, EventBridge, Lake Formation, SageMaker, and CloudWatch.
• Production experience with Apache Iceberg (or Delta Lake / Hudi) table formats — compaction, snapshot management, schema evolution, and time travel.
• Strong PySpark and Python skills; ability to write, review, and optimize distributed data processing jobs at scale.
• Hands-on experience with CDC-based ingestion platforms (Fivetran, Debezium, or equivalent) across heterogeneous source systems.
• Proven experience designing and implementing medallion (Bronze/Silver/Gold) or equivalent multi-hop lakehouse architectures.
• Experience with data pipeline orchestration: AWS Step Functions, Apache Airflow, or equivalent; event-driven pipeline design patterns.
• Strong SQL skills; experience with Athena, Trino, Presto, or equivalent query engines for large-scale analytical workloads.
• Familiarity with data governance tooling: catalog management (Glue Catalog, Apache Polaris/Iceberg REST), data lineage, access controls, and audit frameworks.
• Experience with Infrastructure as Code (Terraform or CloudFormation) for data platform provisioning and drift management.
• Solid understanding of dimensional modeling, schema design (star/snowflake), and data normalization for BI and analytics workloads.
• Bachelor's degree in Computer Science, Engineering, or a related field; or equivalent professional experience.

Preferred Qualifications

• Experience operating Trino or PrestoDB on Kubernetes (EKS); tuning for sub-second query latency and multi-tenant workloads.
• Familiarity with streaming platforms (Kafka, Kinesis, or Pub/Sub) and real-time lakehouse patterns.
• Experience with Apache Polaris or other Iceberg REST catalog implementations.
• Exposure to SageMaker MLOps pipelines, Model Registry, and feature store patterns for ML data supply.
• Experience with dbt (data build tool) for SQL-layer transformation and documentation in lakehouse environments.
• Government contracting or ERP domain knowledge (Costpoint, Deltek, Oracle, or similar enterprise platforms) is a strong plus.
• AWS certifications: Data Engineer Associate, Solutions Architect Professional, or equivalent.