Lead Data Engineer

K2 Integrity; New York (NY); On-site; USD 140,000 - 210,000; Full time

Job summary

K2 Integrity is seeking a Lead Data Engineer to own and evolve the data platform, from ingestion to serving layers. You will set patterns for orchestration, observability, lineage, and data contracts to ensure scale and trust across markets and products.

You will mentor engineers, design robust data models, and implement governance controls to meet regulatory requirements while enabling rapid, reliable data flows.

Qualifications

• 6+ years in data engineering with senior ownership of production data platforms.
• Deep hands-on experience with Databricks (Spark, notebooks, Delta Lake, orchestration/workflows) and PostgreSQL (complex SQL, stored procedures, materialized views).
• Proven experience building orchestration for complex dependency-heavy flows, including dependency-aware refresh, error handling, and rollback.
• Demonstrated data observability/ data quality work: automated checks, anomaly and deviation detection, alerting.
• Experience implementing or operating data lineage and understanding its role in debugging, impact analysis, and audit.
• Strong Python and SQL; solid software-engineering fundamentals (version control, testing, IaC).
• Experience in regulated domains (financial crime, AML, KYC) preferred.
• Experience with data contracts and producer/consumer interface design preferred.
• Familiarity with entity resolution and privacy regimes (ISO, GDPR) preferred.
• Experience designing platforms to scale across multiple markets and product lines preferred.

Responsibilities

• Architect orchestration, observability, lineage, and data contracts as reusable platform capabilities, and document patterns for the team.
• Design and evolve logical and physical data models across the platform, balancing consistency with flexibility for new markets.
• Build and harden ingestion and transformation pipelines; manage loads into and out of the operational serving layer.
• Implement automated data-quality and anomaly detection; define what “healthy” looks like for each key dataset.
• Establish data best practices: naming conventions, design standards, entity linkage that other services build on.
• Contribute to governance: PII masking, retention rules, and compliance controls that vary by market.
• Mentor engineers and raise the bar on rigor, testing, and operational readiness.

Skills

Python

SQL

Data modeling

Data governance

Data quality

Data lineage

Tools

Databricks

PostgreSQL

Orchestration tools

Version control

Infrastructure as code

Job description

Summary

We are looking for a Lead Data Engineer who can own more than pipelines. Beyond building and maintaining ETL and integrations, you'll own and implement data engineering best practices that make a platform trustworthy at scale: orchestration, observability, lineage, data modeling, and data contracts.

This is a design-forward, lead level role. You'll set the patterns the rest of the team builds on, and you'll be the person who can trace any record end to end, where it came from, every transformation it passed through, and every downstream consumer that depends on it so that when something upstream changes, you know exactly what it touches and can be confident the change propagates cleanly.

What You'll Own

• Ingestion. Own how data enters the platform across all channels: automated feeds, batch loads, and source-system extracts from CRM and case-management systems. Design ingestion that validates structure and mandatory elements at the door, handles varying formats and volumes across markets and products. You'll build ingestion to be reusable and configurable, so onboarding a new source or market is a known pattern rather than a one-off, and you'll make sure every inbound record carries the lineage and contract metadata the downstream layers depend on.
• Orchestration. Design the data flow layer that routes datasets through the correct processing sequence and propagates change to everything that depends on it. You'll build dependency-aware refresh, ordered execution, retry and error handling, and clean rollback, with controls on changes to core routing and strong backup/disaster recovery practices. You'll favor versioned, auditable orchestration-as-code in the existing stack over bolt-on tooling.
• Observability. Stand up the monitoring that proves the platform is healthy and ensure data integrity. This means data-quality checks at key transformation points, deviation monitoring for volume and expectations, automated flagging of anomalies, and exception reporting on breaches. You'll put monitoring on the seams where systems hand off to each other.
• Lineage. Build native lineage from ingestion through to reporting, so every element is traceable across transformations. You'll make lineage a first-class capability that supports debugging, impact analysis, audit, and compliance.
• Data modeling. Own the models that underpin the platform: the medallion layers (raw → validated → interim → canonical, with history/SCD handling), canonical entity and master-ID design, and the linkage between datasets. You'll design master-data and naming conventions and build models flexible enough to absorb differing data requirements by market and product rather than locking to a rigid schema. Sound entity/master-ID modeling is central to this role.
• Data contracts. Establish declared contracts between producers and consumers so interfaces are explicit and changes are safe.
• Pipelines and integrations. Design, build, and harden ETL/ELT across ingestion channels into the lake house and out to the operational serving layer. Own reliability, performance, and reusability.

Job Responsibilities

• Architect orchestration, observability, lineage, and data contracts as reusable platform capabilities, and document the patterns for the team.
• Design and evolve logical and physical data models across the platform's layers, balancing consistency with flexibility to onboard new markets and products.
• Build and harden ingestion and transformation pipelines; manage loads into and out of the operational serving layer.
• Implement automated data-quality and anomaly detection; define what “healthy” looks like for each key dataset.
• Establish data best practices: naming conventions, design standards, entity linkage that other services build on.
• Contribute to governance: PII masking, retention rules, and compliance controls that vary by market.
• Mentor engineers and raise the bar on rigor, testing, and operational readiness.

Requirements

• 6+ years in data engineering, with senior-level ownership of production data platforms.
• Deep hands-on experience with Databricks (Spark, notebooks, Delta Lake, orchestration/workflows, cluster management) and PostgreSQL (complex SQL, stored procedures, materialized views, performance tuning).
• Proven experience building orchestration for complex, dependency-heavy flows, including dependency-aware refresh, error handling, and rollback.
• Demonstrated data observability / data quality work: automated checks, anomaly and deviation detection, alerting — with a clear understanding that job success is not the same as correct data.
• Experience implementing or operating data lineage and a grasp of why traceability matters for debugging, impact analysis, and audit.
• Strong data modeling across relational and lake-house paradigms: layered/medallion architectures, slowly changing dimensions, and entity/master-data modeling.
• Strong Python and SQL; solid software-engineering fundamentals (version control, testing, code review, infrastructure-as-code).
• Experience in regulated or compliance-heavy domains (financial crime, AML, KYC, sanctions screening, risk) preferred.
• Experience with data contracts and producer/consumer interface design preferred.
• Familiarity with entity resolution and the downstream propagation challenges it creates preferred.
• Exposure to data governance and privacy regimes (ISO, GDPR) preferred.
• Experience designing platforms intended to scale across multiple markets and product lines preferred.

What We Value

• Rigor over throughput: you build controls and contracts, not just pipelines.
• Change by design: you plan for the platform to evolve, and make evolution safe.
• Ownership of correctness — you treat the accuracy of what the platform serves as your responsibility.
• Clear thinking about interfaces and structure: you close gaps between systems before they become incidents.