Lead Data Engineer, Consumer

Versant Media; New York (NY); On-site; USD 170,000 - 190,000; Full time

Job summary

Versant Media is seeking a hands-on Lead Data Engineer to guide end-to-end data product engineering, delivering trusted Silver-layer datasets from source systems into the enterprise data platform. This role blends technical leadership with delivery accountability across ingestion, transformation, governance, and team execution.

Based in New York, NY, the position focuses on building scalable data pipelines, data quality, and governance practices to enable Gold-layer analytics and AI consumption.

Qualifications

• 8+ years in data engineering or software/data-platform work, with 2+ years leading engineers or delivery teams.
• Proven design and delivery of enterprise-scale ingestion and transformation pipelines.
• Strong hands-on skills in SQL, Python, Spark/PySpark and modern ELT/ETL patterns.

Responsibilities

• Lead delivery of source-to-Silver data products across the data stack.
• Oversee a distributed/offshore team of Data Engineers and drive delivery commitments.
• Translate roadmaps into engineering plans with milestones and dependencies.
• Design, oversee ingestion, transformation, standardization, orchestration, and publication of data.

Skills

SQL

Python

Spark/PySpark

ELT/ETL

Databricks

Delta Lake

CI/CD

Bronze/Silver/Gold

change-data-capture

Tools

Databricks

Delta Lake

Unity Catalog

Job description

Versant Media is seeking a hands-on Lead Data Engineer to guide end-to-end data product engineering, delivering trusted Silver-layer datasets from source systems into the enterprise data platform. This role combines technical leadership with delivery accountability across ingestion, transformation, governance, and team execution.

Key Responsibilities

• Lead delivery of source-to-Silver data products.
• Lead and develop a distributed/offshore team of Data Engineers responsible for delivery from source ingestion through curated Silver-layer data products.
• Translate product roadmaps and requirements into engineering plans, including milestones, estimates, dependencies, and delivery commitments.
• Design and oversee ingestion, transformation, standardization, orchestration, and publication of data from operational, SaaS, file, streaming, API, and partner sources.
• Ensure Silver-layer data products are validated, standardized, documented, reusable, performant, secure, and ready for Gold-layer analytics and AI consumption.
• Establish repeatable patterns for batch, incremental, change-data-capture, and event-driven processing.
• Build scalable engineering foundations through reusable frameworks, templates, libraries, and pipeline patterns that reduce repeated effort across products and domains.
• Apply modern engineering practices, including source control, peer review, automated testing, CI/CD, observability, release management, and incident remediation.
• Define and maintain standards for naming, partitioning, schema evolution, error handling, replay/recovery, performance, cost management, and documentation.
• Partner with the Data Platform team to use approved workspace, compute, storage, security, and deployment patterns.
• Identify technical debt and lead pragmatic improvements to increase delivery speed, reliability, and maintainability.
• Collaborate with Data Modelers on canonical entities, conformed dimensions, data contracts, source-to-target mappings, and enterprise modeling standards.
• Partner with Product Managers and domain leaders to clarify outcomes, source-system realities, priority use cases, and acceptance criteria.
• Coordinate dependencies with source-system owners, platform teams, analytics teams, and external partners.
• Establish data profiling, reconciliation, quality testing, freshness monitoring, lineage, and alerting for each delivered data product.
• Ensure source-to-Silver traceability, including authoritative source identification, transformation logic, ownership, metadata, and data-quality expectations.
• Implement access controls, sensitive-data classification, masking, retention, and regional/data-residency requirements.
• Drive resolution of data defects, schema changes, pipeline failures, and quality issues through clear ownership and service-level expectations.
• Set priorities, technical direction, delivery expectations, and quality standards for the engineering team.
• Coach engineers in data engineering, cloud development, testing, observability, and product-oriented delivery practices.
• Create an effective onshore/offshore delivery model with defined handoffs, overlap hours, documentation standards, ceremonies, and escalation paths.
• Communicate delivery progress, risks, tradeoffs, and decisions clearly to technical and business stakeholders.
• Build a culture of ownership, continuous improvement, and reliable execution.

Required Qualifications

• 8+ years of data engineering, software engineering, or data-platform experience, including 2+ years leading engineers, technical workstreams, or large-scale delivery.
• Demonstrated experience designing and delivering enterprise-scale ingestion and transformation pipelines.
• Strong hands-on expertise in SQL, Python, Spark/PySpark, and modern ELT/ETL patterns.
• Experience with Databricks, Delta Lake, or a comparable lakehouse platform; experience with cloud storage, orchestration, and CI/CD.
• Strong understanding of Bronze/Silver/Gold (or equivalent layered data-platform patterns).
• Experience implementing data quality, observability, lineage, metadata, and production support practices.
• Proven ability to partner with architects, data modelers, product leaders, and domain stakeholders.
• Experience leading distributed teams and creating effective delivery practices across time zones.
• Strong communication skills, including the ability to explain technical choices, risks, and tradeoffs to non-technical stakeholders.

Technologies

• SQL
• Python
• Spark
• PySpark
• ELT
• ETL
• Databricks
• Delta Lake
• CI/CD
• Bronze/Silver/Gold
• change-data-capture

Preferred Qualifications

• Experience with Unity Catalog or comparable governance, catalog, and access-control capabilities.
• Experience with data contracts, schema evolution, change data capture, APIs, event streaming, and large-volume data processing.
• Experience supporting multiple data domains, global data products, or regional data-residency requirements.
• Experience with infrastructure-as-code, DataOps, and automated environment provisioning.
• Experience in a regulated, high-scale, or operationally sensitive environment.

Role Details

• Location: New York, NY (onsite)
• Compensation: USD 170,000 - 190,000 per year
• Experience Level: 8+ years

Additional Information

• As part of the selection process, external candidates may be required to attend an in-person interview with a VERSANT Media employee at one of its locations prior to a hiring decision.
• Qualified individuals with a disability or disabled veterans who require support during the application and/or recruitment process may request a reasonable accommodation by emailing [e-mail].
• VERSANT Media is committed to fair and equitable compensation practices.
• VERSANT Media is not accepting unsolicited assistance from search firms for this employment opportunity.