Sr Data Engineer

Valid8 Financial, Inc.; New York (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Valid8 Financial, Inc. is seeking a Senior Data Engineer to innovate and enhance our data platform, primarily focusing on scalable data architectures to drive media monetization. The role involves designing data pipelines, collaborating with various teams, and ensuring data quality in a fast-paced environment.

Ideal candidates will possess over 5 years of data engineering experience, along with strong skills in SQL, Python, and modern cloud platforms such as GCP and AWS. A background in the media industry is preferred.

Qualifications

• 5+ years of experience in building scalable data engineering platforms.
• Strong experience in modern data pipelines and architectures.
• Advanced skills in SQL, Python, and PySpark.

Responsibilities

• Design and maintain scalable data pipelines for media monetization.
• Ingest and transform high-volume media data from multiple sources.
• Collaborate with teams to support ML and enable self-service insights.

Skills

Data engineering

SQL

Python

PySpark

Cloud platforms (GCP & AWS)

ETL/ELT processes

Data quality assurance

Agile methodologies

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

Dagster

Data Lake

Data Warehouse

Tableau

Metabase

Job description

We're looking for a Senior Data Engineer that is innovative, curious, and collaborative to help evolve the data platform that powers our sell-side media business. This role supports data-driven workflows and a broad range of media monetization use cases, with an emphasis on scalable, open data architectures.

You’ll work on high-volume, revenue-critical media data while helping modernize how data is stored, processed, and served across the organization. This is a hands-on role for engineers who enjoy building, migrating, and improving platforms, not just maintaining them.

What You’ll Do

• Design, build, and maintain scalable, reliable data pipelines that support sell-side media monetization.
• Ingest, integrate, transform, and model high-volume media and campaign data from multiple sources, delivering analytics-ready datasets that meet quality, accessibility, and business requirements.
• Develop lakehouse-style data models that balance flexibility, performance, and cost efficiency, enabling reporting, analytics, operational workflows, and downstream consumption.
• Build, optimize, and maintain ETL/ELT processes for both batch and near-real-time workloads, orchestrated with modern workflow tools (e.g., Airflow, Dagster).
• Collaborate with machine learning, client-facing, product, application, and analytics teams to understand requirements, support ML feature productionization, and enable self-service insights.
• Ensure data quality, lineage, observability, governance, and security, safeguarding cloud data assets and maintaining trust in revenue-critical systems.
• Contribute to platform evolution and migration efforts, including evaluating tools, improving workflows, and reducing architectural complexity.
• Continuously optimize pipelines, queries, and storage for performance, scalability, and cost efficiency.
• Mentor junior engineers and participate in technical design and architecture reviews.
• Detect, investigate, and resolve data anomalies to maintain pipeline reliability and data trustworthiness.

Required Experience & Skills

Core Requirements

• 5+ years of experience in building highly scalable and reliable data engineering and analytics platforms
• Strong experience building and optimizing modern data pipelines, architectures, and datasets using Data Lake, Data Warehouse, and Lakehouse paradigms
• Advanced SQL, Python, and PySpark skills and experience building analytics-ready data models
• Experience using Iceberg, Delta, and Parquet data formats
• Experience using Dagster and Airflow orchestration tools
• 2+ years hands-on experience with cloud platforms, such as GCP & AWS
• Experience designing and building data pipelines on object storage-based platforms (e.g., S3, Wasabi, Dremio, Snowflake, Delta Lake) to process and manage large-scale analytical datasets
• Solid understanding of data quality, testing, monitoring, and operational reliability
• Strong communication skills and ability to work closely with technical and non-technical stakeholders
• Experience collaborating with Software Engineers using Agile methodologies to build web applications that access, visualize, and sometimes update big data stores in a hybrid OLAP and OLTP environment.
• Bachelor's degree or equivalent work experience (minimum 5 years) in Computer Science or related field

Preferred/Nice-to-Have

• Experience in media industry strongly preferred including familiarity with sell-side concepts (linear TV or digital inventory, campaign delivery, pacing, audience-based selling and measurement)
• 1+ years experience supporting or leading data platform migrations, hybrid architectures, or warehouse-to-lakehouse transitions using Dremio is a strong plus
• Experience with Tableau, Metabase, or other data visualization tools
• Experience working in an operational environment with time-sensitive customer commitments