Senior Data Engineer

Simulmedia; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Simulmedia is seeking an experienced Data Engineer to join our Data Services team. You will design, build and operate the data platform powering our TV and streaming advertising platform, ingesting and transforming multi-billion-row datasets from external partners, and delivering robust, accessible data to internal products.

The role emphasizes strong Python and SQL skills and experience with Spark/Databricks, Redshift and Airflow, along with AWS and Docker.

Qualifications

• Bachelor’s degree in CS, Engineering, or equivalent practical experience.
• 7+ years of work experience as a data engineer.
• Proficiency in Python as the primary development language in recent years.
• Expert-level SQL: complex analytics on very large tables and debugging discrepancies.
• Hands-on experience with distributed data processing platforms (Spark/Databricks preferred; EMR, Snowflake or BigQuery also relevant) and columnar data warehouses (Redshift, Snowflake, BigQuery, ClickHouse, etc).
• Ability to design complex data models: normalized, dimensional and temporal (slowly changing dimensions, etc).
• Experience with workflow orchestration tools (Airflow or similar).
• Experience integrating third-party data feeds with schema drift handling and versioned reference data.
• Experience building REST services in Python (FastAPI, Flask, etc).
• Experience developing, maintaining, and debugging large server-side codebases.
• Working knowledge of AWS (S3, IAM, ECS or similar compute) and Docker.
• Knowledge of engineering best practices and testing (unit/integration tests, code review, CI/CD).
• Desire to take ownership and learn new technologies; ability to communicate with U.S.-based teams.
• Knowledge of Delta Lake / medallion lakehouse architectures is a plus.
• Experience migrating legacy pipelines with parity requirements is a plus.
• Experience with advertising, media or measurement data is a plus.

Responsibilities

• Design and build batch data pipelines ingesting multi-billion-row datasets from external data providers and internal systems.
• Model complex real-world data including slowly changing dimensions and temporal data, safely evolving models as upstream schemas change.
• Develop and operate workloads on lakehouse platform (Databricks / Spark / Delta) and Redshift, migrating pipelines as needed.
• Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting.
• Prove correctness with parity checks and large historical backfills; investigate data discrepancies at row level.
• Build and maintain Python services and REST APIs serving data to internal products.
• Optimize performance and cost: query tuning, table design, workload management and right-sizing compute.
• Own deployments: monitor production pipelines, incident triage and root-cause analysis.
• Collaborate cross-functionally with product managers, data scientists, and stakeholders; work in an Agile team.
• Take ownership and experiment with new technologies to improve our software.

Skills

Python

SQL

Spark/Databricks

Airflow

REST APIs

AWS

Docker

CI/CD

Postgres

Redshift

Snowflake

BigQuery

Delta Lake

Education

Bachelor's degree in CS/Engineering

Tools

Databricks

Redshift

Snowflake

BigQuery

ClickHouse

Job description

Simulmedia is looking for an experienced and dynamic Data Engineer with a curious and creative mindset to join our Data Services team. The ideal candidate will have a strong background in Python, SQL and large-scale data pipelines. This is an opportunity to join a team of amazing engineers, data scientists, product managers and designers who are obsessed with building the most advanced TV and streaming advertising platform in the market. As a Data Engineer you will design, build and operate the data platform that powers the company: pipelines that ingest and transform very large datasets from external data partners, data models that the whole company queries, and services that make that data available to internal products. You will work on a team that empowers the other teams to use our huge amount of data efficiently. Using a large variety of technologies and tools, you will solve complicated technical problems and build solutions to make our pipelines robust and fault tolerant and our data easily accessible throughout the company.

Location: Ukraine is mandatory. Our offices are located in Kyiv and Lviv. Teams are located in Kyiv and Lviv and primarily work remotely with occasional offline meetings.

Responsibilities:

• Design and build batch data pipelines that ingest, validate and transform multi-billion-row datasets from external data providers and internal systems
• Model complex real-world data: dimensional models, reference data, and temporal data whose attributes change over time (e.g. slowly changing dimensions), and evolve those models safely as upstream sources change their schemas and semantics
• Develop and operate workloads on our lakehouse platform (Databricks / Spark / Delta) and our data warehouse (Redshift), including migrating existing pipelines from the warehouse to the lakehouse
• Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting
• Prove correctness, not just completion: design parity checks and reconciliation queries when replacing an existing pipeline, run large historical backfills, and investigate data discrepancies down to the row level
• Build and maintain Python services and REST APIs that serve data to internal products
• Optimize for performance and cost: query tuning, table design, workload management and right-sizing compute
• Own what you ship: monitor production pipelines, participate in incident triage and root-cause analysis, and harden systems so the same failure does not happen twice
• Collaborate cross-functionally with product managers, data scientists and stakeholders across the company to deliver on product roadmap
• Work within an Agile team that releases cutting-edge new features regularly
• Take a high degree of ownership and freedom to experiment with new technologies to improve our software

Qualifications:

• Bachelor’s degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
• 7+ years of work experience as a data engineer
• Proficiency in Python and using it as the primary development language in recent years
• Expert-level SQL: comfortable writing, reading and tuning complex analytical queries against very large tables, and debugging why two result sets disagree
• Hands-on experience with a distributed data processing platform (Spark/Databricks strongly preferred; EMR, Snowflake or BigQuery also relevant) and with a columnar data warehouse (Redshift, Snowflake, BigQuery, ClickHouse, etc)
• Ability to design complex data models: normalized, dimensional and temporal (slowly changing dimensions, effective-dated records, point-in-time correctness)
• Experience with workflow orchestration tools (Airflow or similar): building DAGs, managing dependencies and running backfills
• Experience integrating third-party data feeds: handling schema drift, late or missing deliveries, vendor data-quality defects and versioned reference data
• Experience building REST services in Python (FastAPI, Flask, etc)
• Experience developing, maintaining, and debugging problems in large server-side code bases
• Working knowledge of AWS (S3, IAM, ECS or similar compute) and Docker
• Good knowledge of engineering best practices and testing (unit test, integration test, code review, CI/CD)
• The desire to take a high level of ownership of the things you work on
• Ability to learn new things quickly, maintain a high bar for quality, and be pragmatic
• Must be able to communicate with U.S based teams
• Experience with Delta Lake / medallion lakehouse architectures is a plus
• Experience migrating legacy pipelines between platforms with strict parity requirements is a plus
• Experience with advertising, media or measurement industry data is a plus
• Ability to communicate effectively with the U.S.-based teams and work 11:00 AM — 8:00 PM EEST (11:00 - 20:00).

Our Tech Stack:

• Almost everything we run is on AWS (S3, ECS, EMR, RDS and more)
• Python is our primary language; SQL is everywhere
• Databricks (Spark, Delta Lake) is our lakehouse platform; Redshift and Postgres are our warehouses and operational databases
• Airflow orchestrates our pipelines
• Docker for packaging; GitHub Actions and Jenkins for CI/CD
• Grafana, Sentry and OpenSearch for observability
• Datasets measured in billions of rows

Interview Process:

1. Pre-screening (30 mins)
2. Technical Interview (1h)
3. System Design (1.5h)
4. Product Interview (30 mins)
5. Offer