Senior Data Engineer

OAG Aviation Worldwide; United States; Hybrid; USD 57,000 - 100,000; Full time

Benefits offered by this job

Private health insurance

Company bonus scheme

Company pension plan

Flexible remote work

Lunch and snacks in office

Job summary

OAG Aviation Worldwide is modernizing its Flight Status data platform by building high-throughput ingestion pipelines using Azure Databricks and Delta Lake. You will implement scalable streaming and batch ETL, enforce data quality, and improve observability across the data lakehouse.

The role emphasizes DevOps, CI/CD, and AI-assisted tooling to accelerate delivery. You will mentor engineers and collaborate with cross-functional teams to define data contracts and governance.

Qualifications

• Senior data engineer with a track record of building and scaling production-grade streaming and batch ETL/ELT pipelines.
• Hands-on proficiency with Azure Databricks, Spark (PySpark), Structured Streaming, Delta Lake, and Unity Catalog for governance.

Responsibilities

• Deliver real-time and batch data pipelines to ingest multi-source flight data into the Lakehouse platform.
• Implement reusable data patterns, data quality controls, and observability to ensure data quality and reliability.
• Tune compute, streaming triggers, and storage for cost efficiency, latency, and scalability.
• Design automated job orchestration with Databricks Workflows and maintain CI/CD pipelines for reliable deployments.
• Incorporate AI-assisted tools to boost productivity and edge-case testing in pipelines.
• Collaborate with product, analytics, and platform teams to define data contracts and mentor engineers.

Skills

Senior data engineering

Databricks & governance

Delta Lake

Structured Streaming

CI/CD & DevOps

AI tooling adoption

Collaborative leadership

Tools

Azure Databricks

PySpark

Unity Catalog

Delta Lake

Databricks Workflows

Azure DevOps/GitHub Actions

Job description

OAG is the essential intelligence partner transforming global travel. For nearly a century, we’ve been the foundation the travel industry builds on, trusted, proven, essential. Today, we’re going beyond.Wecombine unmatched data quality, quantity and access with breakthrough technology and deep industryexpertiseto deliver what the industry truly needs: not just more data, but better answers. From real-time operational intelligence to strategic decision-making, we power the entire travel ecosystem, from airlines and airports to technology companies and financial institutions.

We are actively embedding AI throughout our intelligence layer, combiningnearly acentury of proprietary data with the latest technology to deliver answers that were simply not possible before. We exist because travel moves the world, and we move travel. By transforming complexity into clarity and questions into answers, we give our partners the intelligence they need to thrive in an evolving world. For more information visit www.oag.com

ABOUT THE ROLE:

You will join a team of software and data engineers modernizing OAG’s core Flight Status data platform. You will play a key role in building high-throughput, near real-time ingestion and processing pipelines that handle global flight status data using Azure Databricks and Delta Lake. As OAG actively embeds AI across its intelligence layer and adopts modern agentic development practices, you will leverage AI-assisted tools to accelerate delivery, improve system observability, and support AI-driven data products.

WHAT YOU’LL BE DOING:

• Deliver Real-Time Data Pipelines: Build, scale, and maintain high-volume batch and Structured Streaming pipelines to ingest multi-source flight data into the Lakehouse platform.
• Drive Engineering Standards: Implement reusable data patterns, automated data quality controls, and observability to ensure data consistency, accuracy, and reliability.
• Optimize Platform Performance: Monitor and tune compute resources, streaming triggers, and storage for cost efficiency, low latency, and high scalability.
• Automate Workflows & DevOps: Design automated job orchestration using Databricks Workflows and maintain CI/CD pipelines for reliable deployments.
• Leverage Modern AI Workflows: Integrate AI-assisted tools into daily engineering practices to boost productivity, edge‑case testing, and pipeline automation.
• Collaborate & Lead: Work closely with product, analytics, and platform teams to define data contracts, review code, and mentor engineers across the domain.

WHAT WE’RE LOOKING FOR:

• Senior Data Engineering Experience: Strong track record building and scaling production‑grade streaming and batch ETL/ELT data pipelines.
• Databricks & Governance Mastery: Hands‑on proficiency with Azure Databricks, Apache Spark (PySpark), Structured Streaming, Delta Lake, and Unity Catalog (for fine‑grained access control, data governance, and lineage).
• Advanced Pipeline Design: Deep understanding of modern data patterns, including Medallion architecture, Change Data Capture (CDC), SCD Type 1/2, MERGE/upsert strategies, and schema evolution.
• Orchestration & Streaming: Practical experience building job dependencies with Databricks Workflows and integrating event streaming platforms (e.g., Azure Event Hubs, Kafka).
• DevOps & Observability Focus: Strong experience with automated testing, CI/CD tools (Azure DevOps or GitHub Actions), and data observability practices.
• AI Tooling Adoption: Active experience using AI coding assistants (e.g., GitHub Copilot, Cursor) to accelerate development while maintaining high code quality and ownership.
• Ownership & Collaboration: Strong analytical skills, a results‑oriented mindset, and excellent technical communication skills.

IF YOU HAVE THE FOLLOWING THAT WOULD BE GREAT:

• Databricks AI/ML Ecosystem: Experience with MLflow, Databricks AI Functions, or vector search in streaming/data workflows.
• Specialized Data Quality Tools: Hands‑on experience with dedicated data quality and anomaly detection frameworks (e.g., Great Expectations, Soda).
• Infrastructure as Code (IaC): Practical experience with Terraform for cloud resource management.
• Industry Certifications: Databricks Certified Data Engineer Professional or Microsoft Certified: Azure Data Engineer Associate.

WE WILL OFFER YOU:

A progressive, non-hierarchical culture where innovation drives everything we build

Flexible working arrangements that allow remote work or office attendance

An attractive compensation and benefits package, including private health insurance from day one of joining, a company bonus scheme and company pension scheme

22 days holiday, growing with each year of service, volunteering days and an additional day off during your birthday month

Inclusive family benefits designed to support you through parenthood and fertility journeys

Annual learning and development budget

Work remotely anywhere in the world for up to 1 month per year

Free lunch and snacks provided in the office.

Monthly salary range from 4583 EUR to 8000 EUR gross (exact compensation may vary based on skills, experience and competencies).

OAG is committed to creating an inclusive workplace where everyone feels valued and able to perform at their best.

We welcome applications from people of all backgrounds and experiences. If yourequirereasonable adjustments at any stage of the recruitment process, please let us know —we’rehappy to have an open conversation about how we can support you.

Salary Range

€4.583—€8.000 EUR