Senior Data Engineer: Real-Time Pipelines & AI (Databricks)

OAG Aviation Worldwide; United States; Hybrid; USD 57,000 - 100,000; Full time

Benefits offered by this job

Private health insurance

Company bonus scheme

Company pension plan

Flexible remote work

Lunch and snacks in office

Job summary

OAG Aviation Worldwide is modernizing its Flight Status data platform by building high-throughput ingestion pipelines using Azure Databricks and Delta Lake. You will implement scalable streaming and batch ETL, enforce data quality, and improve observability across the data lakehouse.

The role emphasizes DevOps, CI/CD, and AI-assisted tooling to accelerate delivery. You will mentor engineers and collaborate with cross-functional teams to define data contracts and governance.

Qualifications

• Senior data engineer with a track record of building and scaling production-grade streaming and batch ETL/ELT pipelines.
• Hands-on proficiency with Azure Databricks, Spark (PySpark), Structured Streaming, Delta Lake, and Unity Catalog for governance.

Responsibilities

• Deliver real-time and batch data pipelines to ingest multi-source flight data into the Lakehouse platform.
• Implement reusable data patterns, data quality controls, and observability to ensure data quality and reliability.
• Tune compute, streaming triggers, and storage for cost efficiency, latency, and scalability.
• Design automated job orchestration with Databricks Workflows and maintain CI/CD pipelines for reliable deployments.
• Incorporate AI-assisted tools to boost productivity and edge-case testing in pipelines.
• Collaborate with product, analytics, and platform teams to define data contracts and mentor engineers.

Skills

Senior data engineering

Databricks & governance

Delta Lake

Structured Streaming

CI/CD & DevOps

AI tooling adoption

Collaborative leadership

Tools

Azure Databricks

PySpark

Unity Catalog

Delta Lake

Databricks Workflows

Azure DevOps/GitHub Actions

Job description

OAG Aviation Worldwide is modernizing its Flight Status data platform by building high-throughput ingestion pipelines using Azure Databricks and Delta Lake. You will implement scalable streaming and batch ETL, enforce data quality, and improve observability across the data lakehouse.

The role emphasizes DevOps, CI/CD, and AI-assisted tooling to accelerate delivery. You will mentor engineers and collaborate with cross-functional teams to define data contracts and governance.