Senior Data Platform Engineer CR 12/11/26

Margo Group

United States

Remote

USD 150,000 - 210,000

Full time

10 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Urbano DX in the United States is seeking a Senior Data Platform Engineer to drive the evolution of our Azure-based data platform and lakehouse architecture. You will own the orchestration, governance, and cost-aware operation of critical data infra used by product teams.

You will migrate workflows from ADF to Apache Airflow, manage Terraform-based Azure resources, secret management, Unity Catalog governance, and implement observability and CI/CD with Azure DevOps.

Qualifications

  • Experience designing and evolving data platform architectures.
  • Proven ability to lead platform migrations and governance.
  • Strong hands-on expertise with Azure data services and IaC.

Responsibilities

  • Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards.
  • Work on the migration from Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework (DAG generation, reusable operators, dependency and scheduling patterns).
  • Ensure Airflow is reliable and observable : SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.
  • Manage Azure resources with Infrastructure as Code (Terraform) : Databricks workspaces, Data Lake Storage , Data Factory, Key Vault , networking and supporting services.
  • Own secrets and identity management : Key Vault, service principals and managed identities, secret scopes and rotation.
  • Own access management : Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege , plus access requests and periodic reviews.
  • Own Azure DevOps : repositories, CI/CD pipelines , branch policies and multi-environment promotion , including deployment validation and Terraform state.
  • Track cost and governance of Azure resources: tagging, budgets, right‑sizing and cost attribution (e.g. query tagging).
  • Contribute to the implementation of the Data Catalog and Data Quality Framework and it’s lineage across the platform.
  • Drive observability : pipelines that are observable, debuggable and alert‑driven, with clear ownership of failures.
  • Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.
  • Participate in architecture reviews and technical decision-making, stating trade‑offs (performance, cost, reliability, complexity) with data.
  • Define and promote engineering standards : naming conventions, patterns and guidelines, through code reviews and design reviews.
  • Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
  • Build and maintain developer tooling and automation , including AI‑assisted engineering workflows (shared agent instructions, skills, automated PR review).
  • Provide second‑line support to the Run/Operations team for complex production topics.
  • Troubleshoot and resolve platform incidents , failures and performance issues; lead root cause analysis and post‑mortems.
  • Ensure proper handover, documentation and operational readiness of platform changes.
  • Continuously improve runbooks, monitoring and operational processes.

Skills

Technical leadership
System architecture
Cross-functional collaboration
Communication

Tools

Apache Airflow
Azure Data Factory
Terraform
Databricks
Unity Catalog
Azure DevOps
Key Vault
IAM

Job description

We are looking for a Senior Data Platform Engineer to help drive the evolution of our Azure-based Data Platform and Lakehouse architecture .

This role is focused on the platform itself . The professional will own and evolve the foundations that data product teams build on: Apache Airflow orchestration (migrating from ADF) , Azure resource management (Key Vault, identity and access, networking, storage), Azure DevOps CI/CD and Infrastructure as Code , Databricks and Unity Catalog governance , observability and cost efficiency , while remaining hands‑on and close to the teams that use the platform.

Key Responsibilities
Orchestration Platform (Apache Airflow & ADF)
  • Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards .
  • Work on the migration from Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework (DAG generation, reusable operators, dependency and scheduling patterns).
  • Ensure Airflow is reliable and observable : SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.
Azure Resource Management
  • Manage Azure resources with Infrastructure as Code (Terraform) : Databricks workspaces, Data Lake Storage , Data Factory, Key Vault , networking and supporting services.
  • Own secrets and identity management : Key Vault, service principals and managed identities, secret scopes and rotation.
  • Own access management : Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege , plus access requests and periodic reviews.
  • Own Azure DevOps : repositories, CI/CD pipelines , branch policies and multi-environment promotion , including deployment validation and Terraform state.
  • Track cost and governance of Azure resources: tagging, budgets, right‑sizing and cost attribution (e.g. query tagging).
Hands‑on Data Platform evolution
  • Contribute to the implementation of the Data Catalog and Data Quality Framework and it’s lineage across the platform.
  • Drive observability : pipelines that are observable, debuggable and alert‑driven, with clear ownership of failures.
  • Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.
Technical Leadership
  • Participate in architecture reviews and technical decision-making, stating trade‑offs (performance, cost, reliability, complexity) with data.
  • Define and promote engineering standards : naming conventions, patterns and guidelines, through code reviews and design reviews.
  • Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
  • Build and maintain developer tooling and automation , including AI‑assisted engineering workflows (shared agent instructions, skills, automated PR review).
Support & Operations (Run Team Collaboration)
  • Provide second‑line support to the Run/Operations team for complex production topics.
  • Troubleshoot and resolve platform incidents , failures and performance issues; lead root cause analysis and post‑mortems.
  • Ensure proper handover, documentation and operational readiness of platform changes.
  • Continuously improve runbooks, monitoring and operational processes .
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Platform Engineer
Senior Data Platform Engineer

MARGO • United States

Remote
USD 120,000 - 180,000
Senior Data Engineer - CR 12/11/26
Senior Data Engineer - CR 12/11/26

Margo Group • United States

Remote
USD 120,000 - 170,000
Senior Data Engineer
Senior Data Engineer

MARGO • United States

Remote
USD 110,000 - 170,000
Senior Data Platform Engineer — Azure Lakehouse & Airflow
Senior Data Platform Engineer — Azure Lakehouse & Airflow

Margo Group • United States

Remote
USD 150,000 - 210,000
Senior Data Platform Engineer (Azure, Databricks, Terraform)
Senior Data Platform Engineer (Azure, Databricks, Terraform)

Accesa • United States

Remote
USD 140,000 - 200,000
Senior Data Engineer
Senior Data Engineer

Confidentiel • Pittsburgh

Hybrid
USD 150,000 - 190,000
Sr. Data Engineer, Data Platform
Sr. Data Engineer, Data Platform

Mirion Technologies • United States

On-site
USD 120,000 - 160,000
Data Platform Administration Architect
Data Platform Administration Architect

Eklm • United States

Remote
USD 150,000 - 190,000
Lead Platform Engineer
Lead Platform Engineer

Optomi • St. Louis (MO)

On-site
USD 130,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Peyton Resource Group • Houston (TX)

On-site
USD 120,000 - 150,000