Senior Data Platform Engineer

MARGO

United States

Teletrabalho

USD 120 000 - 180 000

Tempo integral

Há 10 dias

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

MARGO is seeking an senior data platform engineer to own and evolve the orchestration and data platform stack. You will lead Airflow production, Azure resource management via Terraform, and the Databricks/Unity Catalog ecosystem while driving observability and cost governance.

You will participate in architecture decisions, define engineering standards, and mentor teams across data engineers and product owners. Expect hands-on development and cross-functional collaboration.

Qualificações

  • 6+ years of experience in data engineering and platform/infrastructure topics.
  • Strong Python and SQL skills with PySpark and Delta Lake semantics.
  • Hands-on Airflow in production: design, deployment, ops, upgrades, orchestration of Databricks and ADF workloads.

Responsabilidades

  • Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards.
  • Migrate from Azure Data Factory to Airflow and evolve the metadata-driven orchestration framework.

Conhecimentos

Python
SQL
Airflow
Databricks
Terraform
Azure DevOps
Unity Catalog
Delta Lake
PySpark
Data Lakehouse
Observability

Ferramentas

Databricks
Azure Data Factory
Terraform
Airflow deployment

Descrição da oferta de emprego

Key Responsibilities
Orchestration Platform (Apache Airflow & ADF)
  • Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards .
  • Work on the migration from Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework (DAG generation, reusable operators, dependency and scheduling patterns).
  • Ensure Airflow is reliable and observable : SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.
Azure Resource Management
  • Manage Azure resources with Infrastructure as Code (Terraform) : Databricks workspaces, Data Lake Storage , Data Factory, Key Vault , networking and supporting services.
  • Own secrets and identity management : Key Vault, service principals and managed identities, secret scopes and rotation.
  • Own access management : Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege , plus access requests and periodic reviews.
  • Own Azure DevOps : repositories, CI/CD pipelines , branch policies and multi-environment promotion , including deployment validation and Terraform state.
  • Track cost and governance of Azure resources: tagging, budgets, right-sizing and cost attribution (e.g. query tagging).
Hands-on Data Platform evolution
  • Contribute to the implementation of the Data Catalog and Data Quality Framework and it’s lineage across the platform.
  • Drive observability : pipelines that are observable, debuggable and alert-driven, with clear ownership of failures.
  • Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.
Technical Leadership
  • Participate in architecture reviews and technical decision-making, stating trade-offs (performance, cost, reliability, complexity) with data.
  • Define and promote engineering standards : naming conventions, patterns and guidelines, through code reviews and design reviews.
  • Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
  • Build and maintain developer tooling and automation , including AI-assisted engineering workflows (shared agent instructions, skills, automated PR review).
Support & Operations (Run Team Collaboration)
  • Provide second-line support to the Run/Operations team for complex production topics.
  • Troubleshoot and resolve platform incidents , failures and performance issues; lead root cause analysis and post-mortems.
  • Ensure proper handover, documentation and operational readiness of platform changes.
  • Continuously improve runbooks, monitoring and operational processes .
Technical Skills

6+ years of experience in data engineering, with significant time on platform or infrastructure topics

Strong technical expertise with Python and SQL (advanced), including PySpark and Delta Lake semantics

Deep hands-on experience with Apache Airflow in production: DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads

Hands-on experience with Azure resource management:
  • Key Vault, managed identities and service principals, secret management
  • Access management: Azure RBAC, Entra ID, Unity Catalog permissions
  • Azure DevOps (repos, pipelines, service connections, environments) and CI/CD for data platforms
  • ADLS, networking and security basics, Azure Data Factory
  • Infrastructure as Code (Terraform) and multi-environment promotion
  • Hands-on experience with Databricks and Unity Catalog (compute, jobs, governance)
  • Observability and monitoring, alerting and incident management
  • Solid understanding of Data Lakehouse architecture, Medallion architecture and metadata-driven architecture; awareness of cost management and performance
Soft Skills

Strong problem-solving and analytical thinking

Ability to work in cross-functional teams and to influence without authority

Ownership mindset and attention to detail

Good communication skills with both technical and business stakeholders: leads with the recommendation, quantifies, and is transparent about uncertainty

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Data Platform Engineer CR 12/11/26
Senior Data Platform Engineer CR 12/11/26

Margo Group • EUA

Teletrabalho
USD 150 000 - 210 000
Cloud Platform Manager
Cloud Platform Manager

Confidentiel • Pittsburgh

Híbrido
USD 140 000 - 190 000
Senior Data Engineer
Senior Data Engineer

Confidentiel • Pittsburgh

Híbrido
USD 150 000 - 190 000
Data Platform Administration Architect
Data Platform Administration Architect

Eklm • EUA

Teletrabalho
USD 150 000 - 190 000
Senior Azure Databricks Engineer (ESS-424)
Senior Azure Databricks Engineer (ESS-424)

Environmental and Safety Solutions, Inc. • Glendale (AZ)

Presencial
USD 140 000 - 180 000
Sr. Data Engineer, Data Platform
Sr. Data Engineer, Data Platform

Mirion Technologies • EUA

Presencial
USD 120 000 - 160 000
Senior Data Engineer
Senior Data Engineer

Topcoder • California

Presencial
USD 120 000 - 170 000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Híbrido
USD 140 000 - 190 000
Hybrid work model
Senior Data Engineer
Senior Data Engineer

MARGO • EUA

Teletrabalho
USD 110 000 - 170 000
Azure Databricks Admin
Azure Databricks Admin

Nityo Infotech • Connecticut

Presencial
USD 120 000 - 150 000