Data Platform Engineer

Bridgenext

United States

Remote

USD 140,000 - 190,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Bridgenext is looking for a DataOps engineer in the United States to automate Databricks Jobs/Workflows, DLT, and Unity Catalog at scale. You will design multi‑task graphs, implement CI/CD pipelines, and build templates with standardized logging and governance.

You will also provision resources with Terraform and Databricks SDK, ensure RBAC, and drive data quality checks and auditability across the data platform. This is a remote‑friendly, collaborative role supporting fast‑paced growth.

Qualifications

  • 5–8 years of experience in data engineering or cloud infra roles.
  • Experience with Databricks administration and workflow development.
  • Proficiency in Terraform for IaC and automated provisioning.
  • Strong Python scripting skills for automation.
  • Solid experience with AWS cloud services and CI/CD practices.

Responsibilities

  • Automation & Orchestration: Design, build, and maintain Databricks Workflows/Jobs for batch and SQL workloads.
  • Implement CI/CD pipelines (GitHub Actions/Azure DevOps/GitLab) for IaC and environment promotion.
  • Create templates and governance with idempotent job definitions and logging/notifications.
  • Enforce RBAC templates and integration with service principals and SCIM.
  • Automate provisioning via Terraform (Databricks provider) and REST API 2.1 for Databricks components.
  • Unity Catalog automation, governance, and migration provisioning.
  • Define SLOs for pipelines and drive data quality checks and lineage.

Skills

Databricks
Terraform
AWS
Python
CI/CD
Git
Agentic AI

Tools

Databricks SDK
REST API
GitHub Actions

Job description

Company Overview

At Bridgenext, we engineer Growth Operating Systems. Most enterprises have spent millions on their revenue stack and still aren't seeing the growth they expected.They have tools that function, but no system that wins. We help growth-hungry companies close that gap by turning fragmented platforms, siloed teams, and disconnected data into one integrated Growth OS. More than a technology company or marketing agency, we're a global digital consultancy with experts in engineering, data, AI, creative and more. Our teams are made up of experts who believe in engineering impact, starting with putting people at the center of everything we do. Every team member directly shapes our work, culture, and values. Nothing matters more to us than a kind, respectful, fulfilling environment that supports everyone. Our flexible, inclusive culture gives you the autonomy, resources, and opportunities to thrive.

Position Description

We are looking for DataOps engineer with hands‑on experience to automate our Databricks Jobs/Workflows, DLT, and SQL tasks, implement and operate Unity Catalog at scale.

Key Responsibilities
  • Automation & Orchestration (Jobs/Workflows & CI/CD) Design, build, and maintain Databricks Workflows/Jobs (multi‑task graphs, retries, triggers, alerts) for batch, and SQL workloads.
  • Implement CI/CD pipelines (GitHub Actions/Azure DevOps/GitLab): plan/apply for IaC, validate job JSON, environment promotion, approvals, and rollback.
  • Build templates, parameterized job definitions (ingestion/batch/streaming/SQL) with standardized logging, notifications, and governance.
  • Enforce run‑as service principals, job versioning, change auditability, and artifact immutability.
  • Automate provisioning via Terraform (Databricks provider), Databricks SDK (Python/Go), and REST API 2.1 for Jobs, DLT, Repos, SQL Warehouses, and Unity Catalog.
  • Unity Catalog – Automation, Governance & Migration Catalog & Schema Provisioning (Automation‑first): Idempotent creation of catalogs, schemas, default locations, owners, grants, volumes, storage credentials, and external locations using Terraform/SDK/REST.
  • Grant‑as‑Code: Standardize RBAC with templates (OWNERSHIP, USE CATALOG/SCHEMA, SELECT, MODIFY, CREATE, READ/WRITE FILES); integrate SCIM groups and service principals.
  • Security & Audit: Dynamic views for row/column‑level security, data classification tags, audit logging, token policies, periodic permissions drift detection & remediation.
  • Delta Sharing: Automate providers/recipients, share objects, and scheduled access reviews/expiry.
  • Unity Catalog Migration (HMS → UC or multi‑workspace to UC): discovery, wave planning, external/managed table registration, permission model migration, dual‑run validation (row counts, schema parity, query replay), cutover & rollback.
  • Reliability & Data Quality Define SLOs for pipeline success, latency, and data freshness; drive ≥99% T1 success and low MTTR.
  • Implement data quality checks (DLT expectations, dbt‑expectations/GX), idempotency, schema evolution strategy, checkpointing & replay.
  • Build incident response playbooks; lead RCAs and preventative actions.
Must Have Skills

5–8 years of experience in data engineering, platform engineering, or cloud infrastructure roles. Strong hands‑on experience with Databricks administration (workspace management, cluster policies, access controls, cost optimization). Working knowledge of Databricks workflow development (job orchestration, scheduling, pipeline monitoring). Proficiency in Terraform for infrastructure as code and automated provisioning. Solid experience with AWS cloud services and architecture. Good Python scripting skills for automation and tooling. Good understanding of CI/CD practices and version control (Git). Exposure to or experience implementing Agentic AI solutions, Genie spaces and apps primarily or anything similar.

Professional Skills

Strong problem‑solving skills and ability to work in a collaborative, fast‑paced environment. Excellent English communication skills Bridgenext is an Equal Opportunity Employer #LI-AP1 #LI-REMOTE

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Data Platform & Automation Engineer
Remote Data Platform & Automation Engineer

Bridgenext • United States

Remote
USD 140,000 - 190,000
Principal Engineer - Databricks Data Engineer
Principal Engineer - Databricks Data Engineer

Nagarro • United States

Remote
USD 120,000 - 180,000
Senior Data Engineer, Databricks
Senior Data Engineer, Databricks

GAP • United States

Remote
USD 140,000 - 200,000
Data Engineer
Data Engineer

Bridgenext • United States

Remote
USD 160,000 - 210,000
Remote work
Equal opportunity employer
Data Engineer - Databricks
Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000
Senior Data Platform Engineer
Senior Data Platform Engineer

Arrowhead Pharmaceuticals, Inc. • Los Angeles (CA)

On-site
USD 150,000 - 190,000
Benefits package
Senior Data Platform Engineer
Senior Data Platform Engineer

Arrowhead Pharmaceuticals, Inc. • Madison (WI)

On-site
USD 140,000 - 180,000
Competitive salary
Excellent benefits package
Senior Data Platform Engineer
Senior Data Platform Engineer

Arrowhead Pharmaceuticals, Inc. • San Diego (CA)

On-site
USD 150,000 - 210,000
Excellent benefits
Competitive compensation
Senior Databricks Data Engineer
Senior Databricks Data Engineer

GAP Careers • Austin (TX)

On-site
USD 140,000 - 200,000
Senior Databricks Platform Engineer – Mexico/India (Remote)
Senior Databricks Platform Engineer – Mexico/India (Remote)

Digitalt3 • United States

Remote
USD 150,000 - 210,000