Senior AI Data Engineer: Own Pipelines & Data Quality

Rpotential; San Francisco (CA); Hybrid; USD 140,000 - 200,000; Full time

Job summary

rPotential is seeking a Sr. AI Product Data Engineer to own and improve our production data pipelines. You will manage Databricks across jobs, Unity Catalog, and cost while collaborating with product and business teams to understand and resolve data issues.

The role requires 5+ years of data engineering experience, strong Python/SQL skills, and the ability to work in a hybrid setup with three in-person days per week in the SF Bay Area.

Qualifications

• 5+ years of data engineering experience with production pipeline ownership.
• Strong Databricks and Unity Catalog experience, including workspace administration.
• Strong Python and SQL.
• Experience improving an existing data environment while it remained live.

Responsibilities

• Own and improve our production data pipelines.
• Standardize pipeline structure, scheduling, retries, testing, monitoring, and backfills.
• Build monitoring around freshness, coverage, failures, and data quality.
• Own Databricks across jobs, Unity Catalog, permissions, environments, and cost.
• Work in our product monorepo alongside the engineering team.
• Work with product and business teams to understand new datasets and resolve data issues.
• Make it easier to take new data sources from prototype to production.

Skills

Data engineering

Python

SQL

Databricks

Unity Catalog

Production pipelines

Data quality

Business judgment

Ambiguous data handling

AI coding tools

Tools

Databricks

Unity Catalog

Postgres

Job description

rPotential is seeking a Sr. AI Product Data Engineer to own and improve our production data pipelines. You will manage Databricks across jobs, Unity Catalog, and cost while collaborating with product and business teams to understand and resolve data issues.

The role requires 5+ years of data engineering experience, strong Python/SQL skills, and the ability to work in a hybrid setup with three in-person days per week in the SF Bay Area.