Data Engineer — Databricks, AI-Driven Pipelines

Remote; San Francisco (CA); On-site; USD 130,000 - 190,000; Full time

Job summary

CoreView is seeking a Data Analytics Engineer to own and operate production data pipelines, focusing on usage/adoption analytics within Databricks and Unity Catalog. You will collaborate with Product, Post-Sales, and cross-functional teams to scope new signals and build scalable data products.

You will work with SQL and Python to productionize pipelines, ensure data quality, and extend instrumentation for future metrics, while advancing CI/CD and governance across the data stack.

Qualifications

• 3 to 5 years building production data pipelines.
• Strong SQL and Python for analytics pipelines.
• Experience with Databricks Unity Catalog and CI/CD.

Responsibilities

• Take operational ownership of the usage and adoption pipelines and their alerting.
• Reconcile figures derived from our product-usage tool.
• Separate development and production environments and mature CI/CD so deployment is repeatable and rollback is reliable.
• Build and maintain Bronze-to-Gold pipelines in Databricks, Unity Catalog metric views, and the permissions/row-level-security model behind them.
• Extend automated data quality checks across the underlying gold-layer tables and validate outputs against existing Power BI reports.
• Co-own the scoping of the next set of usage and adoption signals with Product and Post-Sales: prioritising upsell signals and the ROI narrative and confirming what needs new instrumentation.
• Handle the realities of business system data: schema drift, inconsistent field naming, soft deletes, incremental loads
• Lead and be responsible for the consolidation of additional data sources — including support tickets, audit logs, CRM, and other customer-facing systems — while defining feasibility, effort, cost, and implementation approach required to support a future unified data platform.
• Establish documented pipelines, an access model for raw and transformed data, and a working intake process for cross-department data requests.
• Act as a credible point of contact for CoreView's Databricks estate
• Bring automation and AI-assisted tooling into the day-to-day work — for example automated monitoring and anomaly detection on metric movements — as the platform matures.

Skills

SQL

Python

Databricks

ETL/ELT

CI/CD

Data modelling

Git

CRM data

Data governance

Tools

Databricks Unity Catalog

dbt

Power BI

CI/CD pipelines

Job description

CoreView is seeking a Data Analytics Engineer to own and operate production data pipelines, focusing on usage/adoption analytics within Databricks and Unity Catalog. You will collaborate with Product, Post-Sales, and cross-functional teams to scope new signals and build scalable data products.

You will work with SQL and Python to productionize pipelines, ensure data quality, and extend instrumentation for future metrics, while advancing CI/CD and governance across the data stack.