Data Engineer

ness; United States; On-site; USD 120,000 - 160,000; Full time

Job summary

ness is seeking a seasoned Databricks engineer to design and operate a multi-layer data platform (bronze, silver, gold) with strict governance. You will implement ingestion from diverse sources, align to shared contracts, and ensure quality across end-to-end pipelines using Unity Catalog and advanced Delta Lake features.

You will own ETL from REST APIs, optimize Spark workloads, and implement CI/CD for Databricks projects within a multi-cloud data lake environment.

Qualifications

• Advanced Databricks engineering with medallion architecture, Delta Lake and Workflows.
• Unity Catalog governance including schemas, permissions and lineage.
• Strong Python, PySpark and SQL with notebook development.
• Ingestion from REST APIs with pagination, throttling and watermarks.
• Cloud storage across AWS, Azure and GCP for data movement and staging.
• Spark performance optimization focusing on partitioning and resource tuning.
• CI/CD for Databricks and Git-based development workflows.

Responsibilities

• Build ingestion into bronze layer for sources: gateway, observability logs, admin APIs, SaaS usage, billing exports; land raw and untransformed.
• Align ingestion with shared bronze landing contract to avoid duplication.
• Create the silver layer: typed, deduplicated, conformed to canonical dimensions.
• Develop gold marts with attribution, cost basis and provisional status.
• Implement attribution and allocation logic including precedence and ratio-based splitting.
• Operate within Unity Catalog governance with owned datasets, permissions, lineage and cataloging.
• Apply data quality rules and monitoring for completeness, freshness, and tag coverage.
• Manage impact of caller-identity data on cost/usage reporting and downstream volumes.
• Work to per-source cadence (daily or monthly) within CI/CD and promotion practices.

Skills

Databricks engineering

Unity Catalog governance

Python & PySpark

SQL

REST API ingestion

Cloud object storage

CI/CD for Databricks

Git-based workflows

Incremental ingestion patterns

Tools

Delta Lake

Auto Loader

Databricks Workflows

Unity Catalog

Job description

Key responsibilities

• Build ingestion into the bronze layer for assigned sources: gateway and observability logs, productivity
  tool admin APIs, AI-enabled SaaS usage, hyperscaler billing exports and reference data. Land raw and
  untransformed, on a scheduled refresh, replayable if the downstream design changes.
• Work to the shared bronze landing contract so each tool is ingested once and serves both this program
  and the parallel productivity initiative, rather than being integrated twice.
• Build the silver layer: typed, deduplicated and conformed to the canonical dimensions, refreshed
  independently of any downstream publication schedule.
• Build gold marts carrying attribution method, attribution level, cost basis and provisional status alongside
  cost and usage.
• Implement the attribution and allocation logic designed by the analysts, including precedence resolution
  and ratio-based splitting of shared endpoint cost.
• Work within Unity Catalog governance — shared bronze and silver, separate gold marts with a recorded
  owner per dataset — including permissions, lineage and cataloging.
• Implement data quality rules and monitoring: completeness, freshness and tag-coverage checks with
  alerting, so pipeline problems surface before they reach a divisional invoice.
• Manage the volume impact of enabling caller-identity data in the cost and usage report, which multiplies
  row counts by the number of calling identities per model.
• Work to the per-source cadence — daily where controls and anomaly detection depend on it, monthly
  where they do not — within the team's existing CI/CD and promotion practices.

Essential skills and experience

• Advanced Databricks engineering: Delta Lake, medallion architecture, Databricks Workflows, Auto
  Loader and incremental ingestion patterns.
• Unity Catalog to a governance standard — catalogs, schemas, permissions, lineage — not merely as a
  place tables happen to live.
• Strong Python and PySpark, and strong SQL. Notebook-based development.
• Ingestion from REST APIs including pagination, throttling, incremental watermarks and credential
  handling, plus cloud object storage across AWS, Azure and GCP.
• Performance and cost optimization of Spark workloads: partitioning, clustering, file sizing and cluster
  configuration.
• CI/CD for Databricks — asset bundles or equivalent — and Git-based development workflow.
• Able to work to an existing catalog structure and coding standard rather than introducing a parallel
  approach.