Senior Data Platform Engineer - Fabric + Delta Lake

Aperia; Dallas (TX); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Health insurance

Health savings account

Vision insurance

401(k) matching

Life insurance

Paid time off

Parental leave

Disability insurance

Fitness membership

Volunteer time off

Job summary

Aperia Solutions in Dallas is building a next-generation risk analysis and scoring platform on Microsoft Fabric. This role owns its medallion data architecture end-to-end, designing Bronze/Silver/Gold Delta tables, Fabric Spark feature pipelines, Direct Lake semantic models, and physical-layout optimizations.

This is our highest-utilization engineering role on the project, collaborating with senior tech leads and DevOps engineers.

Qualifications

• 5+ years data engineering experience, with at least 2 years hands-on Spark development (PySpark preferred).
• Parquet format expertise — footers, statistics allowlists, row-group sizing, dictionary encoding, column indexes.
• Microsoft Fabric hands-on experience — workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, SQL analytics endpoint.
• SQL performance tuning skills for both OLAP (Fabric) and OLTP (PostgreSQL) workloads.
• Ability to communicate architecture clearly in writing and to work autonomously on well-scoped features.

Responsibilities

• Design and implement Bronze, canonical Silver, and Gold Delta table schemas with PAN-safe physical layout — statistics allowlist configuration, liquid clustering, deliberate retention settings, V-Order optimization.
• Build Fabric Spark notebooks for the feature pipeline (rolling velocity features across 24h / 7d / 30d windows, merchant-level rolling features, hierarchical rollups, cross-merchant features via pan_token joins).
• Author per-flavor-version Silver adapters — one function per flavor, unioned by name, tested against golden fixtures.
• Build the canonical MERGE DAG (Bronze → canonical Silver, idempotent on _natural_key per Aperia's canonical Fabric internal spec v1.4).
• Build the Gold materialization pipeline for both grains (fact_transaction_risk, fact_merchant_risk) with correct score-source stamping and idempotent merge semantics.
• Design and implement Direct Lake semantic models per reporting surface; performance-test against representative data volumes.
• Configure OneLake namespace layouts, workspace catalogs, and cross-workspace shortcut relationships with the residuals programme.
• Establish idempotent Delta write patterns using delta-rs from Python containers (segmented parallel writes with single atomic commit).
• Fabric capacity sizing (F-SKU selection, load testing, Reservation vs PAYG decisions).
• Partner with the DevOps engineer on Fabric IaC (Terraform capacity provisioning plus Fabric REST workspace bootstrap).
• Consult on ClickHouse contingency if Direct Lake latency proves insufficient for client-tier UIs.

Skills

Data engineering

Spark (PySpark)

Parquet expertise

SQL performance tuning

Microsoft Fabric

Python

ETL pipelines

Tools

PostgreSQL

ClickHouse

Job description

Aperia Solutions in Dallas is building a next-generation risk analysis and scoring platform on Microsoft Fabric. This role owns its medallion data architecture end-to-end, designing Bronze/Silver/Gold Delta tables, Fabric Spark feature pipelines, Direct Lake semantic models, and physical-layout optimizations.

This is our highest-utilization engineering role on the project, collaborating with senior tech leads and DevOps engineers.