[C3A] Senior Data Platform Engineer (Microsoft Fabric)

Software Mind; Województwo mazowieckie; Remote; PLN 180,000 - 320,000; Full time

Benefits offered by this job

Private healthcare

Life insurance

Multisport card

Language classes

Knowledge sharing

Job summary

Software Mind is seeking an experienced data engineer to join a project building a risk analysis platform on Microsoft Fabric. You will design and implement data pipelines, Delta Lake tables, and semantic models, collaborating with DevOps and senior engineers.

The role emphasizes autonomy, strong Python skills, and deep knowledge of Spark, Parquet, and SQL tuning for both Fabric and PostgreSQL workloads. Remote-friendly, international projects, and opportunities to travel.

Qualifications

• 5+ years of data engineering experience.
• 2+ years with Spark (PySpark preferred).
• Deep Delta Lake knowledge: log, retention, VACUUM, OPTIMIZE.
• Microsoft Fabric hands-on: workspaces, notebooks, lakehouses.

Responsibilities

• Design Bronze/Silver/Gold Delta schemas with optimized layout.
• Build Fabric Spark notebooks for feature pipelines.
• Create per-flavor Silver adapters and canonical MERGE DAGs.
• Develop Gold materialization for fact tables and scoring.
• Build Direct Lake semantic models; test performance.
• Configure OneLake namespaces and cross-workspace relationships.
• Implement idempotent Delta write patterns using delta-rs.
• Size Fabric capacity; assist Terraform IaC and REST bootstrap.
• Advise on ClickHouse contingency if latency is high.

Skills

PySpark

Python

Delta Lake

Microsoft Fabric

Parquet

SQL tuning

Pandas

PyArrow

Delta-rs

Data modeling

Tools

Terraform

Azure

ClickHouse

Job description

Software Mind develops solutions that make an impact for companies around the globe. Tech giants & unicorns, transformative projects, emerging technologies and limitless opportunities - these are a few words that describe an average day for us. Building cross-functional engineering teams that take ownership and crave more means we're always on the lookout for talented people who bring passion and creativity to every project. Our culture embraces openness, acts with respect, shows grit & guts and combines cooperation with enjoyment.

Project - the aim you'll have

Client is building a next-generation risk analysis and scoring platform on Microsoft Fabric, and this role owns its medallion data architecture end-to-end. You'll design and build Bronze / Silver / Gold Delta tables, Fabric Spark feature pipelines, Direct Lake semantic models, and physical-layout optimizations. This is our highest-utilization engineering role on the project - central to both Phase 1 platform foundation and Phase 2 risk product delivery, working alongside a Senior Technical Lead, Senior DevOps engineer, and mid-level developer.

Role - how you'll contribute

• Design and implement Bronze, canonical Silver, and Gold Delta table schemas with PAN-safe physical layout - statistics allowlist configuration, liquid clustering, deliberate retention settings, V-Order optimization.
• Build Fabric Spark notebooks for the feature pipeline (rolling velocity features across 24h / 7d / 30d windows, merchant-level rolling features, hierarchical rollups, cross-merchant features via pan_token joins).
• Author per-flavor-version Silver adapters - one function per flavor, unioned by name, tested against golden fixtures.
• Build the canonical MERGE DAG (Bronze canonical Silver, idempotent on _natural_key per Aperia's canonical Fabric internal spec v1.4).
• Build the Gold materialization pipeline for both grains (fact_transaction_risk, fact_merchant_risk) with correct score-source stamping and idempotent merge semantics.
• Design and implement Direct Lake semantic models per reporting surface; performance-test against representative data volumes.
• Configure OneLake namespace layouts, workspace catalogs, and cross-workspace shortcut relationships with the residuals programme.
• Establish idempotent Delta write patterns using delta-rs from Python containers (segmented parallel writes with single atomic commit).
• Fabric capacity sizing (F-SKU selection, load testing, Reservation vs PAYG decisions).
• Partner with the DevOps engineer on Fabric IaC (Terraform capacity provisioning plus Fabric REST workspace bootstrap).
• Consult on ClickHouse contingency if Direct Lake latency proves insufficient for client-tier UIs.

Expectations - the experience you need

• 5+ years data engineering experience, with at least 2 years hands-on Spark development (PySpark preferred).
• Deep Delta Lake internals knowledge - transaction log format, checkpoints, retention settings (logRetentionDuration, deletedFileRetentionDuration), VACUUM, OPTIMIZE, deletion vectors, statistics collection.
• Parquet format expertise - footers, statistics allowlists, row-group sizing, dictionary encoding, column indexes.
• Microsoft Fabric hands-on experience - workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, SQL analytics endpoint.
• SQL performance tuning skills for both OLAP (Fabric) and OLTP (PostgreSQL) workloads.
• Strong Python data engineering skills - pandas, PyArrow, delta-rs.
• Ability to communicate architecture clearly in writing and to work autonomously on well-scoped features.

Additional skills - the edge you have

• PCI-scoped data platform experience.
• Financial services or payments domain background.
• Power BI semantic model authoring (DAX).
• Familiarity with published data-standard governance patterns - logical model vs physical profile, SCD2 dimensions, natural-key idempotency, extension attribute stores.
• OneLake shortcuts and cross-workspace lakehouse patterns.
• ClickHouse or similar OLAP engine familiarity.
• Terraform for Azure data resources.
• Prior experience with fluid or schema-flexible ingestion patterns.

What we offer:

• Flexible and remote cooperation options
• International projects with leading global clients
• Travels related to international projects
• Non-corporate atmosphere
• Access* to language classes
• Access* to knowledge-sharing initiatives
• Access* to private healthcare and life insurance
• Access* to multisport card

*The availability and terms of individual benefits may vary depending on the chosen form of cooperation.