Sr. Data Engineer

Socket.dev; Chicago (IL); On-site; USD 140,000 - 190,000; Full time

Job summary

Fortellar is seeking a senior data engineer to own and operate production pipelines that migrate and integrate data across regulated environments. This role focuses on building scalable data pipelines, conformed models, and reliable data delivery.

You will design canonical data models, implement change data capture, and ensure auditability and reconciliation for every migrated record. Collaboration with clients and stakeholders is essential.

Qualifications

• 6+ years in data engineering with 3+ years building production pipelines.
• Deep Delta Lake expertise, including MERGE, CDC, and schema evolution.
• Proficiency in SQL and PySpark, with tested, source-controlled code.
• Hands-on experience with Azure data stack (ADF, ADLS, Spark lakehouse).
• Strong client-facing communication and documentation skills.

Responsibilities

• Build and operate CDC and snapshot ingestion from relational systems.
• Design conformed data models to a single trusted record per entity.
• Implement incremental change detection, deduplication, and idempotent replay.
• Create outbound integrations to enterprise platforms.
• Document business rules with owner tracking.
• Build reconciliation metrics and data quality controls.
• Promote code via source control and CI/CD; run unattended jobs.
• Develop reporting tables, semantic models, and analytics datasets.
• Advise clients and document findings for stakeholders.

Skills

Data engineering

Delta Lake

Change Data Capture

Azure data stack

SQL

PySpark

Data quality controls

Client communication

Tools

Azure Data Factory

Azure Data Lake Storage

Spark

Job description

The Role

Fortellar delivers large scale data migration and integration programs for clients in regulated industries. This role builds the pipelines behind that work: extracting data from legacy systems of record, conforming it into a governed lakehouse, integrating it with modern enterprise business platforms, and extending the same foundation into reporting and analytics.

The data is regulated and frequently sensitive. Accuracy is not negotiable. Every migrated record must reconcile to source, and every transformation must be explainable to an auditor.

This is a hands‑on builder role. You will own pipelines, data modelling, business rules, reconciliation, and reporting.

Key Responsibilities

• Build and operate change data capture and snapshot ingestion from relational systems of record, together with reference and lookup data retrieved through REST and OData interfaces.
• Design conformed, canonical data models that resolve fragmented source tables into a single trusted record per business entity.
• Implement incremental change detection and publishing with deduplication, checkpointing, and idempotent replay.
• Build and operate outbound integration to enterprise business platforms.
• Implement and document transformation logic that encodes business rules, with a named owner recorded for every mapping decision.
• Build reconciliation between source and target at record, control total, and business measure level, and report it as a standing metric.
• Implement data quality controls including constraint enforcement, quarantine of rejected records, and freshness monitoring.
• Promote work through source control, automated tests, and deployment tooling across development, test, and production, designing jobs that run unattended under service identities.
• Build reporting tables, semantic models, and analytics ready datasets on the delivered foundation.
• Advise client stakeholders directly, challenge assumptions with evidence, and produce documentation that clients rely on.

Required Qualifications

• 6+ years in data engineering, including at least three years building production pipelines.
• Deep practical command of Delta Lake, including MERGE behavior and cost, Change Data Feed, optimization, and schema evolution.
• Experience with change data capture ingestion from relational sources, including late, duplicate, and out of order events.
• Hands‑on experience with the Microsoft Azure data stack, including Azure Data Factory, Azure Data Lake Storage, and a Spark-based lakehouse compute platform.
• Advanced SQL and strong PySpark, writing tested, reviewable, source‑controlled code rather than notebook scripts.
• Experience delivering system to system integration with guaranteed delivery semantics, covering idempotency, deduplication, checkpointing, and dead letter handling.
• Experience with environment-promotion deployment tooling for lakehouse/Spark workloads (bundle- or CI/CD-based) and promotion across environments.
• Experience implementing data quality and reconciliation controls in an environment subject to external audit.
• Direct client stakeholder experience and excellent written communication.