Data Engineer — Remote, Pipelines & PySpark Expert

Nix; Boston (MA); Hybrid; USD 120,000 - 170,000; Full time

Benefits offered by this job

Flexible working format

Competitive compensation

Career growth

Professional development tools

Job summary

N-iX is seeking a Middle Data Engineer for a six-month engagement to act as the technical owner of data readiness, combining pipeline and query engineering with validation of source, mappings, and outputs. The role involves Snowflake extractions through Databricks, with Blob Storage, validation, Medallion transformations, Power BI review, and Gold delivery to stakeholders.

You will coordinate validation cycles and drive end-to-end incident resolutions.

Qualifications

• 5–6 years' professional engineering experience with independent production ownership.
• Strong Python, PySpark, and SQL, including complex transformations, query optimization, reusable packages, debugging, tests, and reconciliation.
• Strong ETL/ELT and batch-pipeline engineering across relational warehouses, APIs, object storage, and file-based ingestion.
• Experience with a cloud data lake/lakehouse, Medallion patterns, schema/interface contracts, data-quality controls, orchestration, observability, and incident recovery.
• Solid Git engineering practices: branching, pull requests, reviews, automated tests, deployment controls, and documentation.
• Proven ability to translate business/data requirements into filters, mappings, transformations, validation rules, and operational workflows.
• Stakeholder-facing problem solving: explain discrepancies, challenge incomplete requirements, establish technical owners, and drive issues to closure.

Responsibilities

• Independently design, develop, maintain, test, deploy, and optimize ingestion, validation, transformation, aggregation, data-quality, anomaly-detection, and extraction workflows.
• Develop and tune Python/PySpark packages, SQL extraction queries, Databricks jobs, ADF pipelines, Delta tables, and source-to-target contracts.
• Own technical readiness for new cells and refreshes: clarify filters and expected scope, implement or update queries, review mappings/configuration, reconcile outputs, and confirm readiness for business validation.
• Investigate source, NCID, taxonomy, hierarchy, mapping, naming-convention, aggregation, missing-week, outlier, and performance issues across CDF, MDFA/PFME, APIs, manual files, and specific sources.
• Coordinate technical validation cycles with Product, CMIA/markets, WPP/MDFA, CDF, other data providers, and Ekimetrics; convert reported discrepancies into actionable owners and technical evidence.
• Drive complex incident resolution across pipelines, data contracts, access, service principals, secrets, networking, Power BI refreshes, and downstream delivery.
• Review pull requests and test evidence; guide the Junior engineer and delegate scoped engineering/support work without becoming a people manager.
• Maintain architecture documentation, interface contracts, repository documentation, runbooks, deployment procedures, and support/escalation guidance.
• Recommend practical automation, reliability, performance, and maintainability improvements while respecting platform standards and business-validation ownership.

Skills

Python

PySpark

SQL

ETL/ELT

Data pipelines

Git

Stakeholder

Problem solving

Tools

Azure Data Factory

Azure Databricks

Delta Lake

Snowflake

Azure Blob Storage

Power BI

GitHub

Azure Key Vault

Job description

N-iX is seeking a Middle Data Engineer for a six-month engagement to act as the technical owner of data readiness, combining pipeline and query engineering with validation of source, mappings, and outputs. The role involves Snowflake extractions through Databricks, with Blob Storage, validation, Medallion transformations, Power BI review, and Gold delivery to stakeholders.

You will coordinate validation cycles and drive end-to-end incident resolutions.