Data Solutions Engineer

Jobtailor; Durham (NC); On-site; USD 110,000 - 160,000; Full time

Job summary

Jobtailor in Durham, NC seeks a Data Engineer to design and operate end-to-end ELT pipelines using Databricks and Azure Data Factory. You will implement data quality checks, build analytics-ready Delta tables, and enforce governance patterns across pipelines.

The role requires 2+ years on Azure Databricks, SQL, PySpark, and data modeling, with strong collaboration and on-call readiness. US work authorization is required; sponsorship is not offered.

Qualifications

• 2+ years hands-on data engineering or equivalent software data work.
• 2+ years building pipelines on Azure and Databricks.
• Strong SQL, PySpark, and data modeling fundamentals.
• Proficiency with SQL and Python (PySpark) on large datasets.
• Experience with Azure Databricks, Delta Lake, DLT, and Azure Data Factory.
• Governance, lineage, and security knowledge (Unity Catalog/Purview).
• Familiarity with data ingestion patterns and CDC patterns.
• Production-grade ops: monitoring, alerts, runbooks.
• Excellent communication and collaboration skills.
• Authorized to work in the United States; sponsorship not required.

Responsibilities

• Design, implement, and support end-to-end ELT pipelines (ingest → transform → publish) in Databricks/ADF
• Implement data quality checks with alerting and remediation runbooks
• Build curated, analytics-ready Delta tables using dimensional modeling
• Implement CDC and deletion-flag patterns; manage schema drift and partitioning
• Operate jobs with monitoring, logging, alerting; on-call rotation
• Partner with Data Architect to align designs with governance, security, and cost
• Document pipelines, data contracts, and SLAs; improve performance and reliability

Skills

SQL

PySpark

Data Modeling

Azure Databricks

Delta Lake

DLT

Python

CI/CD

Power BI

On-call

Education

Bachelor’s degree in MIS, Computer Science, Engineering

Tools

Azure Data Factory

Delta Live Tables

Azure Databricks

Unity Catalog

Purview

Git

Job description

Responsibilities

• Design, implement, and support end‑to‑end ELT pipelines (ingest → transform → publish) in Databricks/ADF
• Implement data quality checks (DLT expectations, unit tests) with alerting and remediation runbooks
• Build curated, analytics‑ready Delta tables using dimensional modeling for consumption by BI Developers
• Implement CDC and deletion‑flag patterns; manage schema drift and partitioning/Z‑Ordering strategies
• Operationalize jobs with monitoring, logging, alerting; participate in an on‑call rotation as needed
• Partner with the Data Architect to align designs with standards for governance, security, and cost efficiency
• Document pipelines, data contracts, and SLAs; continuously improve performance and reliability

Requirements

• 2+ years of hands‑on data engineering (or comparable software engineering with significant data work)
• 2+ years building pipelines on Azure and Databricks (or equivalent cloud + Spark)
• Strong SQL (analytical queries, window functions), PySpark/Spark SQL, and data modeling fundamentals
• Bachelor’s degree in MIS, Computer Science, Engineering, or equivalent experience
• Proficiency with SQL and Python (PySpark), including performance tuning on large datasets
• Experience with Azure Databricks, Delta Lake, Delta Live Tables (DLT), Azure Data Factory (or Fabric Data Pipelines), ADLS Gen2, and Azure DevOps/Git for CI/CD
• Working knowledge of Unity Catalog and/or Microsoft Purview for governance, lineage, and security
• Familiarity with data ingestion patterns (files, APIs, JDBC), schema evolution, CDC, and deletion detection patterns
• Understanding of dimensional modeling to produce analytics‑ready datasets for Power BI
• Exposure to orchestration/monitoring, cost optimization, alerting, and runbook‑driven operations
• Data pipeline design (batch & streaming), DLT expectations for data quality, and robust error handling
• Source control, branching strategies, and CI/CD for data assets (notebooks, jobs, workflows)
• Practical understanding of privacy, security, and RBAC in cloud data platforms
• Excellent communication, documentation, and cross‑functional collaboration skills
• Analytical mindset; bias toward automation and measurable reliability
• Applicants must be legally authorized to work in the United States and should not require now, or in the future, sponsorship for employment visa sponsorship.