Sr Data Engineer

SFE; United States; Remote; USD 120,000 - 180,000; Part time

Job summary

SFE is seeking a Sr Data Engineer for a remote contract role. You will design and optimize Azure Databricks-based batch pipelines, building bronze/silver/gold data layers and delivering reliable data products for analytics.

The role requires 10+ years of experience with at least 3 years in Databricks development, strong SQL and PySpark skills, and expertise in data governance, CI/CD, and scalable data architectures. Remote work option with enterprise data platforms.

Qualifications

• Strong hands-on experience with Azure Databricks and production-grade batch pipelines.
• Proficient in PySpark/Spark SQL, transformations, window functions, and performance tuning.
• Deep knowledge of Delta Lake, schema evolution, upsert patterns, and data quality controls.

Responsibilities

• Design scalable batch pipelines in Azure Databricks using PySpark and Spark SQL.
• Build bronze, silver, and gold data layers with certified fact/dimension products.
• Implement data quality checks, lineage, and documentation; enable CI/CD deployment.

Skills

Azure Databricks

PySpark

Delta Lake

Medallion Architecture

Data Modeling

Data Quality & Testing

SQL

DevOps

Security/Governance

Azure Data Lake

Job description

Job Title: Sr Data Engineer

Location: Remote

Job type: Contract

Relevant Experience: 10+ Years

Technical / Functional Skills

Overall Experience:

• 8 12 years of overall data engineering experience
• At least 3 years of substantial hands-on Databricks development

Required Skills:

• Azure Databricks Strong hands-on experience building production-grade batch pipelines and lakehouse solutions.
• PySpark / Spark SQL Advanced transformations, joins, window functions, reusable frameworks, debugging, and performance tuning.
• Delta Lake Table design, schema evolution, merge/upsert patterns, optimization, data reliability, and history management.
• Medallion Architecture Practical delivery experience across source-aligned, conformed, and consumption-ready layers.
• Data Modeling Fact/dimension design, conformed dimensions, slowly changing dimensions, aggregates, and semantic-consumption considerations.
• Data Quality & Testing Profiling, rule implementation, reconciliation, unit/integration testing, controls, and defect analysis.
• Azure Ecosystem Working knowledge of Azure Data Lake [remaining text is cut off in the screenshot].
• DevOps Git-based development, code reviews, CI/CD, environment promotion, release controls, and automated deployment.
• Security/Governance Experience implementing least-privilege access and working with catalog, lineage, classification, and row-level security requirements.
• SQL Advanced SQL development and query optimization for large analytical datasets.

Preferred Skills

• Hands-on exposure to Databricks Lakeflow Spark Declarative Pipelines or Delta Live Tables (DLT) and the ability to adopt evolving Databricks-native pipeline standards.
• Experience with Unity Catalog; familiarity with enterprise data governance and data-quality tooling such as Informatica.
• Experience with HR data from Workday, Dayforce, PeopleSoft, recruiting, workforce management, or talent systems.
• Understanding of Power BI consumption patterns, star schemas, refresh needs, aggregates, and performance requirements.
• Experience creating accelerators, reusable engineering templates, metadata-driven pipelines, or automated test frameworks.
• Databricks Data Engineer certification or equivalent cloud/data engineering credential.
• Experience in a blended onshore/offshore Agile delivery model.

Key Technology Stack

Azure | Databricks | PySpark | Spark SQL | Delta Lake | Medallion Architecture | SQL | Azure Data Lake | DevOps | CI/CD | Unity Catalog | Informatica | Power BI | Workday | Dayforce | PeopleSoft

Technical Responsibilities

Mandatory

• Optimize Spark workloads, Delta tables, joins, partitions, file sizes, and cluster usage for performance and cost.
• Apply enterprise security and governance standards including Unity Catalog, permissions, sensitive data handling, and row level access patterns.
• Coordinate with ingestion, data governance, modeling, DevOps, and Power BI teams to define data contracts and acceptance criteria.
• Contribute to CI/CD, source control, automated testing, deployment, monitoring, and production readiness.
• Participate in requirement workshops and explain technical choices to business, HR, analytics, and engineering stakeholders.
• Support knowledge transfer and mentor engineers/analysts.

Generic Managerial Skills:

• Strong oral and written communication skills.
• Ability to collaborate in teams and work independently.

Roles and Responsibilities:

• Design and develop scalable batch pipelines in Azure Databricks using PySpark and Spark SQL.
• Build and maintain bronze, silver, and gold data layers including certified fact/dimension products.
• Translate business metric definitions, rules, filters, granularity, history, and refresh needs into technical designs.
• Create reusable HR domain transformations (retention, hiring, transfers, promotions, reviews, hours, scheduling).
• Implement data quality checks, reconciliation, exception handling, auditability, lineage, and documentation.