Principal Architect, EDT Data Platform Engineering

Jobtailor; Missouri; On-site; USD 120,000 - 180,000; Full time

Job summary

Jobtailor seeks a senior data engineer to translate enterprise architecture into scalable solutions on Azure and Databricks Lakehouse. You will own ingestion pipelines, data contracts, and quality checks, collaborating with delivery teams, data stewards, and business reps to deliver analytics-ready models.

You will design Bronze-to-Silver transformations, manage Unity Catalog standards, and align models with the enterprise semantic layer using Power BI and Fabric.

Qualifications

• Bachelor's degree or higher in a computing/IS-related field.
• 8+ years of data engineering, warehousing, analytics platform experience.
• Hands-on Azure and Databricks Lakehouse delivery experience.
• Proficient in PySpark, Python, and SQL for batch/streaming pipelines.
• Experience with CI/CD, IaC, and data contracts/data quality frameworks.

Responsibilities

• Translate enterprise data architecture into scalable engineering solutions on Azure/Databricks Lakehouse.
• Partner with delivery teams, data stewards, and business reps to build governed pipelines and analytics-ready models.
• Own ingestion frameworks and Bronze-to-Silver transformations with data quality checks.

Skills

Communication

Collaboration

Problem Solving

Stakeholder Engagement

Process Optimization

Education

Bachelor's degree in CS/Engineering/IS

Master's in MIS

Master's in MBA

Tools

Azure DevOps

Terraform

PowerShell

Python

SQL

Unity Catalog

Microsoft Fabric

Databricks Delta Live Tables

Azure Data Lake Storage

Azure SQL

Job description

• Translate enterprise data architecture into scalable engineering solutions on the Azure and Databricks Lakehouse platform
• Partner with delivery teams, business representatives, and data stewards to build governed data pipelines, analytics-ready models, and reusable automation
• Build and maintain ingestion frameworks using Azure Data Factory, Databricks, and Spark
• Implement Bronze-to-Silver transformations with data quality checks, schema validation, and data contract rules
• Optimize batch and streaming pipelines for performance and cost
• Own CI/CD pipelines in Azure DevOps and automate project onboarding, including repositories, pipelines, policies, and templates
• Maintain Infrastructure as Code and automation using Terraform, PowerShell, and Python; troubleshoot pipeline failures and permission issues
• Design curated Gold-layer data models, including star and snowflake schemas, fact tables, dimensions, surrogate keys, and slowly changing dimensions
• Align reusable models, KPI definitions, and business logic with the enterprise semantic layer in Power BI, Microsoft Fabric, and downstream tools
• Manage model changes through impact analysis, backward compatibility, regression support, and documented column derivation logic
• Implement Unity Catalog standards, naming conventions, metadata policies, data quality frameworks, and role- and attribute-based access controls
• Translate high-level architecture into low-level designs, reusable templates, and engineering playbooks
• Guide delivery teams through onboarding, debugging, and design reviews
• Create reusable assets and hands-on guidance that build engineering capability and reduce repeat questions
• Assess complex data systems, identify patterns, and recommend practical solutions
• Explain technical concepts clearly to technical and non-technical stakeholders
• Partner with business representatives and data stewards to define requirements, validate metrics and edge cases, and guide solution decisions
• Identify opportunities for process optimization, redesign, and new or updated policies

Requirements

• Bachelor's degree from an accredited university in Computer Science, Engineering, Information Systems, Business, or an equivalent field of study
• Minimum 8 years of experience in data engineering, data warehousing, and data and analytics platforms
• Recent hands‑on delivery experience on Azure and Databricks Lakehouse platforms
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL, Databricks Delta Live Tables, Delta Lake, Spark, Workflows, and Unity Catalog
• Advanced experience with PySpark or Python and SQL, including batch and streaming pipeline design
• Experience with Azure DevOps pipelines, repositories, and policies; PowerShell automation; CI/CD; and Infrastructure as Code
• Experience with schema enforcement, data contracts, data quality frameworks, metadata‑driven design, monitoring, debugging, and performance tuning
• Preferred: Master's degree in Management Information Systems, Business Administration, or a related discipline, or an Azure analytics certification
• Preferred experience with Microsoft Fabric
• Preferred experience within the medical device or another regulated industry

Core Competencies

Demonstrates expertise in building scalable data engineering solutions on Azure and Databricks Lakehouse platforms, with a strong focus on data pipeline optimization, CI/CD automation, and data quality frameworks. Proficient in translating complex data architectures into actionable engineering designs while ensuring compliance with industry standards.

Highest-signal resume keywords

• Azure Data Factory
• Databricks Lakehouse
• PySpark
• CI/CD Automation
• Data Quality Frameworks

Hard Skills

• Data Engineering
• Data Warehousing
• Batch and Streaming Pipeline Design
• Schema Enforcement
• Metadata-Driven Design
• Performance Tuning
• Data Contracts
• Data Quality Checks
• Impact Analysis
• Data Model Design

Soft Skills

• Clear Communication
• Collaboration
• Problem Solving
• Stakeholder Engagement
• Process Optimization

Certifications & Qualifications

• Azure Analytics Certification
• Master's Degree in Management Information Systems
• Master's Degree in Business Administration

Industry Keywords

• Regulated Industry
• Medical Device
• Data Governance
• Analytics-Ready Models
• Enterprise Data Architecture

Tools & Technologies

• Azure DevOps
• Terraform
• PowerShell
• Python
• SQL
• Unity Catalog
• Microsoft Fabric
• Databricks Delta Live Tables
• Azure SQL
• Azure Data Lake Storage