Data Engineer: Scalable Pipelines on Microsoft Fabric

Aptean; Alpharetta (GA); On-site; USD 95,000 - 125,000; Full time

Benefits offered by this job

Modern data technologies

Collaborative environment

Growth opportunities

Job summary

Aptean is seeking a hands-on Data Engineer to build and maintain scalable data ingestion and transformation pipelines using Microsoft Fabric and Azure. You will work with Python, PySpark, and Delta Lake patterns to support enterprise reporting, analytics, and AI use cases.

Collaborate with engineers, analysts, and business stakeholders to deliver reliable data solutions, apply data quality checks, and participate in CI/CD.

Qualifications

• 3-5 years of experience building data pipelines in production or near-production environments.
• Hands-on experience with Python and PySpark.
• Experience with Azure Data Factory or similar orchestration tools.
• Familiarity with Microsoft Fabric (or strong experience in Azure Synapse/Databricks with willingness to learn Fabric).
• Solid understanding of SQL, data modeling, and performance optimization.
• Knowledge of medallion architecture and Delta Lake concepts.
• Experience implementing data quality checks and basic SCD handling.
• Familiarity with Git, branching, and CI/CD practices (Azure DevOps or GitHub Actions).
• Exposure to Power BI and semantic data models is a plus.
• Understanding of data governance concepts such as schema versioning and access controls preferred.
• Strong problem-solving skills, attention to detail, and ownership mindset.
• Effective communication skills and ability to collaborate with technical and business teams.

Responsibilities

• Build and maintain data ingestion pipelines using Microsoft Fabric (Pipelines, Dataflows Gen2, Notebooks) and Azure Data Factory.
• Develop Python and PySpark notebooks to cleanse, transform, and enrich data across medallion architecture layers.
• Implement Delta Lake patterns including partitioning, schema evolution, and incremental loads.
• Apply data quality checks including null handling, duplicate detection, schema validation, and SCD logic.
• Develop reusable, metadata-driven, and parameterized pipeline frameworks.
• Troubleshoot pipeline failures, performance issues, and data discrepancies; perform root-cause analysis.
• Participate in code reviews, testing, and CI/CD processes using Git-based workflows.
• Support gold-layer datasets and semantic models for Power BI consumption.
• Monitor pipelines and resolve operational issues within defined SLAs; support on-call rotations as needed.
• Maintain technical documentation such as data dictionaries, mappings, and runbooks.
• Collaborate cross-functionally to deliver high-quality data engineering solutions.

Skills

Python

PySpark

SQL

Data modeling

Data quality checks

Git / CI/CD

Power BI awareness

Communication

Tools

Azure Data Factory

Microsoft Fabric

Azure Synapse/Databricks

Job description

Aptean is seeking a hands-on Data Engineer to build and maintain scalable data ingestion and transformation pipelines using Microsoft Fabric and Azure. You will work with Python, PySpark, and Delta Lake patterns to support enterprise reporting, analytics, and AI use cases.

Collaborate with engineers, analysts, and business stakeholders to deliver reliable data solutions, apply data quality checks, and participate in CI/CD.