ETL Data Platform Developer

Machine Learning Technologies, LLC; Redmond (WA); On-site; USD 150,000 - 210,000; Full time

Job summary

Machine Learning Technologies, LLC seeks a senior data engineer to design and implement scalable ETL pipelines using Microsoft Fabric and Azure Data Factory in Redmond, WA. You will build metadata-driven Lakehouse architectures and enable cross-application data access for analytics and AI workloads.

You will collaborate with product teams to align data models with business needs, ensure data quality and governance, and optimize cost and performance across storage and compute layers.

Qualifications

• 7+ years of data engineering experience focused on Microsoft Azure stack.
• Experience with Fabric Lakehouse, Dataflows Gen2, Pipelines, Notebooks.
• Ability to design metadata driven architectures and reusable components.
• Experience with structured and semi-structured data formats (JSON, XML, CSV, Parquet).

Responsibilities

• Design and implement ETL pipelines using Microsoft Fabric and Azure Data Factory.
• Build a metadata driven Lakehouse architecture with threaded datasets to support multiple consumption patterns.
• Develop agent specific data lakes and an orchestration layer for an uber agent that can query across agents to answer customer questions.
• Enable interactive data consumption via Power BI, Azure OpenAI, and other analytics and AI tools.
• Ensure data quality, lineage, and governance across all ingestion and transformation processes.
• Collaborate with product teams to understand data needs and deliver scalable solutions.
• Optimize performance and cost across storage and compute layers.

Skills

SQL

Python

PySpark/Scala

Data modeling

Data governance

Executive communication

Tools

Microsoft Fabric

Azure Data Factory

Power BI

Azure Data Lake Storage Gen2

Job description

Key Responsibilities

• Design and implement ETL pipelines using Microsoft Fabric (Dataflows, Pipelines, Lakehouse ,warehouse, sql) and Azure Data Factory.
• Build and maintain a metadata driven Lakehouse architecture with threaded datasets to support multiple consumption patterns.
• Develop agent specific data lakes and an orchestration layer for an uber agent that can query across agents to answer customer questions.
• Enable interactive data consumption via Power BI, Azure OpenAI, and other analytics and AI tools.
• Ensure data quality, lineage, and governance across all ingestion and transformation processes.
• Collaborate with product teams to understand data needs and deliver scalable solutions.
• Optimize performance and cost across storage and compute layers.

Required Qualifications

7+ years of experience in data engineering with a focus on Microsoft Azure Data stack and Fabric technologies.

Strong expertise in:

• Executive written and oral communication
• Microsoft Fabric (Lakehouse, Dataflows Gen2, Pipelines, Notebooks)
• Azure Data Factory, Azure SQL, Azure Data Lake Storage Gen2
• Power BI and/or other visualization tools
• SQL, Python and PySpark/ Scala
• Experience working with structured and semi structured data (JSON, XML, CSV, Parquet).
• Proven ability to build metadata driven architectures and reusable components.
• Familiarity with agent based architectures and conversational AI integration is a plus.
• Strong understanding of data modeling, data governance, and security best practices.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.