Senior Data Platform Engineer(Microsoft Fabric, ML Ops)

Coretek Services; Kondapur; On-site; INR 1,800,000 - 3,400,000; Full time

Job summary

Coretek Services is seeking an experienced Data Engineering/MLOps engineer to design and operate Microsoft Fabric-based data and analytics pipelines in Azure. You will build workspace architectures, implement governance, and automate end-to-end batch and forecasting workloads with robust CI/CD and secure credentials.

You will collaborate with clients to ensure data quality, monitoring, and accessible Power BI outputs while maintaining versioned deployables and runbooks for operations teams

Qualifications

• 5+ years in data engineering, machine learning engineering, or MLOps, with production delivery on Microsoft Azure.
• Hands-on Microsoft Fabric experience: workspaces, OneLake, Lakehouse and Warehouse structures, Fabric Notebooks, Fabric RBAC, and deployment pipelines.
• Strong Python and SQL, with production-grade code structured for reuse, testing, and scheduled execution.
• Demonstrated MLOps practice: model versioning, reproducibility, artifact and metadata management, environment pinning, and promotion across environments.
• Experience operationalizing batch scoring or forecasting workloads on a schedule, including retry logic, failure handling, and output persistence.
• CI/CD applied to data and notebook assets using Azure DevOps or GitHub Actions, including automated unit and integration testing.
• Orchestration and scheduling experience with dependency management and pipeline-level failure visibility, using Fabric Data Pipelines, Azure Data Factory, Airflow, or equivalent.
• Working knowledge of a data quality framework such as Great Expectations, Soda, or an equivalent rules-based validation approach.
• Solid grasp of Azure identity and security: Entra ID, service principals, managed identities, Azure Key Vault, and RBAC.
• Understanding of Power BI consumption patterns against lakehouse and warehouse tables, sufficient to validate and document downstream access.
• Clear written English and the ability to produce runbooks and design documentation that a client operations team can follow without the author present.
• Minimum four hours of daily overlap with US Eastern business hours for standups, design reviews, and milestone demonstrations.
• Bachelor’s degree in Computer Science, Data Science, Engineering, or a related field.

Responsibilities

• Build Microsoft Fabric workspace structures spanning Sandbox, Dev/Staging, and Production, including environment isolation, workspace naming and ownership, and Fabric RBAC patterns mapped to Entra ID groups.
• Implement governed read access to enterprise data warehouse sources and controlled write-back into data science owned schemas, covering feature tables, model version metadata, model artifact references, prediction outputs, and experiment structures.
• Develop reusable batch prediction pipeline templates covering data extraction, feature preparation, data quality validation, model execution, output validation, table write-back, and alerting.
• Build forecasting pipeline patterns where they diverge from standard batch scoring, including time-series inputs, rolling forecasts, and horizon-based outputs.
• Configure orchestration and scheduling across time-based, trigger-based, and manual execution, with DAG-style visibility that surfaces the failed stage.
• Implement data quality gates covering schema validation, null and missing value thresholds, value range checks, and basic distributional anomaly detection, wired to block downstream model execution and raise an alert on failure.
• Establish model, code, environment, and package versioning standards so any production run is traceable to a versioned combination of code, configuration, environment definition, and data reference, with a demonstrable rollback path.
• Produce operational runbooks, handoff documentation, and knowledge transfer material for client IT and data science teams.

Skills

Azure

Microsoft Fabric

Python

SQL

MLOps

CI/CD

Airflow

Power BI

RBAC

Entra ID

Education

Bachelor's degree

Tools

Azure DevOps

GitHub Actions

Fabric Notebooks

Synapse/Databricks

Key Vault

Job description

Description

Key Responsibilities

• Build Microsoft Fabric workspace structures spanning Sandbox, Dev/Staging, and Production, including environment isolation, workspace naming and ownership, and Fabric RBAC patterns mapped to Entra ID groups.

• Implement governed read access to enterprise data warehouse sources and controlled write-back into data science owned schemas, covering feature tables, model version metadata, model artifact references, prediction outputs, and experiment structures.

• Develop reusable batch prediction pipeline templates covering data extraction, feature preparation, data quality validation, model execution, output validation, table write-back, and alerting.

• Build forecasting pipeline patterns where they diverge from standard batch scoring, including time-series inputs, rolling forecasts, and horizon-based outputs.

• Implement CI/CD for notebooks and platform assets: Git integration for Fabric, branching and pull request standards, automated unit and integration tests, Fabric deployment pipelines, Azure DevOps pipelines where Fabric-native capability falls short, and a single manual approval gate before production promotion.

• Configure orchestration and scheduling across time-based, trigger-based, and manual execution, with DAG-style visibility that surfaces the failed stage.

• Implement data quality gates covering schema validation, null and missing value thresholds, value range checks, and basic distributional anomaly detection, wired to block downstream model execution and raise an alert on failure.

• Configure all unattended execution to run under managed identities or service principals with secrets held in Azure Key Vault. No scheduled or production process may depend on individual user credentials or interactive sessions.

• Establish model, code, environment, and package versioning standards so any production run is traceable to a versioned combination of code, configuration, environment definition, and data reference, with a demonstrable rollback path.

• Build monitoring and observability: compute and job health, ETL and pipeline execution status, data quality alerts, model drift detection and health-check notebooks, alert thresholds, and routing to client-designated channels.

• Define package installation and pinning standards for Python and R, and enforce that only reviewed and approved dependencies reach production environments.

• Validate that data science owned output tables are consumable by Power BI, and document the access pattern.

• Produce operational runbooks, handoff documentation, and knowledge transfer material for client IT and data science teams.

Required Skills and Experience

• 5+ years in data engineering, machine learning engineering, or MLOps, with production delivery on Microsoft Azure.

• Hands-on Microsoft Fabric experience: workspaces, OneLake, Lakehouse and Warehouse structures, Fabric Notebooks, Fabric RBAC, and deployment pipelines. Deep Synapse or Databricks background with demonstrable Fabric work will be considered.

• Strong Python and SQL, with production-grade code structured for reuse, testing, and scheduled execution rather than exploratory notebooks alone.

• Demonstrated MLOps practice: model versioning, reproducibility, artifact and metadata management, environment pinning, and promotion across environments.

• Experience operationalizing batch scoring or forecasting workloads on a schedule, including retry logic, failure handling, and output persistence.

• CI/CD applied to data and notebook assets using Azure DevOps or GitHub Actions, including automated unit and integration testing.

• Orchestration and scheduling experience with dependency management and pipeline-level failure visibility, using Fabric Data Pipelines, Azure Data Factory, Airflow, or equivalent.

• Working knowledge of a data quality framework such as Great Expectations, Soda, or an equivalent rules-based validation approach.

• Solid grasp of Azure identity and security: Entra ID, service principals, managed identities, Azure Key Vault, and RBAC. Specifically, experience making scheduled workloads run headless with no user-bound authentication.

• Understanding of Power BI consumption patterns against lakehouse and warehouse tables, sufficient to validate and document downstream access.

• Clear written English and the ability to produce runbooks and design documentation that a client operations team can follow without the author present.

• Minimum four hours of daily overlap with US Eastern business hours for standups, design reviews, and milestone demonstrations.

• Bachelor’s degree in Computer Science, Data Science, Engineering, or a related field.

Preferred

• Working knowledge of R in a platform context: R kernels in notebooks, renv for dependency pinning, and wrapping client-provided R workloads for scheduled execution. Client data science teams frequently write in R even when the platform is built in Python.

• Time-series forecasting exposure, including rolling origin evaluation and horizon-based output structures.

• Experimentation platform patterns: experiment configuration, metrics, treatment assignment, matched datasets, and result storage.

• Model drift detection and baseline statistics monitoring in production.

• Infrastructure as Code with Bicep or Terraform.

• Certifications such as Fabric Data Engineer Associate (DP-700), Fabric Analytics Engineer Associate (DP-600), Azure Data Scientist Associate (DP-100), Azure Data Engineer Associate (DP-203), or DevOps Engineer Expert (AZ-400).

• Consulting or professional services delivery background, working to fixed scope and milestone acceptance.

• Experience in a regulated or security-reviewed environment where third-party and open-source packages require formal approval before production use.