Senior Data Platform Engineer(Microsoft Fabric,MLOps)

Coretek Services; Kondapur; On-site; INR 3,500,000 - 7,000,000; Full time

Job summary

Coretek Services seeks a Senior Engineer, Machine Learning and Data to build and operationalize Microsoft Fabric-first data platforms for enterprise clients. This platform engineering role does not involve building, training, or tuning models; client data science teams own the models.

You will work within a distributed delivery team alongside US-based architects and project managers, delivering end-to-end technical workstreams and ensuring the platform runs unattended, reproducibly, and under

Qualifications

• 5+ years building and operating production data platforms on Microsoft Azure.
• Hands-on Microsoft Fabric experience: workspaces, OneLake, Lakehouse and Warehouse structures, Fabric Notebooks, Fabric RBAC, and deployment pipelines.
• Strong Python and SQL, with production-grade code structured for reuse, testing, and scheduled execution.
• CI/CD applied to data and notebook assets using Azure DevOps or GitHub Actions, including automated unit and integration testing.
• Orchestration and scheduling experience with dependency management and pipeline-level failure visibility, using Fabric Data Pipelines, Azure Data Factory, Airflow, or equivalent.
• Working knowledge of a data quality framework such as Great Expectations or Soda, or equivalent rules-based validation.
• Solid grasp of Azure identity and security: Entra ID, service principals, managed identities, Azure Key Vault, RBAC.
• Understanding of Power BI consumption patterns against lakehouse and warehouse tables, to validate downstream access.
• Clear written English and the ability to produce runbooks and design documentation for client operations.
• Minimum four hours of daily overlap with US Eastern business hours for standups, design reviews, and demonstrations.
• Bachelor's degree in Computer Science, Data Science, Engineering, or a related field.

Responsibilities

• Build Microsoft Fabric workspace structures for production environments, with environment isolation and RBAC mapping.
• Implement governed read access to enterprise data warehouse sources and controlled write-back into data science schemas.
• Develop reusable batch prediction pipeline templates for data extraction, feature prep, validation, model execution, and alerts.
• Build forecasting pipeline patterns with time-series inputs, rolling forecasts, and horizon-based outputs.
• Implement notebooks and platform assets CI/CD with Git integration, automated tests, and production promotion gates.
• Configure orchestration and scheduling with time-based, trigger-based, and manual executions; provide runbooks.
• Implement data quality gates with schema validation, null checks, value range checks, and anomaly detection.
• Ensure unattended execution runs under managed identities with secrets in Azure Key Vault.
• Establish versioning standards for code, configuration, environments, and data references with rollback paths.
• Build monitoring and observability for ETL, pipelines, data quality alerts, and model drift detection.
• Define package installation/pinning standards for Python/R and restrict production dependencies.
• Validate data science output tables are consumable by Power BI and document access patterns.
• Produce operational runbooks, handoff docs, and knowledge transfer materials for client IT and data science teams.

Skills

Python

SQL

Microsoft Fabric

Azure

CI/CD

Education

Bachelor's degree in CS/DS/Engineering

Tools

Fabric Notebooks

Azure DevOps

Power BI

Job description

Coretek is seeking a Senior Engineer, Machine Learning and Data to build and operationalize Microsoft Fabric-first data and machine learning platforms for enterprise clients. This is a platform engineering role, not a data science role. You will not be building, training, or tuning models. Client data science teams own the models. You build the governed environments, pipelines, CI/CD automation, data quality gates, and observability that let those models run in production on a schedule, reproducibly, and under monitoring.

You will work as part of a distributed delivery team alongside US-based architects and project managers, implementing against an agreed architecture and owning significant technical workstreams end to end. Success in this role means the platform runs unattended, reproducibly, and under service identity, with clear runbooks that let the client operate it after handoff.

This is a full time seat on Coretek's Data and AI practice rather than a placement against a single project. Machine learning platform delivery is the focus, and between those engagements you will work across broader Microsoft Fabric and Azure data platform projects. We are hiring for a long career track, and we expect the mix of work to shift further toward generative AI platform delivery over time.

Requirements

Key Responsibilities

• Build Microsoft Fabric workspace structures spanning Sandbox, Dev/Staging, and Production, including environment isolation, workspace naming and ownership, and Fabric RBAC patterns mapped to Entra ID groups.

• Implement governed read access to enterprise data warehouse sources and controlled write-back into data science owned schemas, covering feature tables, model version metadata, model artifact references, prediction outputs, and experiment structures.

• Develop reusable batch prediction pipeline templates covering data extraction, feature preparation, data quality validation, model execution, output validation, table write-back, and alerting.

• Build forecasting pipeline patterns where they diverge from standard batch scoring, including time-series inputs, rolling forecasts, and horizon-based outputs.

• Implement CI/CD for notebooks and platform assets: Git integration for Fabric, branching and pull request standards, automated unit and integration tests, Fabric deployment pipelines, Azure DevOps pipelines where Fabric-native capability falls short, and a single manual approval gate before production promotion.

• Configure orchestration and scheduling across time-based, trigger-based, and manual execution, with DAG-style visibility that surfaces the failed stage.

• Implement data quality gates covering schema validation, null and missing value thresholds, value range checks, and basic distributional anomaly detection, wired to block downstream model execution and raise an alert on failure.

• Configure all unattended execution to run under managed identities or service principals with secrets held in Azure Key Vault. No scheduled or production process may depend on individual user credentials or interactive sessions.

• Establish model, code, environment, and package versioning standards so any production run is traceable to a versioned combination of code, configuration, environment definition, and data reference, with a demonstrable rollback path.

• Build monitoring and observability: compute and job health, ETL and pipeline execution status, data quality alerts, model drift detection and health-check notebooks, alert thresholds, and routing to client-designated channels.

• Define package installation and pinning standards for Python and R, and enforce that only reviewed and approved dependencies reach production environments.

• Validate that data science owned output tables are consumable by Power BI, and document the access pattern.

• Produce operational runbooks, handoff documentation, and knowledge transfer material for client IT and data science teams.

These are the hard requirements. They are engineering requirements, not machine learning requirements.

• 5+ years building and operating production data platforms on Microsoft Azure.

• Hands-on Microsoft Fabric experience: workspaces, OneLake, Lakehouse and Warehouse structures, Fabric Notebooks, Fabric RBAC, and deployment pipelines. Strong Synapse, Databricks, or Azure Data Factory backgrounds are welcome where the candidate has already moved onto Fabric or can credibly show they will.

• Strong Python and SQL, with production-grade code structured for reuse, testing, and scheduled execution rather than exploratory notebooks alone.

• CI/CD applied to data and notebook assets using Azure DevOps or GitHub Actions, including automated unit and integration testing.

• Orchestration and scheduling experience with dependency management and pipeline-level failure visibility, using Fabric Data Pipelines, Azure Data Factory, Airflow, or equivalent.

• Working knowledge of a data quality framework such as Great Expectations, Soda, or an equivalent rules-based validation approach.

• Solid grasp of Azure identity and security: Entra ID, service principals, managed identities, Azure Key Vault, and RBAC. Specifically, experience making scheduled workloads run headless with no user-bound authentication.

• Understanding of Power BI consumption patterns against lakehouse and warehouse tables, sufficient to validate and document downstream access.

• Clear written English and the ability to produce runbooks and design documentation that a client operations team can follow without the author present.

• Minimum four hours of daily overlap with US Eastern business hours for standups, design reviews, and milestone demonstrations.

• Bachelor's degree in Computer Science, Data Science, Engineering, or a related field.

Machine Learning Operationalization

Working-level capability is required here. Deep specialization is not, and a modeling background is not.

• Experience taking at least one machine learning model into scheduled production and keeping it running. The model itself may have been built by someone else.

• Working understanding of model versioning and reproducibility, meaning what it takes to tie a production run back to a specific combination of code, configuration, environment, and data.

• Experience operationalizing batch scoring or forecasting workloads on a schedule, including retry logic, failure handling, and output persistence.

• Familiarity with monitoring model behavior in production, including drift or degradation in output quality.

• Comfort wrapping and executing model code written by others, in Python and ideally R, without needing to change the model logic.

Preferred

• Working knowledge of R in a platform context: R kernels in notebooks, renv for dependency pinning, and wrapping client-provided R workloads for scheduled execution. Client data science teams frequently write in R even when the platform is built in Python.

• Time-series forecasting exposure, including rolling origin evaluation and horizon-based output structures.

• Experimentation platform patterns: experiment configuration, metrics, treatment assignment, matched datasets, and result storage.

• Model drift detection and baseline statistics monitoring in production.

• Hands‑on with MLflow, the Azure Machine Learning model registry, or an equivalent model tracking and registry tool.

• Exposure to generative AI platform work such as retrieval-augmented generation, prompt and version management, or LLM evaluation. The practice mix is moving in this direction.

• Infrastructure as Code with Bicep or Terraform.

• Certifications such as Fabric Data Engineer Associate (DP-700), Fabric Analytics Engineer Associate (DP-600), Azure Data Scientist Associate (DP-100), Azure Data Engineer Associate (DP-203), or DevOps Engineer Expert (AZ-400).

• Consulting or professional services delivery background, working to fixed scope and milestone acceptance.

• Experience in a regulated or security-reviewed environment where third-party and open-source packages require formal approval before production use.