Sr Engineer - DataOps

Tridiagonal Ai; Pune District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Tridiagonal Ai in Pune invites a seasoned data engineer to design scalable pipelines for industrial time-series data. You will ingest data from sensors, historians and IoT devices, build ETL/ELT processes, and optimize schemas for high-velocity data on cloud platforms.

Requires 4–7 years in DataOps or platform engineering, strong Python and SQL, experience with OSI PI, Honeywell PHD, InfluxDB, SAP data interfaces, and modern orchestration and streaming tools such as Airflow, Kafka, Spark, Data

Qualifications

• 4-7 years of experience in data engineering, DataOps, or platform engineering roles
• Strong proficiency in Python and SQL for data pipeline development
• Experience with time-series databases (OSI PI, Honeywell PHD, InfluxDB)
• Familiarity with CDF platform
• Experience with industrial data sources (OPC-UA, MQTT, Modbus, historians)
• Experience with Azure datalake and data warehouse, IIoT services
• Experience in connecting with SAP S/4Hana, ECC PM, MM modules and ingesting batch data through delivery pipelines
• Experience with data pipeline orchestration tools (Apache Airflow, Dagster, Prefect, or Azure Data Factory)
• Proficiency with stream processing frameworks (Kafka, Spark Streaming, Flink, or Azure Event Hubs)
• Experience with data warehousing and data lake solutions (Snowflake, Databricks, Azure Synapse)
• Strong knowledge of Docker and containerization
• Familiarity with infrastructure as code (Terraform, ARM, Bicep) and CI/CD pipelines

Responsibilities

• Design, build, and maintain scalable data pipelines for ingesting industrial time-series data from sensors, historians, and IoT devices
• Develop and operate ETL/ELT processes for batch and streaming data from diverse sources (SAP, CMMS, inspection reports, documents)
• Build and optimize time-series database schemas for high-velocity industrial data (millions of data points per minute)
• Implement data validation, data quality checks, and monitoring for all data pipelines
• Deploy and manage data infrastructure on cloud platforms (Azure preferred, AWS)
• Orchestrate complex data workflows using Airflow, custom connectors or Azure Data Factory
• Collaborate with data scientists and ML engineers to provision data for model training and inference
• Implement data partitioning, sharding, and retention policies for terabyte-scale datasets
• Build and maintain APIs for data serving to downstream applications and AI models
• Ensure data security, encryption, and access controls across all data stores
• Monitor pipeline performance, troubleshoot failures, and optimize for latency and cost
• Document data lineage, data dictionaries, and pipeline architectures

Skills

Python

SQL

ETL/ELT design

Data pipeline orchestration

Collaboration with data scientists

Cloud platforms

Tools

OSI PI

Honeywell PHD

InfluxDB

CDF platform

Azure Data Lake

Azure Synapse

Snowflake

Databricks

Kafka

Spark Streaming

Flink

Azure Event Hubs

Apache Airflow

Dagster

Prefect

Azure Data Factory

Docker

Terraform

ARM

Bicep

CI/CD

Job description

Role & responsibilities

Candidate should have

• 4-7 years of experience in data engineering, DataOps, or platform engineering roles
• Strong proficiency in Python and SQL for data pipeline development
• Experience with time-series databases (OSI PI, Honeywell PHD, InfluxDB)
• Familiarity with CDF platform
• Experience with industrial data sources (OPC-UA, MQTT, Modbus, historians
• Experience with Azure datalake and data warehouse, IIoT services
• Experience in connecting with SAP S/4Hana, ECC PM, MM modules and ingesting batch data through delivery pipelines
• Experience with data pipeline orchestration tools (Apache Airflow, Dagster, Prefect, or Azure Data Factory)
• Proficiency with stream processing frameworks (Kafka, Spark Streaming, Flink, or Azure Event Hubs)
• Experience with data warehousing and data lake solutions (Snowflake, Databricks, Azure Synapse)
• Strong knowledge of Docker and containerization
• Familiarity with infrastructure as code (Terraform, ARM, Bicep) and CI/CD pipelines

Essential Duties and Key Competencies

• Design, build, and maintain scalable data pipelines for ingesting industrial time-series data from sensors, historians, and IoT devices
• Develop and operate ETL/ELT processes for batch and streaming data from diverse sources (SAP, CMMS, inspection reports, documents)
• Build and optimize time-series database schemas for high-velocity industrial data (millions of data points per minute)
• Implement data validation, data quality checks, and monitoring for all data pipelines
• Deploy and manage data infrastructure on cloud platforms (Azure preferred, AWS)
• Orchestrate complex data workflows using Airflow, custom connectors or Azure Data Factory
• Collaborate with data scientists and ML engineers to provision data for model training and inference
• Implement data partitioning, sharding, and retention policies for terabyte-scale datasets
• Build and maintain APIs for data serving to downstream applications and AI models
• Ensure data security, encryption, and access controls across all data stores
• Monitor pipeline performance, troubleshoot failures, and optimize for latency and cost
• Document data lineage, data dictionaries, and pipeline architectures

Additional Skills (Optional)

• Experience with vector databases (Pinecone, Weaviate, Milvus) for RAG applications
• Familiarity with feature stores (Feast, Tecton, Databricks Feature Store)
• Experience with data version control (DVC, LakeFS)
• Knowledge of MLOps practices and model data pipelines
• Experience with industrial protocols (OPC-UA, MQTT, Modbus) and historian systems
• Understanding of data governance and compliance (GDPR, ISO 27001)
• Experience with real-time anomaly detection pipelines

Preferred candidate profile