Senior Data Integration & Pipeline Engineer

InOpTra Digital; Bengaluru; Remote; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading technology firm in India seeks an experienced Senior Data Integration & Pipeline Engineer to architect and maintain scalable data pipelines. The role requires 5-10 years of experience in data engineering, designing ETL frameworks, and building connectors for SharePoint and PLM systems. Candidates should have strong Python and SQL skills, along with expertise in Apache Airflow. This position is full-time and remote friendly, offering opportunities for innovation in enterprise data management.

Qualifications

• 5–10 years of experience in data engineering, integration engineering, or pipeline development.
• Advanced expertise in ETL/ELT design patterns and data ingestion frameworks.
• Strong debugging and performance optimization skills.

Responsibilities

• Architect, develop, and maintain scalable enterprise data pipelines.
• Manage data storage layers across various cloud environments.
• Develop orchestration workflows using Apache Airflow.

Skills

Data pipeline development

Connector development for SharePoint

ETL/ELT design patterns

Apache Airflow

Python programming

SQL proficiency

Cloud platforms understanding (Azure/AWS/GCP)

Data governance knowledge

Tools

Apache Airflow

CI/CD tools

Collibra

Databricks

Job description

Senior Data Integration & Pipeline Engineer

InOpTra Digital

Experience: 5–10 years

Location: Remote

Employment Type: Full-Time

Position Overview

The Senior Data Integration & Pipeline Engineer is responsible for architecting, developing, and maintaining scalable, secure, and highly reliable enterprise data pipelines across diverse data stores and file systems. This role requires deep expertise in building custom connectors (SharePoint, PLM, SolidWorks), implementing ETL/ELT frameworks, handling large unstructured datasets through chunking and cataloguing, and ensuring efficient orchestration using tools such as Apache Airflow. The engineer will oversee storage management, operational reliability, and end‑to‑end data flow performance across enterprise environments.

Key Responsibilities

Connector Development

• Design, develop, and maintain robust connectors and integrations for Microsoft SharePoint (online and on‑prem), PLM systems (Teamcenter, Windchill, or equivalent), and SolidWorks / SOLIDWORKS PDM.
• Build secure, scalable APIs and ingestion frameworks to extract structured and unstructured data from disparate engineering and enterprise systems.
• Implement metadata extraction, incremental fetch logic, and schema harmonization across connectors.

Data Pipeline Engineering (ETL/ELT)

• Architect, implement, and optimize ETL/ELT pipelines for ingestion, transformation, cataloguing, and distribution of large‑volume datasets.
• Apply chunking, partitioning, and parallelization strategies for large binary, CAD, or document repositories.
• Build and maintain enterprise catalogues enabling discoverability, lineage tracking, and auditability.

Enterprise Storage & Data Management

• Manage data storage layers across object stores, NAS/SAN, cloud storage (Azure, AWS, GCP), and hybrid environments.
• Implement performance tuning, lifecycle policies, tiering, and secure data handling.
• Ensure compliance with enterprise data governance, retention, and classification standards.

Orchestration & Automation

• Develop and maintain orchestration workflows using Apache Airflow or equivalent orchestration frameworks.
• Build DAGs, automate pipeline scheduling, implement dependency management, and handle pipeline‑level exception and retry logic.
• Maintain CI/CD workflows for data pipelines and connector deployments.

Reliability Engineering & Operations

• Own the reliability, monitoring, and observability of all data pipelines.
• Implement automated alerting, health checks, and failure recovery mechanisms.
• Drive SLA/SLO compliance for ingestion and transformation pipelines.
• Troubleshoot performance bottlenecks, failures, and data quality issues.

Cross‑Enterprise Data Support

• Work with all major enterprise data stores and file systems including relational databases, NoSQL stores, data lakes, object storage, and distributed file systems.
• Collaborate with application, infrastructure, and data governance teams to ensure seamless integration and operational readiness.
• Provide technical leadership on data movement best practices, data modelling impacts, and architectural decisions.

Required Skills & Experience

• 5–10 years of experience in data engineering, integration engineering, or pipeline development in enterprise environments.
• Strong experience building connectors for SharePoint, PLM systems, and SolidWorks or engineering applications.
• Advanced expertise in ETL/ELT design patterns, data ingestion frameworks, and handling large unstructured datasets.
• Proven ability with chunking, partitioning, cataloguing, metadata management, and indexing strategies.
• Hands‑on experience with orchestration platforms such as Apache Airflow (DAG design, scheduling, reliability tuning).
• Proficiency with Python, SQL, shell scripting, and API integration frameworks.
• Experience across enterprise data stores and file systems (RDBMS, NoSQL, object storage, file shares, distributed file systems).
• Strong understanding of cloud platforms (Azure/AWS/GCP) and hybrid data architectures.
• Experience with DevOps for data (CI/CD pipelines, version control, containerization).
• Strong debugging, performance optimization, and reliability engineering capabilities.

Preferred Qualifications

• Experience with data cataloguing platforms (Collibra, Alation, custom catalogues).
• Familiarity with data lakehouse ecosystems (Databricks, Snowflake, Synapse).
• Background in engineering system integrations (CAD, PLM, ERP).
• Knowledge of data governance, security, and compliance frameworks.
• Certification in cloud data engineering (Azure DP‑203, AWS Data Analytics, GCP Data Engineer).

Personal Attributes

• Strong analytical, troubleshooting, and problem‑solving abilities.
• Ability to work autonomously while collaborating effectively with cross‑functional teams.
• High attention to detail with a focus on reliability and long‑term sustainability of systems.
• Excellent communication and documentation skills.

Seniority level: Mid‑Senior level