Senior Data Engineer: Data Pipelines, Orchestration, System Design & A

Applied Research Works India; Kolkata District; Hybrid; INR 1,500,000 - 2,100,000; Full time

Job summary

Applied Research Works India seeks a Senior Data Engineer to design, build, and operate reliable data pipelines and orchestration frameworks, enabling AI-driven products and workflows on top of them.

You will own the data flow from ingestion to consumption, set engineering standards for the team, and work closely with analytics, data science, and ML teams in Kolkata.

Qualifications

• 8-10 years of professional experience in data engineering or a closely related field.
• Strong proficiency in Python and advanced SQL.
• Proven experience building and maintaining production data pipelines at scale.
• Hands-on experience with at least one orchestration tool
• Experience with big data and processing frameworks
• Experience with cloud platforms and their data services.
• Solid understanding of data warehousing and lakehouse concepts, data modeling, and tools.
• Working knowledge of AI/ML workflows and tools, including LLM APIs, embeddings, and vector databases.
• Experience with version control, CI/CD, and infrastructure as code (e.g., Git, Terraform, Docker).
• Strong problem-solving, communication, and stakeholder management skills.

Responsibilities

• Design, build, and operate scalable data pipelines for batch and streaming data.
• Develop ETL/ELT processes from diverse sources.
• Build and manage data orchestration workflows.
• Implement monitoring, logging, and observability for pipeline health.
• Enable AI/ML workflows and integrate AI tools into data pipelines.
• Lead design reviews and mentor junior engineers.
• Translate business requirements into technical solutions and communicate with stakeholders.

Skills

Python

SQL

Data pipelines

Big data processing

Cloud platforms

AI/ML workflows

Education

Bachelor's or Master’s degree in CS/Engineering

Tools

Orchestration tool

Git

Terraform

Docker

CI/CD

Job description

Senior Data Engineer: Data Pipelines, Orchestration, System Design & AI Tooling

Location: Kolkata

Experience: 8-10 years

Employment Type: Full-time

About the Role

We are looking for a Senior Data Engineer to design, build, and operate reliable data pipelines and orchestration frameworks, and to enable AI-driven products and workflows on top of them. You will own the data flow from ingestion to consumption, set engineering standards for the team, and work closely with analytics, data science, and ML teams.

Key Responsibilities

Data pipeline design and development

• Design, build, and maintain scalable batch and streaming pipelines for structured and unstructured data.
• Develop robust ingestion, transformation, and loading (ETL/ELT) processes from diverse sources (databases, APIs, event streams, files).
• AWS services (EC2, ECS, Lambda, API Gateway, SQS etc)
• Optimize pipeline performance, cost, and reliability

Data orchestration

• Build and manage workflows using orchestration tools
• Implement scheduling, dependency management, retries, alerting, and backfill strategies.
• Establish monitoring, logging, and observability for pipeline health.

AI tools and enablement

• Build data foundations for machine learning and generative AI use cases, including feature pipelines and data preparation for model training and inference.
• Integrate AI/LLM tools and APIs into data workflows (e.g., document processing, enrichment, classification, summarization).
• Support retrieval-based AI solutions, including embedding pipelines and vector databases
• Collaborate with data scientists and ML engineers to move models from prototype to production.

Data quality, governance, and leadership

• Implement data quality checks, validation, lineage, and documentation.
• Apply security, privacy, and compliance best practices to data handling.
• Lead design reviews and code reviews, and mentor junior and mid-level engineers.
• Translate business requirements into technical solutions and communicate with stakeholders.

Required Qualifications

• 8-10 years of professional experience in data engineering or a closely related field.
• Strong proficiency in Python and advanced SQL.
• Proven experience building and maintaining production data pipelines at scale.
• Hands-on experience with at least one orchestration tool
• Experience with big data and processing frameworks
• Experience with cloud platforms and their data services.
• Solid understanding of data warehousing and lakehouse concepts, data modeling, and tools.
• Working knowledge of AI/ML workflows and tools, including LLM APIs, embeddings, and vector databases.
• Experience with version control, CI/CD, and infrastructure as code (e.g., Git, Terraform, Docker).
• Strong problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Experience building RAG or agent-based solutions in production.
• Knowledge of data governance, cataloging, and lineage tools.
• Cloud or data platform certifications.
• Experience in healthcare industry.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.