Senior Data / ML Engineer

Web Spiders; Kolkata Metropolitan Area; On-site; INR 2,800,000 - 4,200,000; Full time

Job summary

Web Spiders in Kolkata (Rajarhat–Newtown) is seeking a Senior Data/ML Engineer to design, build, and maintain scalable data pipelines on AWS. You will collaborate with data scientists and product teams to deliver reliable data delivery and analytics.

The role requires deep AWS expertise, AI-assisted coding tools, and infrastructure-as-code practices. Office-based with US EST timing; AWS data analytics focus and data governance experience are a plus.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.
• 5+ years of professional experience in data engineering with at least 3 years of hands-on AWS production workloads.
• 3+ years of experience with AWS Services: EMR (Spark/Hadoop), Apache Airflow, S3, Redshift, RDS, Lambda, Sagemaker and ECS.
• Solid experience with Docker (building, optimizing, and deploying containers) and container orchestration.
• Proficiency in Python and SQL; experience with Typescript is a plus.

Responsibilities

• Design, develop, and optimize large-scale ETL/ELT pipelines using AWS services (EMR Spark, Glue, Lambda, Step Functions).
• Orchestrate complex data workflows with Apache Airflow (MWAA or self-managed).
• Architect and manage data storage across S3, Redshift, RDS with best practices for partitioning and cost optimization.
• Build containerized data applications with Docker and ECS/Fargate, including CI/CD automation.

Skills

Python

SQL

AWS

Airflow

Docker

Terraform

Education

Bachelor's or Master's in CS/Engineering/Data Science

Tools

EMR (Spark)

S3

Redshift

RDS

ECS/Fargate

Terraform

Job description

We are seeking a highly skilled Senior Data / ML Engineer to join our Data Science & Engineering team. In this role you will design, build, and maintain robust, scalable data pipelines and infrastructure on AWS, powering analytics, machine learning, and business-critical reporting. You will work closely with data scientists, analysts, and product teams to ensure reliable, performant data delivery across the organization.
The ideal candidate combines deep AWS expertise with modern development practices - including AI-assisted coding, containerization, and infrastructure as code - to accelerate delivery without sacrificing quality.
Experience Level: 5+ Years
Location: Kolkata (Rajarhat–Newtown)
Mode of Working: Work from Office
Timing: Ability to work in the US Eastern time zone. This may be relaxed to half day IST and half day US EST - based on project needs.

Key Responsibilities

• Design, develop, and optimize large-scale ETL/ELT pipelines using AWS services such as EMR (Spark), Glue, Lambda, and Step Functions.
• Orchestrate complex data workflows with Apache Airflow (Amazon MWAA or self-managed), ensuring reliability, observability, and SLA adherence.
• Architect and manage data storage solutions across Amazon S3 (data lake), Redshift (data warehouse), and RDS (relational databases), applying best practices for partitioning, compression, and cost optimization.
• Build and maintain containerized data applications and microservices using Docker and Amazon ECS/Fargate, including CI/CD automation.
• Develop event-driven and serverless data processing solutions with AWS Lambda, SQS, SNS, and EventBridge.
• Leverage AI-powered coding assistants and IDE integrations (e.g., Kiro, Cursor, Claude Code) to accelerate development, code review, and documentation.
• Implement data quality frameworks, monitoring, and alerting to ensure data integrity across all pipelines.
• Collaborate with Data Scientists to productionize ML models and feature pipelines.
• Define and enforce data governance, security, and access-control policies in line with organizational and regulatory standards.
• Contribute to infrastructure-as-code initiatives using Terraform, CloudFormation, or CDK.
  

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.
• 5+ years of professional experience in data engineering, with at least 3 years of hands-on AWS production workloads.
• 3+ years of experience with AWS Services: EMR (Spark/Hadoop), Apache Airflow, S3, Redshift, RDS, Lambda, Sagemaker and ECS.
• Solid experience with Docker (building, optimizing, and deploying containers) and container orchestration.
• Proficiency in Python and SQL; experience with Typescript is a plus.
• Demonstrated ability to use AI-assisted development tools within modern IDEs for rapid prototyping, code generation, testing and debugging.
• Strong understanding of data modeling, data governance, and data security best practices.
• AWS certifications such as AWS Certified Data Analytics - Specialty or AWS Certified Solutions Architect are a plus.
• Experience with infrastructure-as-code tools (Terraform, CloudFormation, CDK).
• Exposure to ML Ops workflows, feature stores, or model serving pipelines (e.g., SageMaker).
• Knowledge of cost-optimization strategies for large-scale AWS data workloads.
  

Tech Stack At a Glance

• Compute & Processing: EMR (Spark), Lambda, ECS/Fargate, Glue, Sagemaker
• Orchestration: Apache Airflow (MWAA), Step Functions
• Storage & Warehousing: S3, Redshift, RDS (PostgreSQL / MySQL)
• Containers & DevOps: Docker, ECS, GitHub Actions
• AI-Assisted Dev: Cursor, Kiro, Claude Code
• Languages: Python, SQL, Bash