Lead Data / ML Engineer

Web Spiders

Kolkata District

Hybrid

INR 2,500,000 - 4,000,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Web Spiders Group seeks a Lead Data/ML Engineer to own a large-scale data and ML platform on AWS. You will design and build PySpark pipelines, productionize ML models and LLM/agent capabilities, and drive end-to-end delivery from design to reliable production.

This hands‑on role expects you to write code daily, diagnose pipeline issues, and lead with practical judgement on LLMs, data quality and governance in a fast-paced environment. Hybrid/Remote options available.

Qualifications

  • Bachelor's or Master's degree in Computer Science, Engineering, Data Science or a related field.
  • 7+ years in data engineering or ML engineering with hands-on PySpark on production workloads.
  • Strong AWS production experience across EMR, S3, Redshift, RDS, Lambda, SageMaker and ECS.
  • Experience building with LLMs: prompt design, embeddings, vector search, RAG, agent or tool‑calling frameworks.
  • Solid ML lifecycle knowledge: feature engineering, training, validation, deployment and monitoring.
  • Experience with large transactional or behavioural datasets as a strong plus.
  • Strong Python and SQL; TypeScript is a plus.
  • Experience with Apache Airflow and Docker in production.

Responsibilities

  • Design, build and optimize large-scale ETL/ELT pipelines in PySpark on AWS (EMR, Glue).
  • Productionize ML models and feature pipelines including batch and real‑time inference and model serving on SageMaker.
  • Build LLM and agent‑powered data workflows and monitor output quality.
  • Orchestrate complex workflows with Apache Airflow ensuring reliability and SLA adherence.
  • Architect storage across S3, Redshift and RDS with cost optimization.
  • Develop containerized data services with Docker and ECS/Fargate and CI/CD via GitHub Actions.
  • Implement data quality, monitoring and alerting frameworks and enforce data governance.

Skills

PySpark
Python
SQL
AWS
ML lifecycle
Airflow
Docker

Education

Bachelor's degree
Master's degree

Tools

EMR
SageMaker
Redshift
Glue

Job description

We are looking for a Lead Data / ML Engineer to take end-to-end technical ownership of a large-scale data and ML platform on AWS. You will design and build PySpark pipelines, bring ML models and LLM and agent-based capabilities into production, and own what you build from first design through to running it reliably in production. This is a hands‑on role. Architecture and leadership experience is welcome, but you will still write code every day, and you will be the person others turn to when a pipeline breaks, a job slows down, or a design needs a second opinion.

We are looking for a Lead Data / ML Engineer to take end-to-end technical ownership of a large-scale data and ML platform on AWS. You will design and build PySpark pipelines, bring ML models and LLM and agent-based capabilities into production, and own what you build from first design through to running it reliably in production.

This is a hands‑on role. Architecture and leadership experience is welcome, but you will still write code every day, and you will be the person others turn to when a pipeline breaks, a job slows down, or a design needs a second opinion.

About the Work:

You will work with high-volume consumer transaction data, turning it into insights and predictive models that help understand and anticipate consumer behaviour. The work spans large-scale data processing, feature engineering, machine learning, and applying LLMs and agents to enrich and make sense of the data.

Experience: 7+ Years

Location: Kolkata (Rajarhat–Newtown)

Mode of Working: WFO / Hybrid / Remote

Timing: Ability to work in the US Eastern time zone. This may be relaxed to half day IST and half day US EST based on project needs.


What We Are Looking For:
  • End-to-end ownership: you take a problem from design to production and stay accountable for it once it is live
  • Hands‑on depth: you write, review and debug production code yourself, not only design it
  • Troubleshooting leadership: you help teammates diagnose failing jobs, performance problems and data issues, and leave them better at solving the next one
  • Practical judgement on LLMs and agents: you know where they add real value in a data pipeline and how to measure whether they are working
Key Responsibilities:
  • Design, build and optimize large-scale ETL/ELT pipelines in PySpark on AWS (EMR, Glue), with strong attention to partitioning, skew, shuffle and cost.
  • Productionize ML models and feature pipelines, including training data pipelines, batch and real‑time inference, and model serving on SageMaker.
  • Build LLM and agent‑powered capabilities into data workflows, such as document and entity extraction, classification and enrichment at scale, RAG pipelines, and tool‑calling agents, along with evaluation and monitoring of their output quality.
  • Orchestrate complex workflows with Apache Airflow (MWAA or self‑managed), ensuring reliability, observability and SLA adherence.
  • Architect storage across S3 (data lake), Redshift (warehouse) and RDS, applying best practices for data modeling, partitioning, compression and cost optimization.
  • Build containerized data services with Docker and ECS/Fargate, with CI/CD through GitHub Actions.
  • Develop event‑driven and serverless processing with Lambda, SQS, SNS and EventBridge.
  • Use AI coding assistants (Claude Code, Cursor, Kiro) to speed up delivery, and set the team's standards for reviewing and validating AI‑generated code.
  • Be the go‑to person for production issues: lead root‑cause analysis on failing or slow jobs, data quality incidents and model problems, and help teammates debug their own work.
  • Review code, guide design decisions and help engineers on the team grow technically.
  • Implement data quality, monitoring and alerting frameworks, and enforce data governance, security and access‑control policies.
  • Contribute to infrastructure as code using Terraform, CloudFormation or CDK.
Required Qualifications:
  • Bachelor's or Master's degree in Computer Science, Engineering, Data Science or a related field.
  • 7+ years in data engineering or ML engineering, with at least 4 years of hands‑on PySpark on production workloads at scale.
  • Strong AWS production experience across EMR, S3, Redshift, RDS, Lambda, SageMaker and ECS.
  • Practical, hands‑on experience building with LLMs: prompt design, embeddings and vector search, RAG, agent or tool‑calling frameworks, and evaluating LLM output quality.
  • Solid understanding of the ML lifecycle: feature engineering, training and validation, model deployment and monitoring.
  • Experience working with large transactional or behavioural datasets is a strong plus.
  • Strong Python and SQL. TypeScript is a plus.
  • Experience with Apache Airflow and Docker in production.
  • A track record of owning systems end to end in production, including on‑call or production support, and of helping other engineers troubleshoot.
  • Strong grasp of data modeling, governance and security best practices.
Good To Have:
  • AWS certifications (Data Engineer Associate, Machine Learning Specialty, Solutions Architect)
  • Experience with feature stores, MLOps tooling or model serving at scale
  • Infrastructure as code (Terraform, CloudFormation, CDK)
  • Experience optimizing costs for large AWS data workloads
Tech Stack at a Glance:
  • Compute & Processing: PySpark on EMR, Glue, Lambda, ECS/Fargate, SageMaker
  • LLM & Agents: LLM APIs, embeddings and vector stores, RAG, agent frameworks
  • Orchestration: Apache Airflow (MWAA), Step Functions
  • Storage & Warehousing: S3, Redshift, RDS (PostgreSQL / MySQL)
  • Containers & DevOps: Docker, ECS, GitHub Actions, Terraform
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data / ML Engineer
Lead Data / ML Engineer

Web Spiders • Kolkata

Hybrid
INR 4,000,000 - 8,000,000
Technical Lead - Data Engineer (Data&AI)
Technical Lead - Data Engineer (Data&AI)

Srijan: Now Material • Gurugram District

On-site
INR 4,000,000 - 7,500,000
Senior Data Engineer
Senior Data Engineer

Lifesight • Karnataka

Hybrid
INR 2,500,000 - 4,500,000
Lead AWS Data Engineer
Lead AWS Data Engineer

Weekday (YC W21) • Bengaluru

On-site
INR 1,000,000 - 1,500,000
Lead Data Engineer (Lead Software Engineer)
Lead Data Engineer (Lead Software Engineer)

DigitalXNode • Ahmedabad District

On-site
INR 1,800,000 - 2,600,000
Senior Data Engineer (with AI/ML experience) India
Senior Data Engineer (with AI/ML experience) India

IDT • Pune District

Remote
INR 4,200,000 - 6,800,000
Remote work opportunity
Annual performance review
Career growth opportunities
+2
Technical Lead – Data Engineering
Technical Lead – Data Engineering

Myntra Designs Private Limited • Bengaluru

On-site
INR 1,000,000 - 1,800,000
Data Engineer
Data Engineer

Tensor Solution Tech Private Limited • Bengaluru

On-site
INR 1,200,000 - 2,800,000
Lead Data Engineer
Lead Data Engineer

Gardahaus • Karnataka

Hybrid
INR 2,800,000 - 3,800,000
Senior Technical Specialist - AWS/AWS Lambda/AWS Glue/AWS SNS
Senior Technical Specialist - AWS/AWS Lambda/AWS Glue/AWS SNS

Datavail Infotech • Bengaluru

On-site
INR 2,400,000 - 4,000,000