Data Scientist Senior Consultant II

Allstate India Pvt Ltd; Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Allstate Corporation in Bengaluru, India is seeking a senior AI/ML engineer to design and build AI-driven Insight Studio for Scholar, transforming large-scale customer conversations into real-time insights. You will productionize AI systems, integrate LLMs and RAG pipelines, and manage end-to-end ML lifecycle.

The role requires deep expertise in ML/data pipelines, MLOps, and cost-efficient, scalable architectures, with hands-on work on LangChain, vector search, and cloud-native deployments.

Qualifications

• Demonstrated data science and AI expertise.
• Experience building scalable AI/ML systems in production.

Responsibilities

• Translate business problems into end-to-end AI solutions.
• Design and operate production-grade AI systems for insights from large-scale data.
• Architect and implement RAG pipelines with structured/unstructured data.
• Build and optimize vector search and embeddings-based retrieval systems.
• Design AI agent workflows and multi-step reasoning.
• Apply prompt engineering and LLM orchestration for quality and cost efficiency.
• Integrate foundation models via AWS Bedrock and deployment strategies.
• Maintain scalable data pipelines for real-time and batch workloads.
• Own end-to-end lifecycle of AI/ML solutions including MLOps practices.
• Monitor AI systems with observability tools (Datadog, CloudWatch).
• Analyze model outputs and iterate to improve business impact.

Skills

Data Science

Python for AI/ML

Production-grade AI/ML

Distributed systems

MLOps / LLMOps

Prompt engineering

LLM evaluation

Tools

LangChain

LangGraph

LlamaIndex

Vector databases

AWS Bedrock

SageMaker

Job description

Job Description

At Allstate, great things happen when our people work together to protect families and their belongings from life’s uncertainties. And for more than 90 years, our innovative drive has kept us a step ahead of our customers’ evolving needs. From advocating for seat belts, air bags and graduated driving laws, to being an industry leader in pricing sophistication, telematics, and, more recently, device and identity protection.

In this role, you will design and build AI-driven Insight Studio for the Scholar product, transforming large-scale customer conversations into actionable, real-time insights. You will hands-on build and productionize AI systems, integrating LLMs, RAG pipelines, and AI agents into scalable, reliable applications that directly power business decisions.

Key Responsibilities

• Translate complex business problems into end-to-end AI solutions, leveraging machine learning and Generative AI techniques aligned with product and business goals.
• Design, build, and operate production-grade AI systems for conversational understanding, summarization, classification, and insight generation from large-scale customer interaction data.
• Architect and implement Retrieval-Augmented Generation (RAG) pipelines, integrating structured and unstructured data sources to deliver context-aware, grounded, and explainable AI outputs.
• Build and optimize vector search and embeddings-based systems, enabling semantic search, similarity matching, and efficient information retrieval at scale.
• Design and implement AI agent workflows, orchestrating multi-step reasoning, tool usage, and decision-making using frameworks such as LangChain, LangGraph, or similar.
• Apply prompt engineering and LLM orchestration techniques to improve output quality, consistency, latency, and cost efficiency in production environments.
• Integrate foundation models via platforms such as AWS Bedrock, including fine-tuning, evaluation, and deployment strategies tailored to business use cases.
• Build and maintain scalable data pipelines and distributed processing systems to support real-time and batch AI workloads.
• Own end-to-end lifecycle of AI/ML solutions, including data preparation, model development, evaluation (including LLM evaluation and guardrails), deployment, and continuous improvement.
• Implement MLOps and LLMOps practices, including CI/CD pipelines, versioning, monitoring, and automated evaluation for both traditional ML and AI systems.
• Monitor and optimize AI systems using observability tools (e.g., Datadog, CloudWatch), ensuring performance, reliability, and cost control in production.
• Analyze model outputs, communicate findings, and iterate on solutions to continuously improve business impact and user experience.

Essential Skills

• Data Science & AI Expertise
• Strong proficiency in Python for AI/ML
• Proven experience building and scaling production-grade AI/ML systems
• Deep understanding of data pipelines, distributed systems, and scalable architectures
• Strong grounding in MLOps / LLMOps workflows and lifecycle management
• Practical experience building applications using: Large Language Models (LLMs) (prompting, fine-tuning, orchestration) Retrieval-Augmented Generation (RAG) architectures Embeddings and vector search systems AI agents and multi-step reasoning workflows
• Strong understanding of LLM evaluation, guardrails, and responsible AI practices

Frameworks & Tooling

• Hands-on experience with: LangChain, LangGraph, LlamaIndex or similar orchestration frameworks
• Experience designing modular, reusable AI pipelines and components
• Data & Storage Experience with vector databases and modern data storage systems
• Familiarity with handling large-scale structured and unstructured datasets

Cloud & AI Platforms

• Experience with: Amazon SageMaker or similar ML platforms
• AWS Bedrock for foundation model access and deployment
• AWS services such as Lambda, S3, ElastiCache
• Experience deploying LLM-powered applications in cloud-native architectures

Observability & Reliability

• Experience with: Datadog, AWS CloudWatch
• Strong ability to monitor, debug, and optimize AI systems in production

Additional Platforms

• Experience with Microsoft Fabric or similar analytics platforms

Primary Skills

Artificial Intelligence Markup Language, Business Case Analyses, Data Analytics, Predictive Modeling, Waterfall Model Shift Time Shift B (India)

About Allstate

Joining our team isn’t just a job — it’s an opportunity. One that takes your skills and pushes them to the next level. One that encourages you to challenge the status quo. One where you can shape the future of protection while supporting causes that mean the most to you. Joining our team means being part of something bigger – a winning team making a meaningful impact. The Allstate Corporation is one of the largest publicly held insurance providers in the United States. Ranked No. 84 in the 2023 Fortune 500 list of the largest United States corporations by total revenue, The Allstate Corporation owns and operates 18 companies in the United States, Canada, Northern Ireland, and India. Allstate India Private Limited, also known as Allstate India, is a subsidiary of The Allstate Corporation. The India talent center was set up in 2012 and operates under the corporation's Good Hands promise. As it innovates operations and technology, Allstate India has evolved beyond its technology functions to be the critical strategic business services arm of the corporation. With offices in Bengaluru and Pune, the company offers expertise to the parent organization’s business areas including technology and innovation, accounting and imaging services, policy administration, transformation solution design and support services, transformation of property liability service design, global operations and integration, and training and transition. Learn more about Allstate India here. At Allstate, we work hard to help people live a good life every day. Allstaters are dedicated to serving clients, customers, and communities, which allows employees to find meaning and value in their work. Allstate offers an environment that fosters innovative thinking where you’ll be able to explore your ideas and feel proud of the work you do. Allstate helps protect nearly 16 million households with auto, home, life, and retirement products. We want every professional connected to Allstate to be committed to giving our customers the best and that means finding the best talent. We want you to be our next great addition.