Base pay range
$120,000.00/yr - $120,000.00/yr
Location
Location: Austin, TX (3 days' work from office)
Discovery Phase
During the discovery stage, it will be 5 days working from office for the first 4 weeks of discovery
Profiles
The profiles we are getting mostly are on their recent GenAI experience only. We need Strong Data Scientist and ML Engineer/GenAI Engineer background, fine if elastic search is not available,
Mandatory Skills
ElasticSearch, OpenSearch, Python, LLM, GenAI, Semantic Search, Re-Ranking, AWS, Search Engineer
Job Description
We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based ElasticSearch system by integrating state-of-the-art semantic search, dense retrieval, and LLM-powered ranking techniques.
This role will drive the transformation of traditional search into an intelligent, context-aware, personalized, and high-precision search experience.
The ideal candidate has hands‑on experience with ElasticSearch internals, information retrieval (IR), embedding-based search, BM25, re‑ranking, LLM‑based retrieval pipelines, and AWS cloud deployment.
Modernizing the Search Platform
- Analyze limitations in current regex & keyword-only search implementation on ElasticSearch.
- Enhance search relevance using:
- Synonyms, analyzers, custom tokenizers
- Boosting strategies and scoring optimization
- Introduce semantic / vector‑based search using dense embeddings.
- Implement LLM‑powered search workflows including:
- Query rewriting and expansion
- Embedding generation (OpenAI, Cohere, Sentence Transformers, etc.)
- Re‑ranking using cross‑encoders or LLM evaluators
- Build RAG (Retrieval Augmented Generation) flows using ElasticSearch vectors, OpenSearch, or AWS‑native tools.
Search Infrastructure Engineering
- Build and optimize search APIs for latency, relevance, and throughput.
- Design scalable pipelines for:
- Indexing structured and unstructured text
- Maintaining embedding stores
- Real‑time incremental updates
- Implement caching, failover, and search monitoring dashboards.
- Deploy and operate solutions on AWS, leveraging:
- OpenSearch Service or EC2‑managed ElasticSearch
- Lambda, ECS/EKS, API Gateway, SQS/SNS
- Implement CI/CD for search models and pipelines.
- Conduct A/B experiments to measure improvements.
- Tune ranking functions and hybrid search scoring.
- Partner with product teams to refine search behaviors with real usage patterns.
Required Skills & Qualifications
- 10 years of experience in AI/ML, NLP, or IR systems, with hands‑on search engineering.
- Strong expertise in ElasticSearch/OpenSearch: analyzers, mappings, scoring, BM25, aggregations, vectors.
- Experience with semantic search:
- Embeddings (BERT, SBERT, Llama, GPT‑based, Cohere)
- Vector databases or ES vector fields
- Working knowledge of LLM‑based retrieval and RAG architectures.
- Proficient in Python; familiarity with Java/Scala is a plus.
- Hands‑on AWS experience (OpenSearch, SageMaker, Lambda, ECS/EKS, EC2, S3, IAM).
- Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker.
- Familiar with typical IR metrics and search evaluation frameworks.
Preferred Skills
- Knowledge of cross‑encoder and bi‑encoder architectures for re‑ranking.
- Experience with query understanding, spell correction, autocorrect, and autocomplete features.
- Exposure to LLMOps / MLOps in search use cases.
- Understanding of multi‑modal search (text + images) is a plus.
- Experience with knowledge graphs or metadata‑aware search.