Lead AI Engineer: Semantic Search & LLM Ranking

Software Technology Inc.; Dallas (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology firm is seeking a skilled Lead AI Engineer to modernize its ElasticSearch system by integrating advanced search techniques. This role involves designing embedding-based retrieval pipelines and deploying services on AWS. Candidates should have a solid background in information retrieval and LLM-powered ranking. The position offers an opportunity to work collaboratively with cross-functional teams to enhance search capabilities and deliver a personalized user experience.

Qualifications

• Hands-on experience with ElasticSearch internals and search system design.
• Strong background in information retrieval, embedding search, and ranking algorithms.
• Proficiency with LLM-based retrieval pipelines and re-ranking techniques.
• Experience deploying and scaling search services on AWS.
• Excellent problem-solving skills and a collaborative mindset.

Responsibilities

• Modernize the existing ElasticSearch architecture to support semantic search and LLM-powered ranking.
• Design and implement embedding-based retrieval pipelines and dense retrieval modules.
• Integrate and fine-tune LLM models for search ranking and re-ranking tasks.
• Deploy and maintain search services on AWS, ensuring scalability and performance.
• Collaborate with cross-functional teams to define search use cases and success metrics.

Skills

ElasticSearch internals

Information retrieval (IR)

Embedding-based search

BM25

Re-ranking

LLM-based retrieval pipelines

AWS cloud deployment

Job description

A leading technology firm is seeking a skilled Lead AI Engineer to modernize its ElasticSearch system by integrating advanced search techniques. This role involves designing embedding-based retrieval pipelines and deploying services on AWS. Candidates should have a solid background in information retrieval and LLM-powered ranking. The position offers an opportunity to work collaboratively with cross-functional teams to enhance search capabilities and deliver a personalized user experience.