Machine Learning Engineer (Video Understanding & Segmentation)

Maxinsights; Santa Clara (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Health insurance

Vision care

Dental coverage

401(k)

Paid holidays

PTO (Paid Time Off)

Sick leave

Job summary

Maxinsights is seeking a Machine Learning Engineer to advance video understanding and segmentation at scale. You will prototype embedding pipelines, build LLM-based indexing, and architect end-to-end workflows that translate raw video into structured, searchable data.

The role requires hands-on experience in computer vision, multi-modal learning, and large-scale systems, with a focus on embedding models, vector search, and orchestration frameworks.

Qualifications

• MS or PhD in Computer Science, Electrical Engineering, or related field, or equivalent practical experience.
• 3+ years of hands-on experience in computer vision or multi-modal machine learning.
• Strong proficiency in Python and PyTorch, with solid software engineering fundamentals.
• Hands-on experience with CLIP or similar vision-language/video embedding models for retrieval or representation learning.
• Experience building or fine-tuning LLM-based systems for video/image understanding (captioning, QA, summarization).
• Familiarity with agentic system design — tool use, multi-step reasoning, and orchestration frameworks.

Responsibilities

• Build and optimize video/image embedding pipelines using CLIP-style and other vision-language embedding models for large-scale search/retrieval.
• Develop LLM-based video understanding systems for indexing, summarization, and QA over long-form video.
• Design and implement video chunking/segmentation algorithms to create temporally-aligned clips.
• Build automated captioning systems combining vision-language models and LLMs for fine-grained descriptions.
• Architect agentic systems and pipelines chaining embedding, captioning, retrieval, and reasoning steps.
• Scale video search infrastructure (vector indexing, retrieval, ranking) for millions of clips.
• Collaborate with annotation and data teams to integrate outputs into training pipelines for embodied AI.
• Evaluate and benchmark models and frameworks; track frontier research for platform improvements.

Skills

Python

PyTorch

Multi-modal ML

Computer Vision

Embedding models

Software engineering

Vector search

Model orchestration

Education

MS or PhD in CS/EE or related

Tools

CLIP-style embeddings

FAISS

Milvus

LangChain / LlamaIndex

Job description

Job Description: We are seeking a highly motivated Machine Learning Engineer to join our core research and development team, focused on video understanding and segmentation. In this role, you will build the systems that let us search, decompose, and describe massive volumes of egocentric and human-robot video at scale — turning raw, unstructured footage into structured, searchable, and richly annotated training data. You will work across video/image embedding models, LLM-based video understanding, and agentic pipelines that orchestrate multiple models into end-to-end workflows. This is a foundational role that directly shapes the data quality and scalability of our entire training data platform.

Responsibilities

• Build and optimize video/image embedding pipelines using CLIP-style and other vision-language embedding models to power large-scale, multi-modal video search and retrieval.
• Develop LLM-based video understanding systems for semantic indexing, summarization, and question-answering over long-form egocentric and third-person video.
• Design and implement instruction-level and action-level video chunking/segmentation algorithms that decompose long videos into structured, temporally-aligned clips.
• Build automated video captioning systems that combine vision-language models and LLMs to produce fine-grained, temporally-grounded descriptions of actions and scenes.
• Architect agentic systems and orchestration pipelines that chain embedding, captioning, retrieval, and LLM reasoning steps into reliable, end-to-end video understanding workflows.
• Develop and scale video search infrastructure (vector indexing, retrieval, ranking) to support semantic and multi-modal queries over millions of video clips.
• Collaborate with annotation, data engineering, and robotics teams to integrate video understanding outputs into downstream training pipelines for embodied AI and robot learning.
• Evaluate and benchmark embedding models, LLMs, and agentic frameworks against production needs; track frontier research and bring relevant techniques into the platform.
• Contribute to internal tooling, documentation, patents, and open-source initiatives where applicable.
• Mentor junior engineers and interns, and help shape the long-term technical roadmap for video understanding.

Minimum Qualifications

• MS or PhD in Computer Science, Electrical Engineering, or a related technical field, or equivalent practical experience.
• 3+ years of hands-on experience in computer vision or multi-modal machine learning, with direct experience in video understanding tasks.
• Strong proficiency in Python and PyTorch, with solid software engineering fundamentals.
• Hands-on experience with CLIP or similar vision-language/video embedding models for retrieval or representation learning.
• Experience building or fine-tuning LLM-based systems for video/image understanding (e.g., captioning, video QA, summarization).
• Familiarity with agentic system design — tool use, multi-step reasoning, and orchestration frameworks (e.g., LangChain, LlamaIndex, or custom agent loops).
• Experience working with large-scale video data pipelines and vector search/retrieval infrastructure (e.g., FAISS, Milvus, or equivalent).

Preferred Qualifications

• PhD with a research focus in video understanding, multi-modal learning, or vision-language models.
• Experience with temporal action segmentation, action localization, or instruction-level video chunking algorithms.
• Experience working with egocentric video datasets or head-mounted-device (HMD) captured data.
• Track record of deploying production-scale video search or retrieval systems.
• Experience integrating foundation or vision-language models (e.g., CLIP, VideoCLIP, RT-1/VLA variants) into perception or decision-making pipelines.
• Publications in top-tier computer vision or ML venues (e.g., CVPR, ICCV, ECCV, NeurIPS, ICLR, etc).
• Experience with humanoid robotics or embodied AI data pipelines is a plus.

Default Benefits

• Health insurance
• Vision care
• Dental coverage
• 401(k)
• Paid holidays
• PTO (Paid Time Off)
• Sick leave