NLP Data Engineer for Large-Scale AI Data Pipelines

Institute of Foundation Models; Sunnyvale (CA); Presencial; USD 120 000 - 180 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Medical benefits

Dental benefits

Vision benefits

Bonus

401K Plan

Paid time off

Parental leave

Employee Assistance Program

Life insurance

Disability insurance

Resumo da oferta

The Institute Of Foundation Models in Sunnyvale, CA is seeking a Data Engineer specialized in NLP and large-scale data processing to build and curate high-quality datasets for cutting-edge AI research. You will implement scalable data pipelines, web crawling, and LLM-output refinement to support researchers and engineers.

The role emphasizes Python-based engineering, data quality, and collaboration across teams, with responsibilities spanning data collection, processing, and documentation in a

Qualificações

• Bachelor's degree in computer science, data science, or engineering is required.
• Master’s degree or PhD preferred, related to data engineering or AI research.

Responsabilidades

• Rapidly collect, curate, and preprocess datasets based on NLPresearchers' specifications, delivering data within tight timelines.
• Develop and maintain efficient web crawling solutions, APIs, and automated data workflows to improve data collection processes.
• Refine and evaluate outputs from large language models (LLMs) to generate structured datasets for model training and benchmarking.
• Implement scalable data pipelines ensuring efficient processing, storage, and distribution to research teams.
• Collaborate with researchers and engineers to ensure data meets quality and relevance criteria.
• Document data collection methodologies, dataset characteristics, and pipeline architecture clearly.
• Engage with peer teams and participate in technical reviews to uphold data quality standards.
• Represent MBZUAI at industry and research forums, showcasing capabilities in large-scale data processing and AI data infrastructure.

Conhecimentos

Python

Data engineering

Web crawling

AWS

Spark

Kafka

Kubernetes

SQL

Algorithms

Data structures

Formação académica

Bachelor's degree in CS/Data Science/Engineering

Master’s or PhD preferred

Ferramentas

Spark

Kafka

Kubernetes

APIs

Descrição da oferta de emprego

The Institute Of Foundation Models in Sunnyvale, CA is seeking a Data Engineer specialized in NLP and large-scale data processing to build and curate high-quality datasets for cutting-edge AI research. You will implement scalable data pipelines, web crawling, and LLM-output refinement to support researchers and engineers.

The role emphasizes Python-based engineering, data quality, and collaboration across teams, with responsibilities spanning data collection, processing, and documentation in a