AI-Driven Data Engineer - Frontier Pipeline Evaluator

Obsidian; New York (NY); Remote; USD 193,000 - 248,000; Full time

Job summary

Obsidian is seeking contributors to support a Frontier Code Agents project focused on evaluating and improving frontier AI coding models within realistic data engineering workflows. You will assess complex ETL pipelines, data warehouses, and distributed systems, using leading AI coding agents to drive rigorous testing and analysis.

The role emphasizes critical thinking, bug detection, and comparison across multiple frontier models to gauge strengths, weaknesses, and scalability under real-world

Qualifications

• 2+ years of professional data engineering experience.
• Experience building ETL pipelines, data warehouses, analytics platforms or distributed data systems.
• Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI.
• Ability to evaluate model-generated data infrastructure and pipeline implementations.
• Experience operating large-scale data platforms is preferred.

Responsibilities

• Use frontier AI coding agents to complete and evaluate complex data engineering tasks.
• Review model-generated implementations involving ETL pipelines, data warehouses, analytics platforms, and distributed data systems.
• Identify bugs, edge cases, scalability issues, and failure modes.
• Compare outputs from multiple frontier models and assess their strengths and weaknesses.
• Apply professional engineering judgment to realistic data engineering scenarios.

Skills

Data engineering

ETL pipelines

Data warehouses

Analytics platforms

Distributed data systems

AI coding agents

Model evaluation

Large-scale data platforms

Tools

Cursor

Claude Code

Codex

Windsurf

Gemini CLI

Job description

Obsidian is seeking contributors to support a Frontier Code Agents project focused on evaluating and improving frontier AI coding models within realistic data engineering workflows. You will assess complex ETL pipelines, data warehouses, and distributed systems, using leading AI coding agents to drive rigorous testing and analysis.

The role emphasizes critical thinking, bug detection, and comparison across multiple frontier models to gauge strengths, weaknesses, and scalability under real-world