Data Platform Engineer for ML Pipelines & Provenance

basis-research; New York (NY); On-site; USD 90,000 - 130,000; Full time

Job summary

Basis-research is looking for a full-time Data Engineer to build trustworthy data pipelines and ensure high-quality data flows across its platform. This role involves collaborating with research teams, implementing data quality frameworks, and curating documented datasets for ML models.

The ideal candidate has a strong background in data engineering, cloud platforms like Snowflake, and distributed computing, along with a passion for solving complex data challenges.

Qualifications

• 5+ years of experience in data engineering for ML/AI systems.
• Strong proficiency in SQL and Python for data processing.
• Experience with cloud platforms like Snowflake and BigQuery.

Responsibilities

• Design and build data pipelines for research and platform offerings.
• Implement data quality frameworks including validation rules.
• Coordinate cross-project data initiatives to prevent duplicate work.

Skills

Data pipeline construction

Data quality frameworks

SQL

Python

Distributed computing frameworks

Cloud data platforms

Data governance

Education

Bachelor's degree in Computer Science or related field

Tools

Snowflake

Kafka

Airflow

Job description

Basis-research is looking for a full-time Data Engineer to build trustworthy data pipelines and ensure high-quality data flows across its platform. This role involves collaborating with research teams, implementing data quality frameworks, and curating documented datasets for ML models.

The ideal candidate has a strong background in data engineering, cloud platforms like Snowflake, and distributed computing, along with a passion for solving complex data challenges.