AI/ML Data Platform Lead — Scalable Pipelines, Equity

TEEMA Solutions Group; Toronto; Hybrid; CAD 100,000 - 130,000; Full time

Job summary

A technology startup is seeking a Data Platform Software Lead Engineer to drive the architecture underlying its AI/ML pipelines. In this role, you'll design and build reliable systems for data ingestion and processing, focusing on large-scale code and text datasets. The ideal candidate has over 8 years of experience in data-intensive engineering and skills in Python, Spark, and AWS. Join a diverse team and influence the technical path of an innovative company.

Benefits

Competitive pay with equity

Work with cutting-edge cloud and ML/AI technologies

Collaborate with a diverse, high-caliber team

Dynamic, innovation-driven workplace culture

Qualifications

• 8+ years in data-intensive software engineering.
• Proficiency in Python, Go, or Scala; Spark or Ray; Airflow or Prefect; Kafka; Redis; Postgres or ClickHouse; GitHub APIs.
• Understanding of how datasets power AI/ML workflows.
• Proven experience in scalable data infrastructure and pipeline development.
• Skills in web crawling, scraping, and large-scale ingestion.
• Cloud-native experience (e.g., AWS, containerized compute, security).

Responsibilities

• Architect and implement scalable data platforms for code/text dataset ingestion, processing, and delivery.
• Build web-scale crawling and metadata extraction tools from open-source code repositories.
• Develop reliable, distributed pipelines with frameworks like Spark, Kafka, and Airflow/Prefect.
• Enable data visualization, sampling, and analytics for research teams to improve model performance.
• Collaborate with researchers, infrastructure, and compliance teams to meet technical and governance requirements.

Skills

Python

Go

Scala

Spark

Ray

Airflow

Prefect

Kafka

Redis

Postgres

ClickHouse

GitHub APIs

Job description

A technology startup is seeking a Data Platform Software Lead Engineer to drive the architecture underlying its AI/ML pipelines. In this role, you'll design and build reliable systems for data ingestion and processing, focusing on large-scale code and text datasets. The ideal candidate has over 8 years of experience in data-intensive engineering and skills in Python, Spark, and AWS. Join a diverse team and influence the technical path of an innovative company.