Data Platform Engineer for AI Pipelines (Databricks/AWS)

present of AI; New York (NY); On-site; USD 150,000 - 210,000; Full time

Job summary

Reality Defender in New York is seeking a senior data platform engineer who will own the data platform powering AI research, including ingestion, processing, governance and export of image, video, audio, and text assets. You will scale pipelines at scale with Databricks, Spark, and AWS, and translate researcher needs into self-serve data tools.

Experience with large media files and governance decisions is highly valued.

Qualifications

• Long experience running large data pipelines.
• Deep familiarity with Databricks, Spark and AWS.
• Experience handling big media files and governance considerations.

Responsibilities

• Own the data platform feeding AI research from ingestion through processing, governance and export.
• Turn researcher needs into self-serve data tools for experiments and models.
• Build and operate scalable data pipelines for image, video, audio and text assets.
• Collaborate with ML researchers to understand data requirements and impact on model quality.

Skills

Databricks

Spark

AWS

Data pipelines

Tools

Terraform

Job description

Reality Defender in New York is seeking a senior data platform engineer who will own the data platform powering AI research, including ingestion, processing, governance and export of image, video, audio, and text assets. You will scale pipelines at scale with Databricks, Spark, and AWS, and translate researcher needs into self-serve data tools.

Experience with large media files and governance decisions is highly valued.