Data Pipeline Engineer - Validation, Versioning & Delivery

Reinforced Learning; Town of Tonawanda (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

Reinforced Learning in the United States (New York) seeks an expert for building and maintaining data pipelines that validate, transform, and ship expert submissions to customers. You will implement schemas, quality gates, and dataset versioning with lineage from raw submissions to delivered files.

Delivery tooling must be secure and operable by dataset leads. The role emphasizes production experience with batch and streaming data in cloud environments, pragmatic tooling decisions, and a strong

Qualifications

• Proficient in Python and data pipelines.
• Experience in production data systems.
• Working with batch and streaming data in cloud environments.

Responsibilities

• Build pipelines to validate and transform expert submissions into customer formats.
• Define schemas and quality gates to prevent bad records from shipping.
• Implement dataset versioning and lineage from raw input to delivered files.
• Develop delivery tooling for secure handoffs into customer environments.
• Ensure pipelines are observable and maintainable for dataset leads.

Skills

Python

Batch processing

Streaming data

Data quality

Tools

Cloud platforms

Job description

Reinforced Learning in the United States (New York) seeks an expert for building and maintaining data pipelines that validate, transform, and ship expert submissions to customers. You will implement schemas, quality gates, and dataset versioning with lineage from raw submissions to delivered files.

Delivery tooling must be secure and operable by dataset leads. The role emphasizes production experience with batch and streaming data in cloud environments, pragmatic tooling decisions, and a strong