Senior Data Engineer, Training Data Platform

Prosus

Deutschland

Vor Ort

EUR 110.000 - 160.000

Vollzeit

Vor 3 Tagen
Sei unter den ersten Bewerbenden

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

Prosus is seeking a senior data engineer to build and operate the LCM training-data platform, handling acquisition, processing, synthetic generation, quality evaluation and dataset versioning. You will work with scientists and ML engineers to turn data into reproducible datasets and measure how data changes affect model performance.

You will implement pipelines and tooling for data provenance, tagging, evaluation and cost-aware operation, enabling researchers to run training runs with traceable

Qualifikationen

  • At least five years of relevant data engineering experience.
  • Strong Python and SQL, with experience writing tested, maintainable code.
  • Hands-on experience with distributed processing using Spark, Ray, Beam.

Aufgaben

  • Build acquisition and processing pipelines for portfolio data and external sources.
  • Develop the shared platform for inspecting sources, configuring processing and publishing training datasets.
  • Operate distributed workflows, monitor reliability and cost, implement access controls and retention requirements.
  • Work with scientists, engineers and teams on data provenance and privacy-aware handling of histories.
  • Implement dataset versioning and lineage across sources, processing code and dataset configurations.
  • Build quality checks for completeness, freshness, duplication, extraction accuracy and coverage.
  • Provide tools for source tagging, sampling and weighting, and maintain evaluation datasets with temporal splits and overlap checks.

Kenntnisse

Python
SQL
Distributed processing

Tools

Spark
Ray
Beam

Jobbeschreibung

Who We Are

You've ordered dinner through Just Eat Takeaway. You've bought and sold on OLX. You've paid with PayU. Chances are, you've already used a Prosus company today. We are Prosus, a global technology company powering some of the world's leading lifestyle e-commerce businesses. Every day, billions of people across India, Latin America, Europe and beyond use our products and services to buy, sell, pay, connect and discover. Our AI-driven ecosystem spans 100+ companies across e-commerce, food delivery, fintech and experiences, serving billions of consumers worldwide. AI powers how we operate and innovate, with more than 60,000 active AI agents helping our teams build, scale and move faster. Through our investment team, we also back the founders and technologies shaping the future: From robotics and autonomous systems to synthetic biology and frontier AI. Guided by our five values Innovation, Results, People, Impact, and Entrepreneurship, culture is our competitive advantage. We thrive with experimentation; we build, we test, we learn, we iterate. Join us in shaping the future of technology at global scale.

About the team

Join the AI team at Prosus, a global technology group with businesses in food delivery, e-commerce, classifieds, travel and payments. We build LCM, the Large Commerce Model: our own foundation model, trained on billions of transactions and interactions from proprietary data across the portfolio. LCM already powers search, recommendations and marketing in production at iFood. We are extending it across our companies around the world and to businesses outside the group, while broadening its capabilities in agents and real-time prediction tasks. Your work will shape what LCM learns next and feed directly into products used by millions of people.

About the role

We're looking for a senior data engineer to build and operate LCM's training-data platform, covering acquisition, processing, synthetic generation, quality evaluation and dataset versioning. Our core data includes events, sessions, orders, catalogues and feedback, supplemented by external sources and generated examples. In our flat team of around 30 engineers, you'll work with scientists and ML engineers to turn these sources into reproducible datasets and measure how changes to the data affect model performance. You'll also build tools and agents that automate preparation, task design and quality analysis. In the first three months, we expect you to take ownership of a part of the platform, such as a pipeline, the dataset versioning layer or the quality checks, deliver a reproducible dataset that a research project uses in a training run, and work with scientists to measure how your data changes affect model quality.

What the role involves
  • Build acquisition and processing pipelines for portfolio data and relevant public and licensed sources, including connectors and content extraction. Turn commerce records into ordered histories and link behaviour to outcomes without leaking future information.
  • Develop the shared platform for inspecting sources, configuring processing and publishing training datasets. Operate distributed workflows, monitor reliability and cost, and implement access controls and retention requirements.
  • Work with scientists, engineers and the relevant teams on data provenance, permitted use across companies and privacy-aware handling of customer histories.
  • Implement dataset versioning and lineage across source snapshots, processing code and dataset configurations, so releases can be reproduced and traced to the training runs that used them.
  • Build quality checks for completeness, freshness, duplication, extraction accuracy and coverage. Combine automated checks with sampled review to investigate problems that aggregate metrics can miss.
  • Provide tools for source tagging, sampling and weighting, and maintain evaluation datasets with temporal splits and overlap checks. Work with scientists on controlled comparisons to establish whether data changes improve model quality.
  • Build synthetic data pipelines supporting pre- and post-training, including instruction examples, preference data and agent trajectories. Implement generation, validation, scoring and filtering. Work with the research team to measure how the generated data affects model performance, and manage generation throughput and cost.
  • Build agents that turn catalogue sources into training datasets: propose tasks based on available data and target model capabilities, configure processing and generation, curate outputs and investigate quality failures. Keep agent actions traceable and validate proposed changes before dataset release.
Minimum qualifications
  • At least five years of relevant data engineering experience, including ownership of substantial production pipelines.
  • Strong Python and SQL, with experience writing tested, maintainable code.
  • Hands-on experience with distributed processing using Spark, Ray, Beam or a compar
Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Senior Data Engineer (m/f/d)
Senior Data Engineer (m/f/d)

GlassDollar GmbH • Berlin

Vor Ort
EUR 90.000 - 130.000
Senior Data Engineer (m/f/d)
Senior Data Engineer (m/f/d)

GlassDollar • Berlin

Vor Ort
EUR 85.000 - 120.000
Staff AI Engineer
Staff AI Engineer

Bluefish AI • Berlin

Hybrid
EUR 80.000 - 120.000
Data Engineer
Data Engineer

MediaRadar • Deutschland

Remote
EUR 90.000 - 120.000
Senior Data Processing Engineer
Senior Data Processing Engineer

European Tech Recruit • Berlin

Hybrid
EUR 70.000 - 110.000
Senior Data Scientist - (Global Search, Consumer)
Senior Data Scientist - (Global Search, Consumer)

Deliveryhero • Berlin

Hybrid
EUR 90.000 - 130.000
Senior Machine Learning Engineer | up to €180k
Senior Machine Learning Engineer | up to €180k

Bluebird • Berlin

Vor Ort
EUR 120.000 - 190.000
Lead Data Science Consultant
Lead Data Science Consultant

Contentserv Technologies Lda. (PT) • München

Remote
EUR 90.000 - 130.000
Senior Data Platform Engineer
Senior Data Platform Engineer

Xcede Recruitment Solutions • Berlin

Vor Ort
EUR 90.000 - 130.000
Lead Data Science Consultant
Lead Data Science Consultant

Centric Software GmbH • Deutschland

Remote
EUR 140.000 - 190.000
Remote work
Travel 1–2 times per year
Competitive salary and benefits
+1