Senior Data Scientist with Bayesian statistical analysis

Intellias

Poland

On-site

PLN 180,000 - 260,000

Full time

46 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Intellias is seeking a skilled data platform engineer to design and operationalize enterprise-grade pipelines on Databricks and AWS. You will handle pseudonymisation, entity resolution, and scalable image processing, while implementing robust MLOps and monitoring capabilities.

Responsibilities include platform setup with client teams, secure ingestion pipelines, feature engineering at scale, and privacy-focused governance with thorough documentation.

Qualifications

  • Expertise in PySpark and Spark SQL with complex data transforms.
  • Hands-on Databricks experience including Unity Catalog and Delta lake.
  • Cloud and security basics in AWS: IAM, KMS, Secrets Manager.
  • Experience with pseudonymisation and data privacy controls.
  • Proficiency in entity resolution and blocking strategies.
  • Ability to process images at scale using GPU pipelines.
  • Knowledge of mixed-data clustering and Bayesian analysis basics.
  • Strong feature engineering and model evaluation practices.
  • Experience with MLOps, monitoring and lineage in data platforms.
  • Clear documentation and runbook creation for handover.

Responsibilities

  • Platform setup: configure Databricks, compute, secrets and access for clients.
  • Secure data ingestion: parsing, pseudonymisation, data quality checks.
  • Feature engineering at scale: embedding jobs, linkage keys, pysuo.
  • Modeling support: feature selection, clustering, comparisons and tests.
  • Automation: orchestrate end-to-end, idempotent workflows.
  • Monitoring: build data-quality and drift monitoring with alerts.
  • Privacy engineering: enforce access controls and data provenance.
  • Documentation: prepare runbooks and conduct live walkthroughs.

Skills

PySpark & Spark SQL
Databricks
AWS
Pseudonymisation in pipelines
Entity resolution
Image processing at scale
Mixed-data clustering
Bayesian statistical analysis
Feature selection
MLOps & monitoring
Handover

Tools

Databricks
AWS
PyTorch
timm
mealpy
PyOD
AssureID

Job description

  • Let’s breathe life into great tech ideas! With 3,000 people globally, Intellias is a company where benchmark technological solutions are born. Join in and take your part in digitalizing the world.
  • Join a transformative data and AI platform initiative aimed at modernizing enterprise-scale capabilities and enabling real-time decision-making. This project delivers a comprehensive roadmap covering AI, MLOps, data governance, and platform scalability, supporting a shift towards data-first operations and intelligent automation.
Requirements:
  • PySpark & Spark SQL: Nested-JSON flattening (explode, structs),`mapInPandas` / pandas UDFs, window functions for velocity and distinct-count aggregates, partitioning and performance tuning
  • Databricks: Unity Catalog (catalogs, schemas, grants, column masks, tags, lineage), external locations on S3, cluster / compute policies incl. GPU, Auto Loader, Delta (MERGE, time travel, VACUUM), Workflows, Repos, secret scopes
  • AWS: S3, IAM roles / instance profiles, KMS decryption in jobs, Secrets Manager, basic cost awareness
  • Pseudonymisation in pipelines: HMAC tokenization with per-key-type keys, normalisation (Unicode NFKD, case, token sort), phonetic keys, never logging raw values, scripted deletion (DROP / VACUUM / secret destruction)
  • Entity resolution (implementation): Blocking / candidate generation with exact and approximate keys, edit-distance tolerance, match scores, surrogate IDs, conflict counting rather than merging
  • Image processing at scale: Running a GPU batch job that decrypts, crops and embeds images in memory (PyTorch / timm), PCA, LSH bucketing
  • Mixed-data clustering: Hands-on: `kmodes` (k-prototypes), `gower` + `kmedoids` with sampling, R `kamila` on Databricks, `StepMix` (latent class), bootstrap ARI
  • Bayesian statistical analysis: Hands-on experience applying Bayesian methods to real-world data problems, including prior/posterior modelling, Bayesian inference, uncertainty estimation, and interpretation of probabilistic results.
  • Feature selection: Mutual information, correlation pruning, PCA. Can run a BPSO / GA search with`mealpy`
  • MLOps & monitoring: M*Lflow experiments and registry (aliases),Lakehouse Monitoring* (profile and drift metrics on Delta tables), SQL alerts, PSI / JS divergence
  • Handover: Writes runbooks and README-level documentation. Leaves a single end-to-end Workflow that someone else can run
Nice to have
  • Experience with AssureID / identity-verification JSON outputs.
  • PyOD and basic PyTorch training (to support A on the VAEs in week 2).
Responsibilities:
  • Platform setup. With customer's platform team, provision and configure the Databricks environment in customer's cloud account: catalog and permissions, compute (including GPU), secrets, experiment tracking.
  • Secure data ingestion. Build repeatable ingestion of raw verification outputs and document images: parsing, pseudonymisation, data-quality checks, schema handling.
  • Feature engineering at scale. Implement the feature tables: batch image-embedding jobs; exact and approximate linkage keys; cross-transaction and velocity aggregates; pseudonymous entity resolution.
  • Modelling support. Implement feature-selection and mixed-data clustering methods. Run model comparisons and stability tests, and support density-model training runs.
  • Automation. Orchestrate the end-to-end pipeline as a scheduled, idempotent, re-runnable workflow.
  • Monitoring. Set up the data-quality and drift-monitoring prototype, with alerting, on feature and score tables.
  • Privacy engineering. Implement access controls, provenance fields, lineage and the scripted deletion procedure. Confirm that no raw personal data reaches the analytic layer.
  • Documentation and handover. Write runbooks and give a live walkthrough, so engineers can operate everything independently
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

DataOps Consultant, Lakehouse Support
DataOps Consultant, Lakehouse Support

Luxoft • Poland

On-site
PLN 180,000 - 240,000
Private Medical & Dental care & Life-?
Internal Mobility program - rotation
...and more!
Senior MLOps Engineer (Databricks)
Senior MLOps Engineer (Databricks)

Grape Up • Polska

Hybrid
PLN 260,000 - 420,000
Data Engineer
Data Engineer

Sigma Software • Województwo mazowieckie

On-site
PLN 180,000 - 260,000
DataOps Consultant, Lakehouse Support
DataOps Consultant, Lakehouse Support

Luxoft • Polska

On-site
PLN 180,000 - 240,000
Azure Data Engineer
Azure Data Engineer

Aon plc • Kraków

Hybrid
PLN 160,000 - 260,000
Hybrid work model
Remote work option
Azure Data Engineer
Azure Data Engineer

Aon plc • Polska

Hybrid
PLN 180,000 - 260,000
Data Engineer
Data Engineer

CodiLime • Kraków, Wrocław, Poznań

Hybrid
PLN 180,000 - 240,000
Flexible working hours
Training budget
Senior Data Platform Engineer CR 12/11/26
Senior Data Platform Engineer CR 12/11/26

Margo Group • Warszawa

On-site
PLN 248,000 - 276,000
Senior Data Engineer
Senior Data Engineer

Sigmasoftware2 • Poland

On-site
PLN 180,000 - 280,000
Data/AI engineer with Python
Data/AI engineer with Python

Luxoft • Polska

On-site
PLN 240,000 - 360,000
Private medical insurance
Dental care
Life insurance
+1
Top locations