Senior Data Engineer — Remote, End-to-End Data Pipelines

Svitla Systems, Inc.; Colombia; Híbrido; COP 200.541.462 - 300.812.193; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Christmas Bonus (50% of monthly pay)

Bonuses for writing/public talks

Personalized learning program

Free tech webinars and meetups

Remote-friendly culture

Team celebrations

Descripción de la vacante

Svitla Systems, Inc. is seeking an experienced Senior Data Engineer to own a Python/Scrapy-based legal data ingestion pipeline in Colombia and Mexico. The role combines large-scale data engineering, web scraping against complex sources, and cost-efficient cloud design, plus maintaining a React frontend.

Key tasks include building 50+ jurisdiction scrapers, preserving original document structure, and enabling robust search and change detection while optimizing resources.

Formación

• 5+ years in data engineering or backend development with production pipelines.
• Strong experience in Python with production web scraping and structured-data processing.
• Practical understanding of anti-bot measures within legal bounds.
• Understanding of relational and non-relational databases; schema design for hierarchical data.
• SQL and data modeling for change detection and display.
• Experience with cloud infrastructure (AWS, GCP, or Azure), IaC (Terraform/CloudFormation), and containers (Docker).
• Knowledge of Azure services (VMs, Blob storage, App Service, containers).
• Knowledge of search architectures (full-text, semantic search — Elasticsearch/OpenSearch).
• Familiarity with messaging/notification systems (email APIs, Slack/Teams).
• Infrastructure-as-code for automated deployments.
• Be comfortable owning a system end-to-end with minimal hand-holding.
• Ability to handle complex legacy systems with limited docs.
• Meticulous attention to detail for legal text processing.
• Cost- and efficiency-oriented mindset.
• Comfort with ambiguity across diverse data sources.

Responsabilidades

• Take over and understand an existing Python/Scrapy ingestion pipeline and codebase.
• Build and maintain web scrapers across 50+ jurisdictions with varied sources and barriers.
• Consolidate tooling to robust scraping capabilities; avoid sprawl.
• Normalize content into structured plain-text for hashing while preserving original HTML for display.
• Preserve source hierarchy (nested clauses, numbering, Unicode markers, tables).
• Maintain multi-destination architecture and flattened schema for search.
• Operate change detection and notifications, aiming for paragraph-level diffs.
• Design cost-efficient infrastructure with autoscaling and right-sizing.
• Maintain React/Vite frontend for regulations display.

Conocimientos

Python

Web scraping

SQL

Cloud platforms

IaC

Docker

Data pipelines

Legacy systems

Vulnerability to anti-bot measures

Schema design

Elasticsearch/OpenSearch

Message systems

Cost optimization

Team ownership

Herramientas

Terraform/CloudFormation

Docker

AWS

GCP

Azure

Elasticsearch/OpenSearch

Descripción del empleo

Svitla Systems, Inc. is seeking an experienced Senior Data Engineer to own a Python/Scrapy-based legal data ingestion pipeline in Colombia and Mexico. The role combines large-scale data engineering, web scraping against complex sources, and cost-efficient cloud design, plus maintaining a React frontend.

Key tasks include building 50+ jurisdiction scrapers, preserving original document structure, and enabling robust search and change detection while optimizing resources.