Senior Data Engineer

Terminal; Colombia; Presencial; COP 120.000.000 - 180.000.000; Jornada completa

Descripción de la vacante

VolFront is seeking a senior data engineer to own the data-loader fleet, the loader repository, its scheduled tasks, the live-quote infrastructure, and the PostgreSQL schema behind it. You will transform product direction into a robust, observable, scalable pipeline.

You will design ingestion standards, manage partitioned tables, optimize FDW and indexing, monitor data quality, and mentor junior engineers within a fast-paced, Azure-based environment.

Formación

• 6+ years building and running production data systems with end-to-end ownership.
• Deep PostgreSQL knowledge: partitioning, indexing, 100M+ row tables, FDW, vacuum/storage health.
• Strong Python for architecting loaders.
• Experience on Windows Server and Linux.
• Data-integrity obsession; fail-loud discipline.
• Pub/Sub over polling for live data.
• Mentoring or guiding junior engineers.
• Overlap with US Central Time window for pre-market monitoring.

Responsabilidades

• Own the market-data pipeline end-to-end across Windows Azure batch VM, Linux quote-cache, and PostgreSQL layers.
• Set ingestion standards: schema design, idempotency, partitioning, backfill, error handling.
• Own and optimize the PostgreSQL data layer at scale (partitioned tables, indexing, vacuum, storage).
• Ensure overnight/evening batch runs land complete data before US market open.
• Lead data-ops monitoring and incident response with runbooks.
• Eliminate recurring failures and tighten durability and observability.
• Harden the fleet with row-count checks, freshness metrics, and robust alerts.
• Reduce toil through logging standardization, IaC, and automation.
• Design and ship new loaders and data sources as the product grows.
• Evolve live-quote architecture (in-memory cache, FDW, HTTP API) and push-delta design.

Conocimientos

End-to-end ownership

PostgreSQL expertise

Python

Windows & Linux

Data integrity

Pub/Sub

Monitoring

Self-directed

US Central overlap

Herramientas

Go

Python

PostgreSQL

Azure

Terraform

Linux

Windows Server

PowerShell

GitHub CI/CD

Descripción del empleo

About Volfront

Options analytics and intelligence consultancy. We combine decades of derivatives expertise with AI to build tools and solutions for volatility traders. Purpose-built by practitioners, for practitioners.

About Volfront

Options analytics and intelligence consultancy. We combine decades of derivatives expertise with AI to build tools and solutions for volatility traders. Purpose-built by practitioners, for practitioners.

About The Role

VolFront builds volAI, a conversational AI terminal for professional options traders. Behind the chat interface is a proprietary data layer: vol surfaces, IV-by-delta, realized vol, earnings analytics, dividend forecasts, option prints, OCC volume, fundamentals, and live quotes - all produced by an in-house fleet of data loaders. When a loader fails, traders see stale or missing data. Data freshness and correctness ARE the product - and our proprietary, in-house data is a core competitive moat that is constantly expanding with bespoke datasets you will not find anywhere else. The role You own the data-loader fleet - the loader repository, its scheduled tasks, the live-quote infrastructure, and the PostgreSQL schema behind it - as the senior engineer accountable for it. The founder sets product and domain direction and does not write the code; you turn that into a pipeline that is correct, current, observable, and continuously improving.

What You’ll Do

• Be the single accountable owner of the market-data pipeline: a large fleet of Windows scheduled tasks on an Azure batch VM (US Central time), systemd timers and Go loaders on a Linux quote-cache host, the sidecar Postgres + FDW layer, and the analytics schema they write to.
• Set the engineering standard for ingestion: schema design, idempotency, partitioning, backfill strategy, error handling, and data-contract discipline across every loader.
• Own the PostgreSQL data layer at scale: partitioned options tables in the 100M-plus-row range, indexing and query performance, vacuum/bloat and storage health, FDW pushdown, connection management. Operate (the floor, not the ceiling)
• Guarantee the overnight and evening batch runs land complete, fresh data before the US market open, every day.
• Direct first-line monitoring: a data-ops engineer triages the morning data-quality email and alert stream under your runbooks; you own the hard incidents, the root causes, and the systemic fixes.
• Drive recurring failure patterns to extinction (API quota exhaustion, IP/firewall changes, vendor schema drift, memory pressure, stuck processes) rather than re-fixing them each morning. Improve (the core of a senior seat)
• Harden the fleet so failures are loud and self-announcing: row-count assertions, freshness checks, dead-man switches, retry/backoff, heartbeat monitoring, better alert routing. A loader that silently writes zero rows and reports success is the bug class you are paid to eliminate.
• Reduce toil and snowflake risk: consolidate logging, standardize task wrappers, improve the centralized monitoring/remediation tooling, move manual steps into code, raise IaC coverage.
• Improve performance and cost: faster loads, leaner storage, fewer wasted compute hours. Build (a core part of the seat)
• Design and ship new loaders, new data sources, and bespoke in-house datasets as the product expands - this is how we widen the data moat, and it is a core, ongoing part of the seat, not an occasional project.
• Evolve the live-quote architecture (in-memory cache, sidecar Postgres + FDW, HTTP API, Go loaders) - pub/sub and push-delta by design, never polling.
• Own deploy safety for the data layer: staging, verification, rollback. Coordinate
• Direct the junior data-ops engineer and the interns on pipeline work.
• Interface with the app/LLM engineers on the read side (the application, its tools) so schema and contract changes never break the product. Tech you will own
• Python (all loaders), PostgreSQL on Azure at scale, partitioning, FDW, performance tuning, psql
• Windows Server: Task Scheduler, batch wrappers, PowerShell
• Linux (Ubuntu): systemd services/timers, journald, SSH via jump host
• Go (the live-quote loaders) - read fluently, extend comfortably
• Azure: VMs, networking/firewall, App Service, storage and scaling
• Git/GitHub workflow; CI/CD and IaC (Terraform or equivalent)
• Vendor data APIs: SpiderRock, AlphaVantage, OCC, EDGAR, news feeds
• AI coding tooling is used heavily for development and diagnostics - fluency working alongside it is expected

What You’ll Bring

• 6+ years building and running production data systems, with real end-to-end ownership (data engineering, backend, or platform/SRE).
• Deep PostgreSQL: partitioning, indexing strategy, query performance on 100M-plus-row tables, FDW, vacuum/bloat and storage health. You diagnose with EXPLAIN ANALYZE, not by guessing.
• Strong Python - you architect loaders, not just patch them.
• Comfortable owning systems on BOTH Windows Server and Linux.
• Data-integrity obsession: fail-loud discipline is a hard requirement. No silent fallbacks, no COALESCE-papering over missing inputs, no swallowed exceptions, no default values for required data. Works as designed or fails loud.
• Pub/sub over polling for anything live (quotes, chains, deltas).
• Monitoring instinct: you believe a job that silently writes zero rows and reports success is a worse bug than a crash, and you build the systems that catch it before a customer does.
• Self-directed senior: small shop, high autonomy, "just do it." You set the bar; you are not hand-held and you do not wait for process.
• Working-hours overlap with US Central Time, including attentiveness in the pre-market window (roughly 6:00-8:30 AM CT) when overnight failures must be caught and fixed – though as owner your job is to make that window quiet.
• Clear written English (design docs, incident notes, runbooks, change logs). Nice to have
• Market-data or finance feed experience (options, equities, OPRA, SpiderRock) – this is gold and cuts the domain ramp significantly.
• Streaming / real-time ingestion architecture.
• Azure specifically (we are all-in on Azure).
• Experience being the sole or primary owner of a production data platform.
• Mentoring or directing junior engineers. *This job posting exists to fill a vacancy.