Senior Data Engineer

Quartile; Brasil; Remote; BRL 180,000 - 360,000; Full time

Job summary

Quartile in Brazil seeks a Senior Data Engineer to own end-to-end data substrate—from ingesting messy third-party APIs to governance in Unity Catalog and a responsive serving layer.

You will implement infrastructure as code with Terraform, deploy through CI/CD, monitor quality, and ensure scalable, cost-conscious data platforms that support AI-driven applications.

Qualifications

• Strong Python and advanced SQL fundamentals for production pipelines.
• Hands-on Databricks experience (Spark/PySpark, Delta Lake, Unity Catalog).
• Experience designing production data pipelines and data models.
• Experience integrating third-party APIs at scale with proper auth and backfills.
• IaC (Terraform) and CI/CD experience.

Responsibilities

• Ingest data from diverse sources via REST APIs, Delta Sharing, and streams.
• Model Delta Lake layers with medallion architecture and governance.
• Build serving layer for AI agents and low-latency data stores.
• Own data quality, monitoring, and alerting with Grafana/OpenTelemetry.
• Ship infrastructure via Terraform and CI/CD without manual steps.
• Monitor performance and cost, optimize resources.

Skills

Python

Advanced SQL

Data modeling

Observability concepts

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Terraform

Azure

OpenTelemetry

Grafana

PostgreSQL

MongoDB

Delta Lake

Job description

The Senior Data Engineer owns that substrate end to end: ingestion from messy third-party APIs, modeling and governance in Unity Catalog, the serving layer agents query at runtime, and the health monitoring that tells us something broke before a CSM finds out in a client meeting. Infrastructure is code, deploys go through CI/CD, and quality is measured - not assumed. You write the Terraform, you own the alerts, you get paged by your own monitors.

REQUIREMENTS:

• Strong software engineering fundamentals in Python and advanced SQL - this role builds tested, deployed, version-controlled pipelines, not one-off notebooks
• Deep hands-on experience with Databricks: Spark/PySpark, Delta Lake, Unity Catalog, Workflows, and SQL warehouses, including performance tuning of real workloads
• Proven experience designing data pipelines and data models in production: medallion or equivalent layered architectures, incremental processing, CDC, and dimensional modeling
• Experience integrating third-party APIs at scale, with the operational maturity that implies (auth, pagination, throttling, partial failure, backfills, idempotency, schema evolution)
• Familiarity with cloud platforms (we run on Azure - ADLS, Container Apps, Key Vault, API Management, Entra ID)
• Experience with Infrastructure as Code (Terraform) and CI/CD pipelines, plus solid git and code-review habits
• A working definition of data quality: tests, expectations, monitoring, and the discipline to catch problems upstream rather than explaining them downstream
• Practical experience using AI coding assistants/agents in real engineering work, and the critical eye to know when their output is wrong
• Excellent problem-solving and analytical skills, and the autonomy expected of a senior engineer: you own a problem end to end - from framing to shipped, monitored outcome - and are accountable for the result, not just the merge
• Bachelor's degree or higher in Computer Science, Information Technology, or a related field - or equivalent practical experience

PREFERRED QUALIFICATIONS:

• Experience serving data to LLM-powered applications, including retrieval and context design
• Experience with operational stores alongside the lakehouse (Databricks Lakebase, PostgreSQL, MongoDB)
• Observability stacks: OpenTelemetry, Grafana, Loki, structured logging
• Delta Sharing for secure data exchange with clients and partners
• Domain background in retail media, digital advertising, or marketplace data
• Databricks or Azure certifications

WHAT YOU'LL DO:

• Design, build, and operate ingestion pipelines from advertising and business platforms (Amazon Ads, Google Ads, Walmart Connect, Criteo, Salesforce, and internal services) via REST APIs, Delta Sharing, object storage, and event streams
• Model curated data layers in Delta Lake under Unity Catalog: medallion architecture, incremental and CDC patterns, clear contracts between layers, lineage, and row/column-level governance
• Build the serving layer for AI: SQL views, Unity Catalog functions, and low-latency stores exposed to agents as tools through the Model Context Protocol (MCP) behind Azure API Management - designing for what an agent needs to reason well, not just for what a BI tool needs to render
• Own data quality and health monitoring as code: freshness, volume, schema-drift and business-rule expectations, with alerting wired into Grafana and a clear triage path when something fires
• Ship everything through Terraform and CI/CD - Databricks and Azure resources, jobs, permissions, and environment promotion from dev to prod, with no manual clicking in the console
• Own performance and cost: warehouse and cluster sizing, partitioning and liquid clustering, job orchestration in Databricks Workflows / Lakeflow, and continuous scrutiny of what each pipeline actually costs to run
• Work fluently with AI coding agents as part of your daily engineering practice - writing precise specs, keeping repository context and documentation in a state where agents produce good output, and reviewing what they generate with real judgment
• Collaborate with cross-functional teams - AI engineers, software engineers, and product - turning product requirements into data the platform can actually serve, and treating internal consumers as customers with contracts and expectations
• Instrument what you ship with OpenTelemetry and structured logging, and improve reliability and latency in production
• Document and maintain the codebase, ensuring code quality and adherence to best practices