Senior Data Engineer

Quartile; United States; Remote; USD 36,000 - 56,000; Full time

Job summary

Sciene, operated within Quartile, seeks a Senior Data Engineer to own data substrate end to end—from ingestion to AI-driven serving layers. You will implement pipelines from ads, CRM, and internal services, model Delta Lake layers with Unity Catalog, and enable low-latency tools for autonomous agents.

You will deploy with Terraform, manage CI/CD, and drive data quality with strong observability. This Brazil-based PJ contract emphasizes end-to-end ownership and governance while collaborating

Qualifications

• Strong software engineering fundamentals in Python and advanced SQL
• Hands-on Databricks experience: Spark/PySpark, Delta Lake, Unity Catalog
• Production data pipelines with medallion architecture and CDC
• Experience integrating third-party APIs at scale (auth, pagination, backfills)
• Familiarity with Azure cloud services and Terraform
• CI/CD pipelines, Git and code review practices
• Data quality focus: tests, monitoring, upstream correctness
• Experience using AI coding assistants in engineering work
• Bachelor's degree or higher in CS/IT or related field

Responsibilities

• Design, build, and operate ingestion pipelines from ad and business platforms
• Model data layers in Delta Lake under Unity Catalog with governance
• Build serving layer for AI, exposing tools via APIs for agents
• Own data quality and health monitoring with alerting and debugging
• Ship infrastructure via Terraform and CI/CD for Databricks/Azure resources
• Manage performance and cost: warehouse sizing and orchestration
• Collaborate with AI engineers, software engineers, and product teams
• Instrument observability with OpenTelemetry and structured logging

Skills

Python

SQL

Databricks

Unity Catalog

Delta Lake

Terraform

CI/CD

REST APIs

Azure

OpenTelemetry

Grafana

PostgreSQL

MongoDB

Education

Bachelor's degree in Computer Science / Information Technology or related field

Tools

Databricks

Azure

Terraform

PostgreSQL

MongoDB

OpenTelemetry

Grafana

Job description

About Sciene

At Sciene, the mission is to empower professional services firms with cutting‑edge, customized AI solutions — enhancing automation, analytics, and optimization across industries while prioritizing security, cost efficiency, and state‑of‑the‑art technology.

Our flagship product, the Sciene AI Companion , is an autonomous customer success platform deployed across Quartile — the world’s largest retail media optimization platform, managing performance marketing for 1,000+ brands. It automates relationship‑heavy enterprise workflows end to end: generating personalized email replies in the CSM's own voice (8x faster), building full presentation decks for client meetings (12x faster), and detecting and diagnosing account fluctuations before anyone has to ask (6x faster). None of this replaces human judgment — it removes the work that was getting in the way of it.

Read more about how we built it: Sciene AI Companion: Building an Autonomous Customer Success Platform on Databricks

OVERVIEW:

Every answer the AI Companion gives is only as good as the data underneath it. Sciene runs a production lakehouse on Databricks and Azure that ingests advertising, CRM, and conversational data from a dozen upstream systems, curates it under Unity Catalog governance, and serves it back to autonomous agents as tools — not as dashboards.

The Senior Data Engineer owns that substrate end to end: ingestion from messy third‑party APIs, modeling and governance in Unity Catalog, the serving layer agents query at runtime, and the health monitoring that tells us something broke before a CSM finds out in a client meeting. Infrastructure is code, deploys go through CI/CD, and quality is measured — not assumed. You write the Terraform, you own the alerts, you get paged by your own monitors.

This is also a role where you work with AI, not just for it. Our engineers ship with coding agents (Claude Code and our MCP‑integrated toolchain) as part of the normal workflow, and your consumer is an autonomous agent making decisions on top of what you produce — which raises the bar on correctness, freshness, and how the data is shaped.

REQUIREMENTS:

• Strong software engineering fundamentals in Python and advanced SQL — this role builds tested, deployed, version‑controlled pipelines, not one‑off notebooks
• Deep hands‑on experience with Databricks: Spark/PySpark, Delta Lake, Unity Catalog, Workflows, and SQL warehouses, including performance tuning of real workloads
• Proven experience designing data pipelines and data models in production: medallion or equivalent layered architectures, incremental processing, CDC, and dimensional modeling
• Experience integrating third‑party APIs at scale, with the operational maturity that implies (auth, pagination, throttling, partial failure, backfills, idempotency, schema evolution)
• Familiarity with cloud platforms (we run on Azure — ADLS, Container Apps, Key Vault, API Management, Entra ID)
• Experience with Infrastructure as Code (Terraform) and CI/CD pipelines, plus solid git and code‑review habits
• A working definition of data quality: tests, expectations, monitoring, and the discipline to catch problems upstream rather than explaining them downstream
• Practical experience using AI coding assistants/agents in real engineering work, and the critical eye to know when their output is wrong
• Excellent problem‑solving and analytical skills, and the autonomy expected of a senior engineer: you own a problem end to end — from framing to shipped, monitored outcome — and are accountable for the result, not just the merge
• Bachelor's degree or higher in Computer Science, Information Technology, or a related field — or equivalent practical experience

PREFERRED QUALIFICATIONS:

• Experience serving data to LLM‑powered applications, including retrieval and context design
• Experience with operational stores alongside the lakehouse (Databricks Lakebase, PostgreSQL, MongoDB)
• Observability stacks: OpenTelemetry, Grafana, Loki, structured logging
• Delta Sharing for secure data exchange with clients and partners
• Domain background in retail media, digital advertising, or marketplace data
• Databricks or Azure certifications

WHAT YOU'LL DO:

• Design, build, and operate ingestion pipelines from advertising and business platforms (Amazon Ads, Google Ads, Walmart Connect, Criteo, Salesforce, and internal services) via REST APIs, Delta Sharing, object storage, and event streams
• Model curated data layers in Delta Lake under Unity Catalog: medallion architecture, incremental and CDC patterns, clear contracts between layers, lineage, and row/column‑level governance
• Build the serving layer for AI: SQL views, Unity Catalog functions, and low‑latency stores exposed to agents as tools through the Model Context Protocol (MCP) behind Azure API Management — designing for what an agent needs to reason well, not just for what a BI tool needs to render
• Own data quality and health monitoring as code: freshness, volume, schema‑drift and business‑rule expectations, with alerting wired into Grafana and a clear triage path when something fires
• Ship everything through Terraform and CI/CD — Databricks and Azure resources, jobs, permissions, and environment promotion from dev to prod, with no manual clicking in the console
• Own performance and cost: warehouse and cluster sizing, partitioning and liquid clustering, job orchestration in Databricks Workflows / Lakeflow, and continuous scrutiny of what each pipeline actually costs to run
• Work fluently with AI coding agents as part of your daily engineering practice — writing precise specs, keeping repository context and documentation in a state where agents produce good output, and reviewing what they generate with real judgment
• Collaborate with cross-functional teams — AI engineers, software engineers, and product — turning product requirements into data the platform can actually serve, and treating internal consumers as customers with contracts and expectations
• Instrument what you ship with OpenTelemetry and structured logging, and improve reliability and latency in production
• Document and maintain the codebase, ensuring code quality and adherence to best practices

*This is a PJ contract based in Brazil.