Senior Data Engineer: AI-Powered Lakehouse on Databricks

Quartile; United States; À distance; USD 36 000 - 56 000; Plein temps

Résumé du poste

Sciene, operated within Quartile, seeks a Senior Data Engineer to own data substrate end to end—from ingestion to AI-driven serving layers. You will implement pipelines from ads, CRM, and internal services, model Delta Lake layers with Unity Catalog, and enable low-latency tools for autonomous agents.

You will deploy with Terraform, manage CI/CD, and drive data quality with strong observability. This Brazil-based PJ contract emphasizes end-to-end ownership and governance while collaborating

Qualifications

• Strong software engineering fundamentals in Python and advanced SQL
• Hands-on Databricks experience: Spark/PySpark, Delta Lake, Unity Catalog
• Production data pipelines with medallion architecture and CDC
• Experience integrating third-party APIs at scale (auth, pagination, backfills)
• Familiarity with Azure cloud services and Terraform
• CI/CD pipelines, Git and code review practices
• Data quality focus: tests, monitoring, upstream correctness
• Experience using AI coding assistants in engineering work
• Bachelor's degree or higher in CS/IT or related field

Responsabilités

• Design, build, and operate ingestion pipelines from ad and business platforms
• Model data layers in Delta Lake under Unity Catalog with governance
• Build serving layer for AI, exposing tools via APIs for agents
• Own data quality and health monitoring with alerting and debugging
• Ship infrastructure via Terraform and CI/CD for Databricks/Azure resources
• Manage performance and cost: warehouse sizing and orchestration
• Collaborate with AI engineers, software engineers, and product teams
• Instrument observability with OpenTelemetry and structured logging

Connaissances

Python

SQL

Databricks

Unity Catalog

Delta Lake

Terraform

CI/CD

REST APIs

Azure

OpenTelemetry

Grafana

PostgreSQL

MongoDB

Formation

Bachelor's degree in Computer Science / Information Technology or related field

Outils

Databricks

Azure

Terraform

PostgreSQL

MongoDB

OpenTelemetry

Grafana

Description du poste

Sciene, operated within Quartile, seeks a Senior Data Engineer to own data substrate end to end—from ingestion to AI-driven serving layers. You will implement pipelines from ads, CRM, and internal services, model Delta Lake layers with Unity Catalog, and enable low-latency tools for autonomous agents.

You will deploy with Terraform, manage CI/CD, and drive data quality with strong observability. This Brazil-based PJ contract emphasizes end-to-end ownership and governance while collaborating