Data Platform Lead / Cloud Systems Architect

GyanSys Brasil; Brasil; Presencial; BRL 500.000 - 900.000; Jornada completa

Descripción de la vacante

Odyssey is seeking a Data Platform Lead / Cloud Systems Architect to own enterprise analytics on AWS. You will architect multi-region Databricks deployments, enforce secure networking, and govern GenAI infrastructure, including MLflow and vector search.

You’ll drive cost discipline, vendor governance, and data mesh contracts to enable scalable, compliant analytics The role emphasizes reproducibility with Terraform and Databricks Asset Bundles, RBAC in Unity Catalog, and a bias-free approach to

Formación

• 8+ years in Data Engineering/Architecture with 3+ years in production using Databricks on AWS.
• Deep familiarity with GDPR, CCPA, HIPAA and PII protection in distributed data environments.
• Expert ELT/BI tools experience with Fivetran, AWS DMS and Sigma Computing.
• RBAC models in Unity Catalog and AWS IAM Identity Center.
• Proficient with AWS networking (Transit Gateway, VPC Endpoints, Private Subnets) and security.
• CI/CD experience via GitHub Actions and Databricks Asset Bundles.
• Vendor management, contracts, escalation and roadmap alignment.
• Decision-making with cost awareness and sunk-cost bias awareness.
• Databricks Certified Data Engineer Pro or AWS Solutions Architect Pro preferred.
• Knowledge of Generative AI concepts and ML techniques in data pipelines.
• Agile/scrum experience; Jira/Confluence.

Responsabilidades

• Architect and maintain a multi-region Databricks on AWS environment with secure networking.
• Design infrastructure for Generative AI, including MLflow governance and Databricks Vector Search.
• Enforce reproducibility by managing all resources via Terraform and DAB with no manual changes.
• Audit legacy choices with a clean slate approach and challenge the status quo.
• Lead FinOps reviews to differentiate necessary investments from sunk costs.
• Own vendor relationships (Databricks, AWS, Fivetran, Sigma) and unblock roadblocks.
• Drive Data Mesh by defining contracts, SLOs, and publication standards.
• Design privacy frameworks for GDPR/CCPA/SOC2; RTBF and PII masking workflows.
• Administer Sigma environment and promote assets Dev → Prod with proper tagging.
• Enforce RBAC across Unity Catalog and AWS IAM for least privilege.
• Manage high-volume data replication with Fivetran and AWS DMS; monitor schema drift.
• Code review pipelines for best practices (Z-ordering, schema evolution).
• Collaborate with analytics, data science, and business teams on scalable data solutions.
• Introduce emerging technologies to improve scalability and productivity.

Conocimientos

Data Engineering

Data Architecture

GenAI concepts

CI/CD

Terraform

GitHub Actions

RBAC

AWS networking

Educación

Databricks Certified Data Engineer Professional

AWS Solutions Architect Professional

Herramientas

Databricks on AWS

Terraform

DAB (Databricks Asset Bundles)

Fivetran

Sigma Computing

AWS DMS

Unity Catalog

GitHub Actions

Descripción del empleo

Summary / Primary Role:We are seeking a Data Platform Lead / Cloud Systems Architect to own Odyssey’s enterprise analytics ecosystem. You will manage everything from terraform deployments to our emerging GenAI infrastructure. Beyond technical execution, you will serve as a strategic guardian against engineering inertia, enforcing a discipline of rational adaptability to ensure investments are driven by objective future value rather than past expenditures.

Principal Duties and Responsibilities:

Architect and maintain a multi-region Databricks on AWS environment, enforcing "secure by default" networking via customer-managed VPCs, PrivateLink, Transit Gateway, and strict IAM cross-account roles.Design the underlying infrastructure for Generative AI, including configuring MLflow for model governance, Databricks Vector Search for RAG applications, and GPU-optimized compute clusters for inference.Enforce reproducibility by managing all platform resources (workspaces, clusters, jobs) via Terraform and Databricks Asset Bundles (DAB), ensuring no manual changes exist in Production.Systematically audit legacy technical choices with a "clean slate" perspective. You are expected to seek disconfirming evidence that contradicts the status quo to prevent confirmation bias and the escalation of commitment to outdated strategies.Lead monthly FinOps reviews using AWS Cost Explorer and Databricks system tables to differentiate between necessary investment and "sunk costs," ensuring we do not irrationally commit resources to failing projects simply because they are already underway.Own strategic vendor relationships (Databricks, AWS, Fivetran, Sigma), holding partners accountable for successful outcomes and resolving support blockers aggressively rather than passively accepting roadmap delays.Drive a Data Mesh culture by treating data assets as "Data Products." Define clear contracts, SLOs (Service Level Objectives), and publication standards to decouple producers from consumers.Design and operationalize data privacy frameworks to satisfy GDPR, CCPA, and SOC2 requirements, including automated workflows for "Right to Be Forgotten" (RTBF) requests and PII masking within the Lakehouse architecture.Administer the Sigma Computing environment, overseeing workspace architecture, version tagging strategies, and the promotion path of analytics assets from Dev to Production.Design and enforce comprehensive Role-Based Access Control (RBAC) policies across Unity Catalog (Catalogs, Schemas, Tables) and AWS IAM, ensuring "least privilege" access while maintaining operational velocity.Manage high-volume data replication pipelines using Fivetran and AWS DMS, ensuring data fidelity, efficient schema drift handling, and cost-optimized sync frequencies across heterogeneous sources.Perform rigorous code reviews on high-impact pipelines, enforcing best practices such as Z-ordering, Liquid Clustering, and Schema Evolution, with the authority to reject sub-standard code.Partner with analytics, data science, and business teams to design scalable data solutions that align to strategic priorities and unlock advanced use cases.Drive innovation by introducing emerging technologies, frameworks, and best practices into engineering workflows to improve scalability, automation, and productivity.

Qualifications and Key Skills:

8+ years in Data Engineering/Architecture, with 3+ years of daily production mastery in Databricks on AWS.Deep familiarity with global data compliance standards (GDPR, CCPA, HIPAA) and experience implementing technical controls for PII protection (e.g., dynamic views, column-level encryption) in a distributed data environment.Expert-level proficiency with modern ELT and BI tools, specifically Fivetran (connector configuration, transformation), AWS DMS, and Sigma Computing (administration, version control, security).Deep understanding of Role-Based Access Control (RBAC) models within distributed data systems, specifically regarding Unity Catalog grants and AWS IAM Identity Center.Deep expertise in AWS networking (Transit Gateway, VPC Endpoints, Private Subnets) and security (SCPs, Encryption, IAM identity center).Proven production experience deploying CI/CD pipelines via GitHub Actions and Databricks Asset Bundles (DAB).Demonstrated experience managing enterprise software vendors, including contract utilization, technical escalation, and roadmap alignment.A track record of making high-stakes technical decisions, including examples of when you recommended stopping a project to save resources (managing Sunk Cost bias).Databricks Certified Data Engineer Professional (strongly preferred) or AWS Solutions Architect Professional.Familiarity with Generative AI concepts and modern ML techniques (LLMs, transformers, ensemble methods, deep learning frameworks) and how they integrate into data engineering workflows.Working knowledge of agile/scrum methodologies and use of agile lifecycle management tools (e.g., Jira, Confluence).English level: advanced. 8+ years in Data Engineering/Architecture, with 3+ years of daily production mastery in Databricks on AWS., Deep familiarity with global data compliance standards (GDPR, CCPA, HIPAA) and experience implementing technical controls for PII protection (e.g., dynamic views, column-level encryption) in a distributed data environment., Expert-level proficiency with modern ELT and BI tools, specifically Fivetran (connector configuration, transformation), AWS DMS, and Sigma Computing (administration, version control, security)., Deep understanding of Role-Based Access Control (RBAC) models within distributed data systems, specifically regarding Unity Catalog grants and AWS IAM Identity Center., Deep expertise in AWS networking (Transit Gateway, VPC Endpoints, Private Subnets) and security (SCPs, Encryption, IAM identity center)., Proven production experience deploying CI/CD pipelines via GitHub Actions and Databricks Asset Bundles (DAB)., Demonstrated experience managing enterprise software vendors, including contract utilization, technical escalation, and roadmap alignment., A track record of making high-stakes technical decisions, including examples of when you recommended stopping a project to save resources (managing Sunk Cost bias)., Databricks Certified Data Engineer Professional (strongly preferred) or AWS Solutions Architect Professional., Familiarity with Generative AI concepts and modern ML techniques (LLMs, transformers, ensemble methods, deep learning frameworks) and how they integrate into data engineering workflows., Working knowledge of agile/scrum methodologies and use of agile lifecycle management tools (e.g., Jira, Confluence)., English level: advanced.