Platform Engineer

UsefulBI Corporation; Alameda (CA); On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

Competitive compensation

Flexibility in work arrangements

Access to training and certifications

Comprehensive medical, dental, and vision coverage

Job summary

UsefulBI Corporation is looking for an experienced Data Platform Lead to architect an AI-ready data platform. The role requires deep expertise in AWS services and Databricks, along with strategic thinking to deliver governed datasets at scale.

You will oversee building and managing data products, ensure data governance, and work with AI and advanced analytics initiatives. A collaborative culture and opportunities for professional growth await qualified candidates.

Qualifications

• Deep expertise with AWS-native services such as S3, Athena, Redshift.
• Proficiency in Databricks and Delta Lake architecture.
• Hands-on expertise with data quality tooling like SODA and Great Expectations.

Responsibilities

• Design AWS Data Platform with security and scalability.
• Implement data governance for data lineage and trust.
• Architect data infrastructure for ML models and GenAI/RAG pipelines.

Skills

AWS services

Databricks

Python

SQL

Spark

Data architecture patterns

Data governance tools

ML models

Education

10+ years of experience in data architecture

AWS Certified Solutions Architect (Professional)

Tools

Apache Airflow

Data build tool (dbt)

Atlan

Tableau

Power BI

Job description

ROLE OVERVIE

WWe are seeking an experienced Data Platform Lead for designing and implementation of an enterprise grade, AI-ready data platform — spanning cloud storage, data engineering pipelines, governance-ready data products, and GenAI / RAG workloads. The ideal candidate brings deep expertise with AWS-native services, Databricks, and modern data stack tools, with a strategic mindset to architect platforms delivering governed, AI consumable datasets at scale

KEY RESPONSIBILITI

• ES1. Storage & Platform Foundati
• onArchitect the AWS Data Platform: Amazon S3, Athena, Redshift, and AWS Lake Formation for governed acce
• ssDesign and operate Databricks Lakehouse (Delta Lake, Unity Catalog, Databricks Comput
• e)Ensure secure, scalable, and cost-optimized foundational infrastructure aligned to enterprise SL
• erLead data engineering pipelines using dbt (transformations & modeling) and Databricks (ML Runtime, Notebook
• s)Implement Data Quality & Testing using SODA; design Workflow Orchestration with monitoring and alerti
• ngDrive a shift-left engineering culture with reusable, well-documented pipeline

3. Data Products Layer — AI-Ready Datase

• tsDefine and build domain-owned, AI-ready Data Products (Customer 360, Patient 360, Clinical, Commercial, Feature-Ready, and RAG-Ready dataset
• s)Champion a Data Mesh / Data Product mindset with clear ownership, SLAs, and discoverabili

ty4. Governance & Catalog Lay

• erImplement unified data governance using Atlan (Governance & Catalog) for discoverability, lineage, trust, and contr
• olEstablish end-to-end Data Lineage tracking (data to production) using classification taggi
• ngDefine and enforce Policies & Access controls: RBAC/ABAC, PII/PHI tagging, sensitive data classification, and policy enforceme
• ntDrive AI Governance practices: AI data usage policies, model lineage, bias detection, and audit readine
• ssEnsure Quality & Trust standards: data quality SLAs, certification workflows, and DQ dashboar

ds5. AI & Advanced Analytics Enableme

• ntArchitect data infrastructure supporting ML Models (predictive, prescriptive, optimization) and GenAI/RAG pipelin
• esDesign pipelines for LLM apps, copilots, and knowledge search; support real-time Decision Syste
• msCollaborate with domain teams to build Domain-Owned AI & ML models on governed, trusted da

ta6. Cross-Cutting Capabiliti

• esSecurity & Compliance: End-to-end data security, IAM, encryption, and audit-ready regulatory compliance (HIPAA, GDPR, SOC
• 2)Monitoring & Observability: Platform-level data and model observability, pipeline health dashboar
• dsCollaboration: Tools and practices to connect people, knowledge, and data across the organizati

REQUIRED SKILLS & QUALIFICATI

• rage10+ years of experience in data architecture and cloud platf
• ormsDeep expertise with AWS services: S3, Athena, Redshift, Lake Formation, Glue,
• IAMProficiency in Databricks: Delta Lake, Unity Catalog, Databricks Workflows, ML
• flowExperience with data lakehouse architecture patterns and medallion architecture (Bronze / Silver / G
• ationStrong hands-on expertise with dbt (data build tool) for SQL-based transformations and data modelingPipeline orchestration experience: Apache Airflow, Databricks Workflows, or equiv
• alentData quality tooling: SODA, Great Expectations, or Monte
• CarloProficiency in Python, SQL, and
• Spark

Spark Data Governance & C

• atalogExperience implementing data catalogs and governance tools (Atlan, Collibra, Alation, or equiv
• alent)Knowledge of data lineage, metadata management, classification fram
• eworksUnderstanding of RBAC/ABAC, PII/PHI regulations, and policy enforcement at the platform

level AI & GenAI Re

• adinessExperience building Feature Stores and ML-ready datasets for model training and inf
• erence.Familiarity with RAG (Retrieval-Augmented Generation) architecture: chunking, embedding generation, vector
• storesUnderstanding of LLMOps, model observability, and AI governance fra
• meworksExperience with vector databases (Pinecone, Weaviate, pgvector, or Databricks Vector
• adershipProven ability to design and document enterprise data architecture using frameworks like TOGAF or
• ZachmanStrong stakeholder engagement skills — ability to translate business needs into technical arch
• itectureExperience working in regulated industries (Life Sciences, Financial Services, Healthcare) is a str
• ong plusAWS Certified Solutions Architect (Professional) or Databricks Certified Professional p

referred NIC

• E TO HAVE Experience with ThoughtSpot, Tableau, or Power BI for self-service BI layer i
• ntegrationFamiliarity with SAS Viya or clinical analytics
• platformsKnowledge of FHIR, HL7, or CDISC data standards for Lif
• e SciencesContributions to open-source data engineerin
• g projectsExperience with multi-cloud or hybrid-cloud arc
• AT WE OFFER Opportunity to architect one of the most comprehensive AI-ready data platforms in

AT WE OFFER

• the industry
• Exposure to Fortune 500 clients across Life Sciences, Financial Services, and Technology
• AWS Generative AI Competency partner — work on cutting-edge GenAI and L
• LM use casesCollaborative, innovation-first culture with clear career
• growth pathCompetitive compensation, flexible work arrangements (remote/hybri
• d available)Access to certifications, training, and industry
• conferencesComprehensive Benefits: Medical, Dental, and Vision insurance coverage for employees and eligible dependents.Retirement Benefits: 401(k) retirement savings plan with company benefits as