Data and AI Architect

EPAM Systems, Inc.; Sydney; On-site; AUD 180,000 - 250,000; Full time

Benefits offered by this job

Career development

Mentoring programs

Certification support

Global opportunities

Job summary

EPAM Systems, Inc. in Sydney seeks a Data and AI Architect to design shared data architecture for Agentic AI solutions on Databricks. You will shape secure ingestion, modeling and governance for IoT and time-series data, enabling RAG retrieval patterns with Delta Lake, Unity Catalog and Vector Search.

You will define scalable, reusable foundations and guide cross-functional teams in engineering, product and security to implement architecture decisions with risk management and quality at scale.

Qualifications

• Databricks architecture in production environments.
• Data modeling for IoT and time-series data.
• Ingestion pipelines across batch and streaming patterns.

Responsibilities

• Design shared data and AI architecture on Databricks for Agentic AI solutions.
• Define data modeling standards for IoT and time-series domains.
• Ingest and process pipelines using Spark and Delta Lake.
• Establish governance, lineage and access controls with Unity Catalog.
• Define secure data access patterns for internal teams and downstream consumption.
• Design RAG retrieval patterns including indexing for Vector Search.
• Produce architecture decision records and reusable templates.
• Partner with stakeholders to align scope, manage risks and drive implementation quality.

Skills

Databricks

Delta Lake

Unity Catalog

Spark

Vector Search

RAG retrieval

Data modeling

IoT data

Time-series data

Security governance

Job description

We are seeking a Data and AI Architect to design shared data architecture for Agentic AI solutions on Databricks. You will shape secure ingestion, modeling and governance for IoT and time-series data and enable RAG retrieval patterns using Delta Lake, Unity Catalog, Spark and Vector Search. You simplify complexity and guide teams toward scalable, reusable designs.You will define the shared data architecture for Agentic AI solutions built on Databricks. Your focus is creating a secure, scalable and reusable foundation for ingestion, modeling, governance and retrieval patterns that enable AI-ready data products.You bring a pragmatic systems mindset, and you can translate complex data and AI requirements into clear architecture decisions. You collaborate across engineering, product and security teams, challenge assumptions constructively and manage risks early.ResponsibilitiesDesign shared data and AI architecture on Databricks across Agentic AI solutionsDefine data modeling standards for IoT and time-series domains aligned to analytics and RAG needsArchitect ingestion and processing pipelines using Spark and Delta LakeEstablish governance, lineage and access controls with Unity CatalogDefine secure data access patterns for internal teams and downstream consumptionDesign RAG retrieval patterns including indexing strategy and datasets for Vector SearchProduce architecture decision records, reference designs and reusable templatesPartner with stakeholders to align scope, manage risks and drive implementation qualityRequirementsDatabricks architecture experience in production environmentsData modeling expertise including event, time-series, dimensional and canonical modeling patternsIngestion pipeline experience across batch and streaming patternsIoT and time-series data experience including scale, retention and query optimizationDelta Lake expertise including table design, performance tuning and lifecycle managementDatabricks Unity Catalog experience covering governance, lineage and fine-grained access controlApache Spark expertise for large-scale processing and optimizationData governance and security experience including policy design and secure-by-default architectureNice to haveExperience implementing Vector Search and managing embedding lifecycleFamiliarity with retrieval quality, evaluation signals and RAG tuning considerationsExperience enabling shared data products across multiple teamsWe offerBy choosing EPAM, you're getting a job at Great Place To Work-Certified in 2024, Glassdoor’s Top 100 Best Places to Work in 2023 & one of Most Loved Workplace, as recognized by Newsweek, 2021 - 2023.Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters.You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices.We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey.At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us.Life at EPAM- We are a leading global provider of digital platform engineering and development services that has rapidly grown in Australia since 2013.- We adopt a global growth strategy, thinking and acting like start-ups, working in multi-disciplinary teams, and delivering results. With a multinational talented team, we have been delivering innovative solutions for cloud migration, cyber security, digital transformation, etc.- You will work in an open and friendly work environment where your professional skills and experience can stand out.How We Hire• Here, we summarize the typical journey to finding a job within EPAM.• Apply and tell us about yourself!• Go through some standard interviews:- General interview with a recruiter- Technical interview with our technology experts- Manager interview or Offer interview with a hiring manager• Get ready to join the team!• Not sure if you meet all the requirements? No problem. Let's talk anyway and find out more!• It takes 1 min of application to start the journey with us. Apply now!