Data & AI Architect for Databricks: IoT & Time-Series

EPAM Systems; Sydney; On-site; AUD 190,000 - 260,000; Full time

Benefits offered by this job

Great Place To Work 2024

Mentoring programs

Online courses

Global opportunities

Job summary

EPAM Systems seeks a Data and AI Architect to design shared data and AI architecture for Agentic AI solutions on Databricks. You will shape secure ingestion, modeling and governance for IoT and time-series data and enable RAG retrieval patterns using Delta Lake, Unity Catalog, Spark and Vector Search.

You will collaborate across engineering, product and security teams to create a scalable, reusable foundation, with architecture decision records, reference designs and templates to guide

Qualifications

• Experience with Databricks in production environments.
• Expertise in data modeling for event, time-series, dimensional and canonical models.
• Proficient in building ingestion pipelines for batch and streaming data.
• Experience with IoT and time-series data including scale, retention, and query optimization.
• Delta Lake table design, performance tuning, and lifecycle management.
• Databricks Unity Catalog governance, lineage, and access controls.
• Strong Apache Spark skills for large-scale processing and optimization.
• Knowledge of data governance and security, with secure-by-default architectures.

Responsibilities

• Design shared data and AI architecture on Databricks for Agentic AI solutions.
• Define data modeling standards for IoT and time-series domains.
• Architect ingestion and processing pipelines using Spark and Delta Lake.
• Establish governance, lineage and access controls with Unity Catalog.
• Define secure data access patterns for internal teams and downstream consumption.
• Design RAG retrieval patterns and vector search indexing.
• Produce architecture decision records, reference designs and reusable templates.
• Collaborate with stakeholders to align scope, manage risks and drive quality.

Skills

Databricks architecture

Data modeling

Ingestion pipelines

IoT data

Time-series data

Delta Lake

Unity Catalog

Apache Spark

Data governance

Security by design

Tools

Delta Lake

Unity Catalog

Vector Search

Databricks

Job description

EPAM Systems seeks a Data and AI Architect to design shared data and AI architecture for Agentic AI solutions on Databricks. You will shape secure ingestion, modeling and governance for IoT and time-series data and enable RAG retrieval patterns using Delta Lake, Unity Catalog, Spark and Vector Search.

You will collaborate across engineering, product and security teams to create a scalable, reusable foundation, with architecture decision records, reference designs and templates to guide