AIML Architect with Databricks AWS

Vytwo Technologies Inc.

Prosper (TX)

Remote

USD 180,000 - 280,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Vytwo is seeking an experienced AI/ML Architect with Databricks on AWS to design and implement scalable lakehouse platforms and multi-terabyte data pipelines. You will lead hands-on engineering and architecture, distill architectural significance from business needs, and drive end-to-end ML lifecycle management using Databricks, Python, PySpark, MLflow, and AWS-native services.

This is a remote role with strong collaboration across teams.

Qualifications

  • Bachelor’s or Master’s in Computer Science, Data Science, Engineering, Statistics, or related field.
  • 10+ years of experience in data engineering, ML engineering, or AI/ML architecture roles.
  • Deep expertise in Databricks on AWS, including PySpark/Spark SQL, Delta Lake, Unity Catalog, MLflow, Databricks Notebooks, Jobs & Workflows.
  • Strong programming ability in Python (pandas, numpy, scikit-learn).
  • Experience with large-scale, multi-terabyte data processing.

Responsibilities

  • Develop, train, and optimize ML models using Python, PySpark, MLflow, and Databricks Machine Learning.
  • Conduct exploratory data analysis to identify patterns and insights in large datasets.
  • Deploy ML models into production using MLflow, Databricks Workflows, or other MLOps pipelines.
  • Build analytics solutions such as forecasting, anomaly detection, segmentation, or recommendation systems.
  • Design ML architectures aligned with Databricks Lakehouse on AWS.

Skills

Databricks on AWS
Lakehouse design
PySpark
ML deployment
Large-scale pipelines
MLOps
Spark tuning

Education

Bachelor’s or Master’s in CS/DS/Engineering

Tools

Databricks Notebooks
Delta Lake
Unity Catalog
MLflow
Databricks Jobs/Workflows

Job description

Job Title: AI/ML Architect with Databricks , AWS

Remote

Role Overview

We are seeking an experienced AI/ML Architect with deep hands‑on expertise in Databricks on AWS to lead the design and implementation of scalable, high‑performance data and machine learning platforms. The ideal candidate combines architectural thinking with strong engineering execution, demonstrating the ability to build modern lakehouse systems, optimize large‑scale pipelines, and drive analytical and ML capabilities across the organization.

This role requires working with large, multi‑terabyte datasets, advanced analytics, and end‑to‑end ML lifecycle management using Databricks, Python, PySpark, and AWS‑native services.

Must Demonstrate (Critical Competencies)
  • Designing Databricks‑based lakehouse architectures on AWS (Delta Lake + S3 + Unity Catalog).
  • Clear separation of compute vs. serving layers in distributed architectures.
  • Low‑latency API strategy where Spark is insufficient (e.g., leveraging optimized services or caching).
  • Caching strategies to accelerate reads and reduce compute cost.
  • Data partitioning, file size tuning, and optimization strategies for large‑scale pipelines.
  • Experience handling multi‑terabyte structured time‑series workloads.
  • Ability to distill architectural significance from ambiguous business requirements.
  • Strong curiosity, questioning, and requirement‑probing mindset.
  • Player‑coach approach: hands‑on technical depth + ability to guide design.
Key Responsibilities
AI/ML & Advanced Analytics
  • Develop, train, and optimize ML models using Python, PySpark, MLflow, and Databricks Machine Learning.
  • Conduct exploratory data analysis (EDA) to identify patterns, trends, and insights in large datasets.
  • Deploy ML models into production using MLflow, Databricks Workflows, or other MLOps pipelines.
  • Build analytics solutions such as forecasting, anomaly detection, segmentation, or recommendation systems.
  • Design ML architectures aligned with Databricks Lakehouse on AWS.
Data Engineering & Lakehouse Architecture
  • Architect and build scalable ETL/ELT pipelines using PySpark, SQL, and Databricks Workflows.
  • Implement Delta Lake best practices, including OPTIMIZE, ZORDER, partitioning, and schema evolution.
  • Design lakehouse layers (Bronze/Silver/Gold) with strong separation of compute and serving layers.
  • Optimize cluster performance and jobs using Spark tuning, caching, and shuffle minimization.
  • Work with multi‑terabyte, time‑series, high‑velocity data in a distributed environment.
  • Ensure robust data availability for downstream ML and analytics workloads.
AWS Cloud Integration

Architect end-to-end data and ML solutions using AWS services, including:

  • S3 for storage
  • IAM for identity & access
  • Glue Catalog for metadata management
  • Networking for secure, high‑throughput data movement
  • Integrate Databricks with AWS-native compute, API layers, and low‑latency endpoints.
Business Collaboration & Leadership
  • Translate business problems into scalable analytical or ML architectures.
  • Communicate complex statistical and architectural concepts to non‑technical stakeholders.
  • Collaborate with product, engineering, and business leaders to drive data-informed initiatives.
  • Provide design leadership while remaining hands‑on in execution.
Skills & Qualifications
Required
  • Bachelor’s or Master’s in Computer Science, Data Science, Engineering, Statistics, or related field.
  • 10+ years of experience in data engineering, ML engineering, or AI/ML architecture roles.
  • Deep expertise in Databricks on AWS, including:
    • PySpark / Spark SQL
    • Databricks Notebooks
    • Delta Lake
    • Unity Catalog
    • MLflow
    • Databricks Jobs & Workflows
  • Strong programming ability in Python (pandas, numpy, scikit-learn).
  • Demonstrated experience with large‑scale, multi‑terabyte data processing.
  • Strong understanding of ML algorithms, distributed systems, and data optimization.
Preferred
  • Experience with MLOps and production deployment pipelines.
  • Strong grasp of AWS‑native data and compute services.
  • Understanding of CI/CD using GitHub Actions, GitLab CI, or similar.
  • Familiarity with deep learning frameworks (TensorFlow, PyTorch).
Key Competencies
  • Strong analytical and problem‑solving skills.
  • Ability to work in fast‑paced, highly collaborative environments.
  • Excellent communication and presentation abilities.
  • Self-driven with exceptional attention to architectural detail.

Flexible work from home options available.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI/ML Architect with Databricks , AWS
AI/ML Architect with Databricks , AWS

Vytwo Technologies Inc. • Dallas (TX)

Hybrid
USD 150,000 - 200,000
Flexible work from home options
AI/ML Architect with Databricks , AWS
AI/ML Architect with Databricks , AWS

Vytwo Technologies Inc. • Prosper (TX)

Remote
USD 180,000 - 240,000
Remote work options
Data Architect - Databricks & AWS
Data Architect - Databricks & AWS

KTek Resourcing • United States

Remote
USD 150,000 - 190,000
Databricks Solution Architect
Databricks Solution Architect

International Business Machines Corporation • Boston (KY)

Remote
USD 120,000 - 190,000
Databricks Solution Architect
Databricks Solution Architect

International Business Machines Corporation • Minneapolis (MN)

Remote
USD 150,000 - 185,000
AWS Databricks Architect
AWS Databricks Architect

HMG AMERICA LLC • United States

Remote
USD 140,000 - 210,000
Databricks Data Architect
Databricks Data Architect

Winfo Solutions • Atlanta (GA)

On-site
USD 140,000 - 210,000
Senior Data Architect
Senior Data Architect

Tredence Inc. • United States

On-site
USD 140,000 - 210,000
Senior Databricks Architect
Senior Databricks Architect

Valiant Technologies LLC • Atlanta (GA)

On-site
USD 170,000 - 210,000
Senior Databricks Architect
Senior Databricks Architect

Quantum Technologies LLC • Atlanta (GA)

On-site
USD 150,000 - 210,000