Databricks AI/ML Engineer — End-to-End Platform

Pyramid It Consulting; United States; Remote; USD 120,000 - 150,000; Full time

Job summary

Pyramid It Consulting seeks a skilled data platform professional to drive end-to-end Databricks-based data processing and AI/ML workloads. You will leverage PySpark for feature engineering and dataset preparation, and operate within a governed environment using Unity Catalog for centralized governance, auditing, and lineage.

The role emphasizes practical ML lifecycle skills, productionizing simple models/POCs, and delivering AI/ML workloads on a data platform.

Qualifications

• Experience with Databricks for AI/ML workloads.
• Proficiency in PySpark and Spark foundations.
• Familiarity with Unity Catalog and data governance.

Responsibilities

• Implement end-to-end Databricks-based data processing and AI/ML workloads.
• Develop feature engineering and dataset preparation pipelines.
• Productionize simple models/POCs on the platform and ensure reproducible experiments.
• Maintain governance, auditing, and lineage in a governed environment.

Skills

Databricks

PySpark

Spark foundations

Unity Catalog

AI/ML

Gen AI

NLP

Photon

Monitoring

Feature engineering

ML lifecycle

Productionizing models

Job description

Pyramid It Consulting seeks a skilled data platform professional to drive end-to-end Databricks-based data processing and AI/ML workloads. You will leverage PySpark for feature engineering and dataset preparation, and operate within a governed environment using Unity Catalog for centralized governance, auditing, and lineage.

The role emphasizes practical ML lifecycle skills, productionizing simple models/POCs, and delivering AI/ML workloads on a data platform.