Databricks GenAI Engineer — ML on Spark Platform

Pyramid It Consulting; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Pyramid It Consulting seeks an experienced data platform professional with strong Databricks expertise. You will execute end-to-end data/AI workflows, perform feature engineering with PySpark, and prepare datasets for production use while leveraging Spark/Photon components.

You will operate within a governed environment using Unity Catalog, implement ML lifecycle steps, and prototype/pilot simple models, delivering AI/ML workloads on the Databricks platform.

Qualifications

• Experience delivering AI/ML workloads on a data platform with Databricks.
• Familiarity with Databricks capabilities like AI/ML, Gen AI and NLP.
• Comfort operating in a governed environment with Unity Catalog.

Responsibilities

• Strong Databricks platform skills for end-to-end execution (Databricks used for query/processing and supports AI/ML capabilities in the target-state architecture).
• PySpark / Spark foundations for feature engineering / dataset preparation.
• Familiarity with Databricks capabilities noted in internal material: AI & ML, Gen AI, NLP, plus platform elements like Spark/Photon and monitoring.
• Comfort operating in a governed environment (Unity Catalog for centralized governance, auditing/lineage).
• Practical ML lifecycle skills (feature engineering, evaluation discipline, reproducible experimentation) and ability to productionize simple models/POCs on the platform.
• Demonstrated experience delivering AI/ML workloads on a data platform (Databricks preferred).

Skills

Databricks

PySpark

Spark foundations

Gen AI

NLP

Unity Catalog

ML lifecycle

Model productionization

AI/ML workloads

Monitoring & governance

Tools

Databricks

Photon

Job description

Role & responsibilities

Strong Databricks platform skills for end-to-end execution (Databricks used for query/processing and supports AI/ML capabilities in the target-state architecture).
PySpark / Spark foundations for feature engineering / dataset preparation.
Familiarity with Databricks capabilities noted in your internal material: AI & ML, Gen AI, NLP, plus platform elements like Spark/Photon and monitoring.
Comfort operating in a governed environment (Unity Catalog for centralized governance, auditing/lineage).
Practical ML lifecycle skills (feature engineering, evaluation discipline, reproducible experimentation) and ability to productionize simple models/POCs on the platform.
Demonstrated experience delivering AI/ML workloads on a data platform (Databricks preferred).