Databricks Engineer with AI/ML

Pyramid It Consulting; United States; Remote; USD 120,000 - 150,000; Full time

Job summary

Pyramid It Consulting seeks a skilled data platform professional to drive end-to-end Databricks-based data processing and AI/ML workloads. You will leverage PySpark for feature engineering and dataset preparation, and operate within a governed environment using Unity Catalog for centralized governance, auditing, and lineage.

The role emphasizes practical ML lifecycle skills, productionizing simple models/POCs, and delivering AI/ML workloads on a data platform.

Qualifications

• Experience with Databricks for AI/ML workloads.
• Proficiency in PySpark and Spark foundations.
• Familiarity with Unity Catalog and data governance.

Responsibilities

• Implement end-to-end Databricks-based data processing and AI/ML workloads.
• Develop feature engineering and dataset preparation pipelines.
• Productionize simple models/POCs on the platform and ensure reproducible experiments.
• Maintain governance, auditing, and lineage in a governed environment.

Skills

Databricks

PySpark

Spark foundations

Unity Catalog

AI/ML

Gen AI

NLP

Photon

Monitoring

Feature engineering

ML lifecycle

Productionizing models

Job description

Role & responsibilities

Strong Databricks platform skills for end-to-end execution (Databricks used for query/processing and supports AI/ML capabilities in the target-state architecture).
PySpark / Spark foundations for feature engineering / dataset preparation.
Familiarity with Databricks capabilities noted in your internal material: AI & ML, Gen AI, NLP, plus platform elements like Spark/Photon and monitoring.
Comfort operating in a governed environment (Unity Catalog for centralized governance, auditing/lineage).
Practical ML lifecycle skills (feature engineering, evaluation discipline, reproducible experimentation) and ability to productionize simple models/POCs on the platform.
Demonstrated experience delivering AI/ML workloads on a data platform (Databricks preferred).