Project / Case study

PRISM: Prompt Reliability Through Semantic Multiplexing

PRISM studies prompt sensitivity in educational multiple-choice question answering using local Small Language Model inference through Ollama on CPU-only hardware.

At a glance

active

  • Python
  • Ollama
  • Small language models
  • ARC-Challenge
  • SciQ
View the repository ↗
PRISM prompt reliability benchmark title graphic

01 / Research question

What it asks

Does a Small Language Model change its behavior when the same underlying question is expressed through different controlled prompt formulations?

02 / Experimental design

Scope

The first experiment freezes 400 science questions from ARC-Challenge and SciQ, then evaluates four local models across five prompt conditions.

03 / Evaluation

How it measures

The pipeline separates inference from evaluation and measures correctness, instruction adherence, cross-prompt consistency, prompt sensitivity, and potentially unreliable response patterns.