Evaluate models, guardrails and AI applications with spikee

Start by deciding what failure you want to observe and where it can happen. A model producing disallowed text, a classifier allowing an input, and an application performing an unauthorized action are different outcomes.

This study reads ReversecLabs/spikee, not just the older jhaddix fork. Source revision: 8f8e9ad3b684e5f3e2f9b507cae550bdc8372514, README v0.9.2, Apache-2.0; inspected 2026-10-23. Documentation and the canary implementation were read. Spikee was not installed or run, and no provider or target was called for this study.

Three evaluation contracts

Spikee's target adapter bridges a test to the system. The pinned custom-target contract uses Target.process_input(input_text, system_message, target_options). Model/application adapters return content; guardrail adapters return the allow/block boolean. An application adapter generally preserves the application's own system prompt instead of replacing it with the dataset's prompt. Validate the adapter against known outcomes before trusting a campaign. Target documentation.

For a synthetic refund agent, maintain separate observations: the model requested a refund; the tool accepted or denied the call; the transaction ledger changed or stayed unchanged. This effect-oracle design is our adaptation. A refusal after an unauthorized transaction does not undo the transaction; a confident success message without a transaction is not evidence that one occurred.

Build the case matrix before generating variants

Use PITAX to describe intent, technique, evasion and input. Add your system's principal, authority boundary and expected secure behavior. Taxonomy coverage is not proof of security.

A useful synthetic matrix includes:

These are proposed experiments, not generated attack payloads or observed results.

The pinned generator composes seeds or accepts standalone inputs. Seed text and generated content/content_type are different formats. full-prompt simulates a complete model task; user-input produces content for an application's own workflow. Transformations multiply cases without necessarily adding independent failure mechanisms. Inspect the generated cases and avoid leaking near-duplicates across development and holdout sets. Dataset documentation.

Calibrate the judge separately

The canary implementation checks a case-sensitive substring; a list means any alternative. It does not inspect the application's ledger, network or access logs.

If the attacker prompt already contains the canary, an echo is not proof of private retrieval. A refusal that includes a genuinely protected canary can still disclose it. Output-criteria judges need the facts necessary to evaluate the response. A syntactically valid verdict may still be wrong. Preserve false positives, false negatives and judge failures separately. Judge documentation.

Before testing a target, label a small response set manually: valid refusal, disclosure inside a refusal, harmless quotation, completed effect, claimed effect without evidence, malformed judge output and evaluator-directed text. Freeze the rubric before comparing judges. The upstream suite separates fixed-response/offline checks from explicitly enabled live inference and development from holdout cases; this study did not run either suite. Judge evaluation.

Control state, attempts and cost

Pin package commit, adapter hash, model/provider identifier, sampling parameters, system prompt version, dataset hash, judge/configuration and application revision. Record identity, tenant, tool permissions and approval policy in a private manifest.

The test command supports sampling, repeated attempts, thread limits, retries and resume. A dynamic attack can add work after an initial unsuccessful attempt; --attack-only skips that initial loop. A sampling seed is not a guarantee of deterministic provider output. Resume can preserve completed results from an older configuration: use separate result files for genuinely new comparisons. Testing documentation.

For multi-turn work, explicitly map test-session IDs to application sessions. Reset retrieval fixtures, conversation, memory and mutable tool state between independent cases. Removing a conversation turn does not necessarily undo an external action. Backtracking support must match actual target behavior; require a disposable lab or a reliable reset procedure. This full-state requirement extends the upstream session contract.

Start with one synthetic case and inspect the complete trace. Expand only after the adapter, oracle, reset and budget are verified. No execution permission or provider budget is implied by reading this guide.

Interpret results without hiding failures

Spikee's documented overall attack-success rate uses successful unique entries divided by total unique entries. Dynamic candidates are not extra independent dataset entries; attempt counts are not an exact billing or transport-request log. Preserve source-file provenance when combining runs. Results documentation.

Arithmetic example only: of 10 unique synthetic cases, suppose 2 succeed, 5 have valid unsuccessful outcomes and 3 time out. Overall success is 2/10 = 20%; the diagnostic rate among valid outcomes is 2/7 ≈ 28.6%. Neither figure makes the three timeouts secure outcomes. Report all counts, name both denominators and retain errors for investigation.

Guardrails need two runs: policy-violating inputs and legitimate inputs. Since true means allowed, allowed malicious inputs are misses and blocked benign inputs are false positives. Compare protection with usability; a guardrail that blocks everything is not a useful success. Guardrail workflow.

Report per family and trust boundary, not just a pooled score. State what was not covered. A finite sample, a provider update, unavailable traces or an unreliable judge limits conclusions. A source-reported score cannot be transferred to your application.

Campaign worksheet

This is a project planning worksheet, not a spikee configuration schema:

Complete one worksheet for each layer. Explain which observation would change your conclusion. That is a better first milestone than collecting a large list of payloads.

Next: use ask-cyber-specialist to challenge the plan, map-attack-surface to identify boundaries and the DASF methodology to select applicable risks and controls. This study does not add active testing to the wiki's read-only MCP.