Wilfred Morgan

The scoring pipeline runs two independent LLM evaluations per submission at temperature 0.0. If the two runs disagree beyond a threshold, the system flags for review rather than averaging — because averaging a 35 and an 85 into a 60 tells nobody anything useful. Each dimension carries a different weight: context architecture and evaluation quality are weighted highest as the strongest predictors of project success. The heuristic fallback scorer is the key architectural decision. When the LLM API is unavailable, a keyword-based scorer activates with a hard ceiling at L3 — it cannot assign L4 or L5 because keyword matching cannot assess nuanced comprehension. A degraded but honest score beats a confident but unreliable one. Report URLs use base64url encoding — no database, no storage backend. The URL IS the report.

Tasks route to models by cost-economics, not preference. Reasoning tasks requiring chain-of-thought go to the frontier model. Batch processing — metadata extraction, validation, classification — goes to the cheapest reliable model. This routing eliminated the most expensive early failure: applying a high-cost model to commodity tasks because nobody specified which to use. Context architecture is the core differentiator. 1000+ knowledge artifacts are organized into a structured retrieval layer queried at execution time. Embedding all context into prompts fails at scale — context windows are finite and retrieval precision degrades with volume. Six failure types with active mitigations: context degradation, specification drift, sycophantic confirmation, tool selection errors, cascading failures, and silent failures. Each has a detection pattern and automated prevention in production.

Two heaviest dimensions use an LLM judge; the rest are deterministic. The core engine is provider-agnostic — a CI structural guard fails the build if a provider SDK or web framework imports into core. Derived reports quote the corpus only in capped spans (≤15 words / 120 chars) under a TTL, so the manifest can name documents and claims without persisting source text. Hosted scoring runs as a token-gated Cloud Function; capture is a separate Netlify function.

Deterministic by design — no LLM and no upload, so identical inputs always produce the same band. A readiness diagnostic that isn't itself a black box. Seven dimensions, three level-anchored questions each; the respondent selects the anchor that matches reality rather than free-texting. Standards lineage is explicit: ISO/IEC 42001 as the structural anchor, NIST AI RMF 1.0 plus the GenAI Profile for risk vocabulary.