Wilfred Morgan

The scoring pipeline runs two independent LLM evaluations per submission at temperature 0.0. If the two runs disagree beyond a threshold, the system flags for review rather than averaging — because averaging a 35 and an 85 into a 60 tells nobody anything useful. Each dimension carries a different weight: context architecture and evaluation quality are weighted highest as the strongest predictors of project success. The heuristic fallback scorer is the key architectural decision. When the LLM API is unavailable, a keyword-based scorer activates with a hard ceiling at L3 — it cannot assign L4 or L5 because keyword matching cannot assess nuanced comprehension. A degraded but honest score beats a confident but unreliable one. Report URLs use base64url encoding — no database, no storage backend. The URL IS the report.

Tasks route to models by cost-economics, not preference. Reasoning tasks requiring chain-of-thought go to the frontier model. Batch processing — metadata extraction, validation, classification — goes to the cheapest reliable model. This routing eliminated the most expensive early failure: applying a high-cost model to commodity tasks because nobody specified which to use. Context architecture is the core differentiator. 1000+ knowledge artifacts are organized into a structured retrieval layer queried at execution time. Embedding all context into prompts fails at scale — context windows are finite and retrieval precision degrades with volume. Six failure types with active mitigations: context degradation, specification drift, sycophantic confirmation, tool selection errors, cascading failures, and silent failures. Each has a detection pattern and automated prevention in production.