PAUL Open: frozen-output model evaluation with human promotion gates
PAUL Open documents a human-directed research process connecting model-generated response pairs, automated diagnostics, blinded human comparison and checkpoint-promotion decisions. Its canonical v1 protocol distinguishes development feedback from sealed assurance. The dated20September2026 status retains SFT as reference and reports V6 not promoted, with reviewer-independence limits. This recommendation covers intended L2 model-evaluation methodology, not a deployed scientific research assistant or demonstrated model superiority.
L2Research workflow (workflow level, not a quality rating) Source: Repository-documented
Research applicability
A reusable model-generation/evaluation methodology is a concrete research process: AI responses form research data and humans direct comparisons, evidence boundaries and promotion decisions. Recommend this bounded method without claiming scientific-assistant deployment or reproducing reported outcomes.
Getting started
Untested editorial suggestion: read the pinned primary method documentation and specify a small non-sensitive task with researcher checks before separately considering execution.
Maintenance
Dated source snapshot only; present availability, maintenance and version parity require separate checks.
License
Source visibility is not reuse clearance; source/model/data licenses and rights require separate checking.
Workflow & human oversight
Scope: Precisely documented intended use only; no live execution or independent outcome validation Human control: Canonical protocol specifies reviewer blinding, immutable outputs, domain-appropriate human judgments and technical/contamination/preservation gates. Actual reviewer identity independence is explicitly unverifiable in reported H8; specification is not proof all controls operated. Stage connections: Researcher-directed training/generation and diagnostics feed blinded response review, locked analysis and a promotion decision. Model outputs are research data; no autonomous research planning is established. Stopping: Protocol describes review targets, collection closure and locked analysis before unblinding; candidate promotion requires more than preference wins. Dated status says V6 failed promotion and SHAE remains sealed. Missing evidence: No raw/private reviewer records, sealed data, training outputs or runtime were accessed; no independent repetition or proof of fully implemented v1 controls. Research-assistant capability targets are not deployed scientific-use evidence.
Workflow level is not an evidence score; higher is not better evidence.
Approach
- Models generate matched response data; automated validity/diagnostic checks prepare frozen comparisons for human assessment
Outcomes (as reported by the originating team)
Source reports and comparisons remain author-reported. No numerical benefit, superiority or independent reproduction is established by this review.
Limitations
- Repository documentation and dated status, not a fresh live model result. Wider private research programs are not included in this recommendation. - Model generation/training and reported human outcomes remain author-reported; technical training validity is not behavioural or scientific superiority. - DPO V2 diagnostic contamination is disclosed; H7 human collection discontinued without result; H8 reviewer identity independence is not technically verifiable. - Canonical v1 says engineered protocol with implementation following review; later H8 status does not prove every specified control. - SFT remains reference, V6 not promoted, sealed assurance not opened according to the dated status. No numeric performance/safety superiority claim adopted. - No training, model inference, form creation, reviewer contact, private data, sealed material, GPU launch or account action performed. - AI editorial review, not human editorial verification.
Reproducibility
AI editorial source review only. No candidate code, model, agent, experiment or reported outcome independently executed or reproduced.
Adoption requirements
Untested editorial suggestions: inspect source version, rights, data boundaries and human checks before any separately authorised adoption. This pass installs, invokes and contacts nothing.
Comments
No comments yet.
Add a comment
Comments are held for moderation before appearing. Critical or corrective comments are welcome — especially evidence that an entry overstates what has been verified.