Methodology

1. Deterministic selection & Eligibility Gate

A product candidate is discovered daily from structured web platforms rotating by day of the week:

• Monday: HN Buzz — Hacker News top stories (mainstream attention)
• Tuesday: GitHub New & Rising — repositories created within the last 14 days (20–3,000 stars), ranked by star velocity (stars per day)
• Wednesday: Show HN Launches — projects announced by their authors
• Thursday: Hugging Face Rising — Spaces created within the last 6 months (20–5,000 likes), ranked by like velocity
• Friday: Emerging Developer Tools — developer-tools repositories created within a year and pushed within 30 days (20–5,000 stars), ranked by star velocity
• Saturday: Hidden Gems — active repositories with 10–500 stars, created within a year and pushed within 30 days, ranked by star velocity
• Sunday: Cross-source Momentum — candidates appearing on both Hacker News (Top or Show HN) and the GitHub New & Rising pool

Rather than choosing the most popular item, each source ranks its candidates by a discovery signal — recency-bounded star or like velocity (growth per day) for GitHub and Hugging Face pools, and community rank for Hacker News pools — deliberately favoring young, rising projects over already-established ones. Candidates are then filtered in that order against the strict **Open Source Eligibility Gate (Selection Policy v2)**. Only the first candidate that satisfies all eligibility criteria is selected for evaluation.

• Public Source: A canonical, public source repository must exist (no README-only empty repos).
• Open-source License: Must have an explicit license normalizable to an approved SPDX identifier (e.g. MIT, Apache-2.0).
• Clear Purpose: Stated purpose and target users/use cases.
• Runnable or Reproducible: Presence of public demo, release, package registry, container image, or clear installation instructions.
• Freshness: Meaningful commits or releases in the past 18 months.

Popularity metrics (such as stars or points) are used solely for deterministic candidate discovery and selection; they do not affect the Jury Score.

JuryPress has two selection paths: the Autonomous Daily Selection described above, and an Operator-approved Reader Request path. Readers submit a request from the Request a Review page as a public GitHub Issue using their own GitHub account, and an operator explicitly starts the evaluation. A reader request goes through the exact same Eligibility Gate, evidence collection, rubric, and quality validation as a daily selection. The text of the request itself (the issue body and any comments) is never used as evaluation evidence — only official sources re-collected by the evidence collector are. Being requested by a reader has no effect on the Jury Score in either direction, and requested reviews are marked with their originating issue on the article page.

2. Evidence collection

Evidence is gathered primarily from the official product source (repository, landing page, API metadata) and the source discussion thread. Third-party reviews are strictly excluded.

3. Evidence records

How a review's statements are tied to the evidence changed in July 2026, and each review states which pipeline produced it (see “Generation metadata” on the article). Both are described here, because we would rather be accurate about two methods than tidy about one.

Reviews published from July 2026 onward

The jury writes the review first, as a work of evaluation and judgment. Afterwards, a separate pass reads the finished article alongside the evidence and records how each statement relates to it — supported by the material, resting on the creator's own description, drawn from community discussion, inferred, or simply the jury's own opinion. Two further outcomes are recorded honestly when they occur: a factual statement the collected evidence does not cover, and a statement the evidence appears to contradict.

That record is an appendix at the foot of the review, not a grade of it. A review is expected to contain opinions, comparisons and conclusions that no single source states — that is what makes it a review rather than a summary. The record does not gate publication, and a review may publish while its evidence map is unavailable; the appendix says so when that is the case. What the jury is not permitted to do is invent specifics: statistics, file names, test results or capabilities absent from the material.

Reviews published before July 2026

Earlier reviews were generated under a stricter regime in which every sentence carried its provenance annotation at the moment of writing, using the classificationssource_confirmed, creator_claim, inference,unknown and runtime_observed. Those articles are unchanged and still display their original per-sentence record.

In both eras, every criterion receives a confidence rating (high, medium, low, or not assessable), and every review lists each source with its URL, type and retrieval time.

The evidence map gates rankings, never publication. A review without a complete, loadable map stays published and readable, and is markedHistorical methodology rather than ranked against reviews produced under the current standard — ranking positions are only meaningful between reviews held to the same record-keeping. That mark describes the record, not the review's conclusions; substantive errors are handled separately, as corrections shown on the review itself. Which reviews this affects, and why the standard changed, is set out in the Methodology History.

Corrections, and what is not corrected

JuryPress corrects errors of fact. It does not revise the jury's opinions. A correction is warranted when a review states, as fact, an externally checkable proposition that is wrong:

• it contradicts the collected evidence or the project's official documentation;
• it presents as established something no evidence supports;
• it carries one part of a compound claim at the strength of another — a supported clause lending its confidence to an unsupported one;
• it declares an absence (“there are no tests”, “there is no issue tracker”) that was never verified, only not found;
• it gets a specific wrong: a figure, a capability, an authentication method, a supported platform.

A correction is not warranted for the jury's judgment, however pointed. Calling a project “open source in license only”, “a code-drop”, or “an extraction vector” is an opinion the review is entitled to hold; the evidence map records it as editorial judgment, not as a claim about the world, and it is not corrected because a reader disagrees with it. Correcting opinion would mean editing the jury rather than keeping the record honest — the distinction JuryPress is built to hold.

When a correction is warranted it is recorded under the review's Corrections and shown on the page; the original text stays, so the correction reads as a correction. When the error is broad enough that the verdict itself cannot stand — the evaluation rested on evidence it should not have — the review is withdrawn from rankings instead of quietly rewritten, and re-reviewed on a fuller evidence base.

4. Five-person evaluation

To balance professional biases, the evaluation coordinates five simulated professional perspectives (personas) evaluating the product simultaneously. JuryPress utilizes structured output engines to generate all five reviews simultaneously in a single, atomic API call. This ensures that all simulated perspectives evaluate the exact same evidence container under identical temperature and token budget parameters.

JuryPress uses the five personas and current rubric defined by Judgie-AI.

JuryPress runs its own automated publishing pipeline; it does not claim that Judgie-AI independently reviewed or endorsed each article.

5. Code-side score calculation & Not Assessable

The AI provides raw scores (0–5) for each criterion. Weighted scores and the Jury Score are recalculated server-side using fixed rubric weights. The AI cannot influence the final score beyond the raw criterion scores.

If a criterion lacks sufficient evidence for a complete assessment, it is rated "not assessable" and receives a null score. If any criterion is unassessable, the product's overall Jury Score is set to null, and the review is marked as **Unranked** due to insufficient evidence.

6. Automatic publication

Most articles are built as a static page and deployed exactly as generated, with no human editing. When an article fails the quality checks, it is withheld rather than published; an editor may then correct the article text and resubmit it through the same quality checks before it can be published. In both cases the jury scores and the per-criterion scores are fixed at generation and cannot be changed by a human.

• Model: Current production model: gemini-3.5-flash
• Season Version: Season 2 — Open-Source Products
• Autonomously generated: Gemini output published as-is, no human editing.
• AI-generated, human-edited: withheld after a quality failure, then human-edited and re-checked. Scores unchanged.
• Scores: Never human-editable in either case, and both pass the same quality checks.

7. Failure handling

JuryPress is response-first: as soon as a response is returned by the model, it is persisted to the Generation Record before any quality judgement is made. The saved response is then judged exactly once.

• Persist first: the model response is written to the Generation Record as soon as it is received, ahead of validation.
• Judged once: the quality checks (schema mismatch, prohibited phrases, missing evidence IDs, homogenized verdicts) run a single time against that saved response.
• Quality failure ends the run: a failing response is recorded as excluded and no article is published. Generation is not repeated to chase a passing result.
• Retries are transport-only: retrying is limited to temporary transport failures that occur before a response is obtained — HTTP 429, HTTP 503, timeouts, and similar.
• Resume never re-generates: if a saved response already exists for the run, resuming reuses it instead of calling the model again.

After an excluded outcome a human may edit the article text. The scores are not changed by that edit, and the edited article must pass the same validator before it can be published.