Two questions about
a paper, kept apart.

This is a work in progress, and it is meant to be. This page will change as the community changes it.

Publishing-house science is slow, expensive and inefficient, and we all know it. Moving past it means finding new ways to lay signal on the work we produce.

Science is an evolving degree of understanding. Every scientific report is, at least at first, one story — one reflection, one group’s idea and interpretation of where our knowledge is moving. Curator exists so that individuals can take an active and formative part in sculpting those ideas into knowledge: not only producing one-time products, formerly called papers, but continuously shaping how they are read and what they are taken to mean.

That sculpting draws on stored human knowledge and insight — some earned through experience, some reachable only by connecting studies nobody has yet connected. It is how the speed and the accuracy of the work improve together, and how we work out which experiment might reveal the next breakthrough.

We need critical reading in addition to efficient publishing in order to do great science.

The two come apart in both directions. Work that carefully reproduces an older result can be high Q and low I — valuable precisely because it confirms something, without moving the field. Work that opens a new direction can be high I and subpar Q, especially when the confirming experiment is not yet achievable.

Below a minimal level of Quality there is little reason for anyone to read a paper at all. Above it, Impact is a separate argument.

There is a second reason to keep them apart, and it is empirical rather than philosophical: scientists agree with each other far more about Quality than about Impact. Q is close to an objective reading of the methods. I depends on the reader’s field, experience and taste. Averaging a tight distribution with a wide one hides exactly the disagreement that is worth seeing.

Curator draws a hard line between noticing a paper and judging it. Judgment lives in the two survey forms, and only there. Noticing has its own gesture, and it takes one tap.

The manuscript-top area, complete. The bee is the only control that lives up here now, and it is not a score: buzzing moves no Q, no I, and no feed multiplier derived from either. It says people are talking, and how many.

Only the lower part of the Impact form moves the posted score. Everything in the upper part — features, audience, keywords — steers which colleagues see the paper, whether or not it ever changes the number.

Your posted Impact is a number you choose yourself, on a slider, after the form has walked you through what it should be made of. The subsets come first and the overall comes last, so the parts inform the whole rather than justify it — but the parts never touch the arithmetic. The slider alone is the score.

Part one in full. Nothing above this caption moves the posted number — it decides to whom the paper is recommended, and why. The old Q1 notes field is gone from here on purpose: the one text field on this form now closes it, after the score.

Part two in full. The subsets are clicked in whole points; the overall alone is the score, placed to a tenth; the notes close the form.

What is posted is what you placed: your review’s Impact is your Q6, to the tenth. The four subsets above it never enter the arithmetic — they are annotation, for readers and for the record, and you weigh them in your own head on the way to the slider. The paper’s community Impact is the average of everyone’s, and it floats as reviews accumulate. There is no formula between your hand and the number.

Quality has one posted number and twelve optional expansions. The posted score answers a single question: overall, the data support the conclusions presented in the paper. One to ten, whole numbers.

Beneath it sit twelve sub-questions, each scored the same way and each optional — whether the title and abstract are supported, whether every figure’s conclusions hold, whether confounders were accounted for, whether the controls, sample sizes and statistics are appropriate, whether the methods permit replication, and whether the writing and references do their job.

There is one place the form insists. Scoring below six means claiming the work sits under the standard of the lowest tier of journals, and Curator will not accept that as a bare number: the sub-questions and a written note all become required, and the note must run at least 120 characters. A serious accusation should carry its reasoning.

A written note is also required on every new review, whatever the score — during initial review of a new manuscript, considering quality in words is the peer-improvement half of the work, not an optional extra.

A signal is not a rating that sits on a page. It moves three things.

Every paper arriving in Curator carries an estimate before anyone has read it, derived from one fact: the journal it appeared in. It is set in italics, in parentheses, next to Curator’s own score.

Since we are trying to shift away from journal brand as a measure of worth, the challenge of equating a name with a series of numbers first deserves a brief look at the value of a name. Below shows the variance in the citation index of individual papers for two journals. Focusing on the high-impact first one, we all note that not all papers in these journals have the same value (as measured by citation index, given its own caveats) as the journal brand itself.

Based on the two very small studies done and described in Provenance below, we will initially be using the following table as an origin of the values in italics.

These three tiers — Highest Tier, Field Specific, Niche — are the same three shapes drawn under the Overall Impact slider. The violin centres are the Impact column of this table; the calibration a reviewer sees while placing a score is the same evidence, in the same units, as the estimate a paper wears before anyone reviews it.

Three bands, and only three, because that is all the evidence supports. Reviewer-to-reviewer variability within any single impact-factor band turned out to be wide — as wide as the gap between the bands themselves, and familiar to anyone who has watched two referees split on the same submission. Finer bins would be false precision. Three is as much resolution as the data carries, and arguably one more than it has earned.

The typography is the argument. Italic and parenthetical is a guess about a journal. Upright and bold is a judgment about this paper. The moment anyone reviews it, the second number replaces N/A and the two can be compared — which is the entire point of the exercise.

Applying these retroactively on the basis of brand is a weakly supported estimate, and we would rather say so on the page than in a footnote. It survives for one reason: it is still better than the alternative of showing nothing, and it makes visible how poor a surrogate journal brand is on a per-paper basis.

One end of the literature has no estimate behind it at all, because nobody has studied it. No analysis to date has characterised the relationship between brand and assessed quality for paper-mill journals. Work arriving from a venue below an impact factor of three therefore enters at Q 5, I 5 — a placeholder, not a finding, and among the first things Curator’s own data should be able to replace.

None of this was designed in the abstract. The shape of the two scales, the decision to separate them, and the numbers in the look-up table all came out of two studies of working scientists reviewing real manuscripts.

Fifty Quality reviews and 112 Impact reviews across eighteen manuscripts submitted to bioRxiv and to journals. Every reviewer worked inside the field of the paper they read. Each manuscript drew at least two Quality and five Impact reviews.

Two findings shaped everything that followed. Authors read the importance of their own work reasonably well: the Impact curves are relatively flat, meaning people submitted to roughly the tier their peers agreed the work belonged in. But the Quality curves cross constantly. Papers submitted to lower impact factor journals frequently earned Quality scores comparable to top-tier submissions — a fact the current system has no way of recording.

The spread in Quality scores was smaller than the spread in Impact scores. Scientists can agree on whether the science is good. They agree far less about what will turn out to matter.

The study is underpowered and we treat it as a source of starting estimates rather than conclusions. One incidental number is worth recording anyway: eighteen months after the reviews were collected, six of the papers still had not been published anywhere.