@ukrio@emilymbender@academicchatter In my field, I don't think the description of slow is complete. Robotics has been largely a discipline of "Look ma, no hands!" - it's not just about the rush to publish, it's that our idea of what information in the paper counts as the contribution is a problem.
We're missing "What is the thing you are trying to understand? What scientific question are you answering?"
@ukrio@emilymbender@academicchatter The piece of information that will be useful to someone else is buried three paragraphs into the results section, and the conclusion boils down to "We made it work! You should use this approach!"
If we don't focus on what we're trying to learn, we won't notice that the paper doesn't actually contain data that would help other researchers.
From my (biologist) perspective, there is a place for proof-of-concepts: They can show solutions to problems the field has been struggling with. They become useful if they are documented well enough that other groups can adapt their approach.
@robotistry@ukrio@emilymbender@academicchatter at @fzj_rdm we are working on precisely this problem. We’ve developed theory and pedagogical methods to read and understand papers. When we applied them, we found most scientific literature grossly inadequate. So we’ve been developing theory and pedagogy to understand how to document science so it’s comprehensible and usable. The basic premise is that scientists >
@robotistry@ukrio@emilymbender@academicchatter@fzj_rdm are human and understand and organize information within human constraints. This has let us to understanding science as goal-directed behavior. We are working with a few research teams and they find the approach a revelation.
@ukrio I think the problem is an AI/ML community problem and not an arXiv problem. Much of the same applies to major ML conferences, where peer-review is often poor and biased towards hype, the short format encourages glossing over important details, and there's a strong pressure to send sloppy work hoping that it'll make it through. @emilymbender@academicchatter
@ukrio@emilymbender@academicchatter I'm a strong supporter of arXiv, but she makes some valid points. The same criticisms could be made, and with more validity, for the many predatory and grey journals. The real problem is the pressure to publish at any cost - as long as this continues we will see a flood of poor-quality papers and sadly any old rubbish can be published somewhere these days.
@ukrio@emilymbender@academicchatter The blog compares the worst of arXiv to an idealized version of peer review, and so sets up an unfair comparison. In practice, peer review often does not mean a paper was "examined critically and thoughtfully by 2–4 independent people with relevant knowledge."
@ukrio@emilymbender@academicchatter The Medium post reads much more balanced than the quote taken from the tweet. I can relate to many points, in particular the rush to publish and to put out papers quickly. I have two main issues with the post:
The reality (let's say it is the reality) of preprints is compared with an idealized picture of #PeerReview. Do reviewers "[p]erform thorough and careful evaluation"? We probably won't notice in the cases where they do, but 1/
@ukrio@emilymbender@academicchatter
there are plenty of examples where obvious errors, and sometimes also less obvious errors, have not been caught. Another problem is reviewer recruitment problems. 15+ invites are needed to recruit two reviewers; a non-negligible number of cases (I guess) of editors aborting the process bc reviewers cannot be recruited; papers only getting published after 3-5 years. Slow, responsible science is good, but not for these reasons 2/
The flooding of arXiv and other preprint papers with low-quality/fake papers (assuming it happens) is only a symptom of an overreliance of publishing and citation metrics by researchers, funding agencies etc. If we would ignore preprints, except maybe for the reasons listed in the Medium post, there would be other way to game the system. These other ways may be a little bit more demanding than posting preprints, but 3/
@ukrio@emilymbender@academicchatter
there have been plenty of cases in the past where shady papers passed peer review at legit journals; or where authors rigged peer review by setting up fake accounts to review their own papers; or where editors have been on board with authors to get papers through the process.
I don't know how to quickly get to the source of publish-or-perish and metric mania, but this is the real issue underlying it, IMO 4/
Add comment