tag:blogger.com,1999:blog-5841910768079015534.comments2024-03-18T08:28:01.624+00:00BishopBlogdeevybeehttp://www.blogger.com/profile/15118040887173718391noreply@blogger.comBlogger2627125tag:blogger.com,1999:blog-5841910768079015534.post-20759988587875807072024-02-12T20:22:43.912+00:002024-02-12T20:22:43.912+00:00I think it is important to clarify "top 2% sc...I think it is important to clarify "top 2% scientist in the world according to Stanford University’s rankings" for readers. Stanford University has never prepared or promoted any such ranking. John P.A. Ioannidis, Professor of Medicine at Stanford, studies citation patterns. In partnership with Elsevier, he prepared the dataset linked in the "3rd Feb" update to the post. The dataset is just that. It is a way for people to study citation patterns using a common dataset that has passed some quality control. Now, less-than-honest individuals have decided to claim that if their name is in this dataset, they have been "ranked top 2% by Stanford". Such a statement is false. It is also important to note that Ioannidis initially put together such dataset to control for "[e]xtreme self-citations and “citation farms” ... [that] citation metrics spurious and meaningless", ref. https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000384. In some sense, if this dataset reveals the extreme self-citations of an author, it is quite the opposite of being "recognized as top 2%"...Context matters.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-35881169619388034922024-02-10T06:15:31.420+00:002024-02-10T06:15:31.420+00:00Open peer review could clear many of these drawbac...Open peer review could clear many of these drawbacksAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-65953852519770119222024-02-09T18:26:18.858+00:002024-02-09T18:26:18.858+00:00There are estimated to be (IIRC) about 20,000 acad...There are estimated to be (IIRC) about 20,000 academic journals. We might want to read and evaluating a sample of 30 papers from each of those to get an idea how good they are. I generally take about two hours per paper, so that's 1.2 million person-hours of people with PhDs. $100 million should cover it.Nick Brownhttps://www.blogger.com/profile/07481236547943428014noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-29881613708878648002024-02-09T10:42:35.813+00:002024-02-09T10:42:35.813+00:00Thinking about this major problem of poor quality ...Thinking about this major problem of poor quality papers, would it be possible for someone to make a 'Journal Integrity Score' under which a random sample of papers in each journal get checked for sense and statistics and journals with a low score are blacklisted. Having a metric feels like it might be the only way to get journals to up their game and to reward the good journals that do things properly.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-23877428991207430162024-02-05T10:43:29.633+00:002024-02-05T10:43:29.633+00:00An excellent idea! Perhaps someone who has the clo...An excellent idea! Perhaps someone who has the clout (like a Prof) should organise an international conference on methods for the detection of scientific fraud. In clinical trials one reason for choosing a set of core outcomes might be to form a set within which the pattern of relationships of variables, i.e. the 'face' of such multivariate data, is hard to convincingly simulate or forge. The contributions and edited comments from such a conference could be published as a resource for such courses.Miland Joshinoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-81101677822562923232024-02-03T12:10:14.086+00:002024-02-03T12:10:14.086+00:00What I find amazing about academia is there is usu...What I find amazing about academia is there is usually no difference in kind between the obvious scams, and the established practise. There is little difference between predatory journals and established journals, just degrees maybe of review quality. It is hard to spot any difference between the obvious scam conferences my inbox is littered with, and the established conferences I am required to attend for my career. Similarly, here is there any difference between what these institutions are doing and what more established Universities have been doing for years. Why not look at the University of Manchester with Steiglitz and Logothetis (just two I know of). Logothetis received a 6 figure salary - he would fly in to Manchester and come straight from the airport to give his contractually required annual in person talk, often being late, then have to cut questions short to go straight back to the airport. I don't see any difference there apart from the amounts of money involved. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-26731300684669666912024-02-02T09:22:03.502+00:002024-02-02T09:22:03.502+00:00Anglo-Saxon Attitudes.Anglo-Saxon Attitudes.Smut Clydehttps://www.blogger.com/profile/09409476490132867809noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-19457304735811996892023-12-06T18:23:19.926+00:002023-12-06T18:23:19.926+00:00I have many questions about their 'placebo'...I have many questions about their 'placebo'. The active treatment delivers red light at 640 nm... not out of any special rationale in terms of mechanism, but because LEDs with that wavelength are easy to source. The placebo device is still red light but at some other mixture of wavelengths.<br />Why is it a placebo? Why shouldn't it work as well as the active device?Smut Clydehttps://www.blogger.com/profile/09409476490132867809noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-54785501184548155462023-12-04T05:59:38.982+00:002023-12-04T05:59:38.982+00:00I don't understand the data--your Figure 2 (th...I don't understand the data--your Figure 2 (their Chart 1) shows the mean 'irritability and agitation' subscale scores (apparently a key measure) is ~30 for *both* groups at baseline (this is confirmed from the raw data on ResearchGate). The mean score for the 'experimental' group then declined over subsequent tests. Near identical means at baseline? I didn't expect that ...Geoff Hammondnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-72720072401251738922023-12-01T17:23:58.012+00:002023-12-01T17:23:58.012+00:00Ah, hence the monastic tonsure. It made them smart...Ah, hence the monastic tonsure. It made them smarter!Same anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-65184084870185891802023-11-29T02:04:35.204+00:002023-11-29T02:04:35.204+00:00Photons do get through your skull, though not many...Photons do get through your skull, though not many, and well-scattered. But there is one form of non-invasive brain imaging that measures blood flow across the cortex (as a proxy for cortical activity) by counting infrared photons on the way out.<br /><br />My bald spot is actually an adaptation, allowing more photons to reach my parietal lobes and contributing to my superior intelligence.Smut Clydehttps://www.blogger.com/profile/09409476490132867809noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-43660030073375659742023-11-27T17:36:56.747+00:002023-11-27T17:36:56.747+00:00Am I missing something? Don't we need even the...Am I missing something? Don't we need even the ghost of a plausible mechanism? And don't we have a skull, scalp, and hair all preventing (among other things) light from shining into our brain?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-39361320424583338312023-11-25T12:47:43.575+00:002023-11-25T12:47:43.575+00:00Great analysis! Weird that the statistical consult...Great analysis! Weird that the statistical consultant didn't notice any red flags but I've seen that happen a lot even when they're brought on to investigate data anomalies cases. Their blindspot is that they tend not to know what real data looks like because they cover too many different fields and spend much of their time in the realm of statistical theory.Aaron Charltonhttps://OpenMKT.orgnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-17068794421373469972023-11-22T03:14:41.377+00:002023-11-22T03:14:41.377+00:00Another factor in play here is what appears to be ...Another factor in play here is what appears to be a decline in the rigour (and hence quality) of peer review. Without incentive to devote much time for reviewing submissions (better off to write up your own work) the tendency is for reviewing to become superficial and sometimes just a quick 'wave it through'. (This of course is not restricted to detecting fraud, which requires specialist skills or dogged devotion.)<br /><br />Geoff HammondAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-17491863648071326762023-11-22T00:44:14.132+00:002023-11-22T00:44:14.132+00:00Hi I'm a creator of SciScore an AI tool that w...Hi I'm a creator of SciScore an AI tool that works with peer review to flag mainly errors of omission for rigor and reproducibility items in manuscripts. I would love to see more tools develop and be used in the manuscript review process. Though it was difficult to integrate our tools with EM and eJP, Scholar One has been impossible. For peer review tools to be functional we really need a way to get past these gatekeepers. Very few toolmakers that create cool tools are allowed to offer them to the journals. anitabandrowskihttps://www.blogger.com/profile/17017734800812304080noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-42042257610587499612023-11-21T12:25:53.324+00:002023-11-21T12:25:53.324+00:00At least every researcher disclosing misconducts r...At least every researcher disclosing misconducts related to his own results should try to do something though this looks to be a full-time job... An example : http://cristal.org/Mesli-et-al.pdfArmel Le Bailhttp://cristal.org/noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-61142361046008848472023-11-20T12:02:57.485+00:002023-11-20T12:02:57.485+00:00I like the idea! But if we just teach our students...I like the idea! But if we just teach our students the mechanics of fraud detection we run the risk of making them better cheaters. I had this epiphany while teaching about p-hacking in my statistics class. We also need to instill in them what Richard Feynman referred to as "a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards." I know that this should go without saying, but I think it also needs to be said . . . repeatedly. We need to teach them the difference between "getting a result" in the short term and "getting it right" in the long run. My current idea is to have them read Feynman's "Cargo cult science" speech (the source of the above quote) and then discuss it in class. I'd be curious to hear others.Rick Bornhttps://www.hms.harvard.edu/bss/neuro/bornlab/noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-18749978723016864662023-11-20T10:03:47.268+00:002023-11-20T10:03:47.268+00:00This is Dorothy Bishop replying - having trouble s...This is Dorothy Bishop replying - having trouble signing in! I would anticipate that people interested in this course would come from a range of backgrounds, and would tend to be people with good data processing skills and include some people with expertise in AI. Yes, you can't really understand problems without knowing underlying methods and we know there are issues with reproducibility of ML research, for instance. But another concern is with the individual studies that get into big data datasets, which may use quite basic statistical methods. Another concern is use of AI language to obfuscate: type "gobbledegook sandwich" into the PubPeer search box and you will see what I mean. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-71859718701679481932023-11-20T00:00:58.808+00:002023-11-20T00:00:58.808+00:00I'm curious about the immediate, low-hanging t...I'm curious about the immediate, low-hanging techniques to assess research fraud. Serendipitously, I've been using plagiarism detectors, GRIM/SPRITE/DEBIT, statcheck, and related tools as part of a broader project to create a peer review guide. <br /><br />For your first three bullet points, I've found that latests LLMs (e.g. ChatGPT) to be quite helpful. Maybe there are ways to integrate them into peer review software.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-272723037367858032023-11-19T16:50:49.007+00:002023-11-19T16:50:49.007+00:00I'm charmed by this idea, and once upon a time...I'm charmed by this idea, and once upon a time I might have signed up for such a course. But I note that teachers of this art in the original series tended to meet rather unfortunate ends, and I worry that in this your metaphor is a little bit too apt. As you note, science as a whole doesn't currently have great incentives in place for ensuring that what is published is true. Cheaters get tenure at Harvard; fraud sleuths get sued; I don't think this is an accident. I don't think universities (and certainly not publishing houses) would have reason to change the arrangement until not doing so becomes prohibitively expensive. We'd need an adversary. A William Proxmire for the twenty-first century? It makes me squirm to think of rooting for that.Erin Jonaitisnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-27533219407080601822023-11-19T16:18:58.498+00:002023-11-19T16:18:58.498+00:00The same is more or less the case in The Netherlan...The same is more or less the case in The Netherlands. An exception is Wageningen University. This university has scanned manually all PhD theses which were not yet available as a digital copy. Anyone has free access to https://library.wur.nl/WebQuery/wda?dissertatie/nummer=*Klaas van Dijknoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-40682795567321130552023-11-19T14:13:57.966+00:002023-11-19T14:13:57.966+00:00You rightly flag up big data approaches to synthes...You rightly flag up big data approaches to synthesise findings, something that might be used to find new drug targets or whatever in cancer research for example. So... should your course also have at least a taster of machine learning (AI) awareness or something, so that students know what's going on?anonymous13https://www.blogger.com/profile/03392117369543433452noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-57448693709110831182023-10-13T13:59:25.428+01:002023-10-13T13:59:25.428+01:00Thanks Franck. When you say "similar misgivin...Thanks Franck. When you say "similar misgivings" do you mean you had misgivings about making the information public, or misgiving about the restrictions that arose because of data protection rules?Dorothy Bishophttps://deevybee.blogspot.comnoreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-69256276244462530702023-10-13T10:03:40.703+01:002023-10-13T10:03:40.703+01:00Interestingly, for PhD, we have this national, fre...Interestingly, for PhD, we have this national, freely accessible database: https://www.theses.fr/<br />I don't know why PhDs are treated as an exception. Maybe the public status of research?Franck Ramushttps://www.blogger.com/profile/02656240693713885894noreply@blogger.comtag:blogger.com,1999:blog-5841910768079015534.post-85476112941885646392023-10-13T10:02:22.109+01:002023-10-13T10:02:22.109+01:00We've had similar requests at ENS, including f...We've had similar requests at ENS, including for a high-profile case of suspected fraud. And we had similar misgivings about personal data protection. Current policy is: show us a copy of the diploma provided by the alleged former student, and we will check its authenticity.<br />FranckFranck Ramushttps://www.blogger.com/profile/02656240693713885894noreply@blogger.com