We are in a very interesting time in our history that allows us to see science done in real time right before our eyes. This is both fascinating and frustrating. It’s fascinating because the process that usually happens in smaller research circles is now more widely available on Twitter, in the news, and shared freely on social media. It’s also frustrating because part of the scientific process is the self-correcting that happens as researchers are working out the kinks in the questions and answers, and that course correction can lead to all kinds of problems for folks who aren’t used to watching the process. I’m going to tackle the process of scholarly research and try to break it down into chunks so you can be better equipped to know how to handle all the studies flying around at the moment.
First, let’s make sure we’re all on the same page.
When I use the term “research,” I am referring to work done according to the scientific method that has been submitted for scrutiny, usually through publication or professional/academic conferences. When I am searching for peer-reviewed research, I use different databases that contain journals and proceedings from well-established publishers and institutions.
What I am not referring to are journalistic articles about research, self-published work, or even researchers talking about their work. I’ll get into when these things can be helpful or detrimental in a future post.
What’s in a paper?
When an author or group of authors complete a study, what usually follows is a peer-reviewed written article published in an established journal that is then available for public consumption. Most papers of a scholarly nature follow an established pattern that helps the readers to follow the process and conclusions of the authors, but also makes it easier to find certain things in a paper. These are the things to look for in a scholarly article:
1. Abstract: This is a short summary of the whole paper that briefly outlines the question, the methods, and the conclusions. You can often get the main ideas of a paper from the abstract.
2. Review of Literature: This is an overview of any relevant research that has already been published that pertains to the question at hand. When I was learning to write scholarly research, I was taught that the lit review was like a funnel starting with the broadest articles in the field and moving down to the more specific articles that dealt with the question. The lit review is the trail the researchers are laying down to show you why their question is important and how it has been viewed in the past.
3. Research questions: You’ll often find this in the paper with a sentence that starts something like, “The purpose of this study is” and then the next few sentences will lay out, sometimes in bulleted lists, exactly the questions that are being addressed.
4. Methods: Consider this the “recipe” of the study. A good method lays out every step the researchers followed, down to the measurements and statistical tests used. The idea is that anyone who wanted to replicate the study could read the method and follow the procedures. This is vital to the peer review process.
5. Results: If you’re just reading a paper, this is probably going to be the driest section of the whole thing. You’ll often find here lots of charts, numbers, stats, etc. If the methods are the recipe, the results are what come out of the oven when the timer goes off.
6. Discussion: This is my favorite part of a paper to read and to write. Here the researchers get to discuss their conclusions. They not only look at the results, but they can interpret what they mean. A really important part of the discussion is looking at any new or unforeseen problems that came up during the process, new questions that the authors would propose, and any ideas for future research that could take the topic further.
7. Bibliography: A gold mine for anyone doing research (especially doctoral students writing a dissertation), an extensive bibliography is where the researcher can go to find anything mentioned in the lit review or other parts of the paper.
A paper that is missing any of these sections, or if any of those sections are less than vigorous, is a red flag. For example, if a paper gives conclusions and discusses what they mean without a results section that tells you how they came up with those conclusions, that’s a pretty good indication that the paper should not be taken as seriously as others. A good paper must tell you the detailed what, how, and then why.
After the experiments or methods are followed, analyzed, and then written up, the paper is submitted to a journal or a conference panel for rigorous review.
The Peer-Review Process
The peer-review process is exactly what it sounds like: Other researchers or experts in a certain field look over an article before publication to assess the methods, the results, and the conclusions of the authors. Usually the review process is blind, meaning both the authors of the paper and the reviewers are unknown to each other in order to maintain the integrity of the process. The editorial board or panel that reviews the work, if they are doing their jobs well, is looking for things that are wrong, as well as things they agree with. They look for the gaps and the holes that may have escaped the author’s attention, they pose new questions, and suggest ways the research may be improved.
There are three possible outcomes that come from this review process: the paper is accepted for publication, the paper is returned to the author for revision and then another review, or the paper is rejected. If the paper is accepted, it is then published and submitted to the wider world via a professional journal or a conference presentation. If the board requests revisions, the author has the opportunity to make any suggested changes or justify their original decisions, and then resubmit for review again. Papers that are rejected outright are not published, and usually not available to the public or the scholarly community, usually because there are major flaws in the author’s process or conclusions. It is a thorough process that usually prevents bad research from filtering into the literature. It’s not a perfect system because it is run by humans who sometimes make errors, but it’s a good system with a great deal of checks and balances.
At this particular time there are a seemingly high number of papers that are reaching public consumption before the peer-review process is complete. These are called pre-prints, and they are very often available on major journal websites and can be read and passed around by anyone. This does not mean the paper is bad or should be rejected outright by readers, but it does mean that the paper’s methods, results, and conclusions have not been carefully reviewed yet, and so it is wise to hold those papers to a greater level of scrutiny until the peer-review process is complete. However, I’m finding that often those preprints are getting good peer-review on Twitter and other social media by experts. Reading what the established experts in the field are saying about preprints is a pretty good indication of whether or not that paper will hold water or not. It’s a little messy, to be honest, but when the experts are talking amongst themselves and we get to look on, it can be very helpful. An assumption, though, is that a person can distinguish between an expert in any given field and a charlatan. I’ll get into how to spot an expert in a future post.
Because research at this time is moving at a breakneck speed, it is important to know that the information you’re looking at is still considered valid. Many journal databases will include dates of submission, review, and publication at the top of the article. Those dates are important, particularly at a time like this when information is evolving rapidly. It is essential to know if an article you are reading might have outdated or updated information.
What about “research” that isn’t Research?
One of the most over-used and least understood phrases in the world today is “Do your own research.” There are two unspoken assumptions that usually underlie this phrase: First, that a lay person can understand a field as equally as an expert who has put in years of training, and second, that a lay person can access all the data that is available in order to make an informed decision. I’m going to get into these two assumptions in a future post, but I do want to mention here that it is to our detriment as a society when we assume that all opinions are equally valid. In logic, that’s called the false equivalency fallacy. In many situations, there are most definitely opinions that hold more weight than others. It is helpful when we can correctly identify those opinions that should be weighed more heavily.
For example, when it comes to singing and working with a singer who is having particular difficulties, my opinion as a vocologist and trained voice teacher is going to be worth more than a particle physicist, and more than a non-singer. That shouldn’t be controversial, but lately, and very unfortunately, experts are being held in suspicion more than is really fair. A lay person won’t know where all their insufficiencies lie. The breadth of knowledge in any field is so much more than anyone outside that field can comprehend. But this is exactly why we have experts. They put in the time and the education so they can do the hard thinking and the hard work that would be out of the reach of someone who isn’t trained to their level. This doesn’t mean that non-experts in any field are dumb. Not at all. We simply must have the humility to acknowledge that we are out of our depth when comparing our lay person’s knowledge to that of an expert, whatever the field. A marine biologist would absolutely blow me away in a discussion of dolphin behaviors, but I’d likewise be much more advanced in being able to discuss the neurological underpinnings of vocal amplitude. It is wise and prudent to rely on experts to guide us through things that we don’t have the foundation for.
In the next post I’m going to lay out a way to approach research as a lay person, giving some clues to be able to sift the wheat from the chaff in the research world. If you’ve got questions about how to approach research, leave me a comment and I’ll do my best to address it as I go along.
In the mean time, I wish you health and joy. The world is still full of many reasons to sing.