The Paper Reviewing ProcessPosted: October 18, 2013| Author:Nick Feamster|Filed under:advice, research, reviewing|
Learning how to review papers not only (obviously) makes you a better reviewer, but it can also help you as an author, since an understanding of the process can help you write your paper submissions for an audience of reviewers. If you know the criteria that a reviewer will use to judge your paper, you are in a much better position to tailor your paper so that it has a higher chance of being accepted.
There are many good resources that describe the paper reviewing process already, including those that explain the process (and its imperfections) and those that provide instructions for writing a good review (as well as techniques to avoid). There are also a few nice summaries of the review process for conferences in different areas of computer science that lend visibility into the process (e.g., here and here). Program committee chairs sometimes provide guidelines for writing reviews, such as these. I will not reiterate or summarize thosepreviousarticles here, but they are all definitely worth a read. Instead, I will discuss the importance of the review process and how it differs from simply reading a paper; I’ll also talk about how to prepare (and ultimately write) a review.
I will not talk about the paper selection process (i.e., what determines whether a paper is ultimately accepted or rejected), but will instead focus on the creation of a paper review. Program committee meetings are an important part of the paper selection process—at least in computer science—and I will be devoting a complete post to this topic next week. Meanwhile, I recommend reading Matt Welsh’s post on the psychology of program committees.
The Review Process
Why understanding the review process is important. Whether you end up reviewing a lot of papers as a Ph.D. student, your research will definitely be subject to the paper review process. It is imperative as a researcher to understand this process. Knowing the process can help you better write your paper for an audience of reviewers (and a program committee), and it can also help you maintain perspective when your paper is accepted or rejected. The process is far from perfect, and the outcome of the process is neither validation nor condemnation of your work. How you react—and how you adapt your research or follow through on it after the acceptance (or rejection)—is far more important to long-term success.
In the “Introduction to the Ph.D.” class at Georgia Tech, I ask students to create a research idea and write it up; a subsequent set of assignments asks the students to review and evaluate the ideas as part of a “mock” program committee. The process isn’t exactly the same as the review process for a full paper, but it is a lightweight way to have students experience the process first-hand in a low-stakes setting, and see both sides of the process (submission and review) at the same time. In next week’s blog post, I will discuss program committee meetings in general, as well as some observations from this year’s (and previous years’) in-class experiences with the mock PC.
Reviewing vs. reading. There are some significant distinctions between reading papers vs. reviewing them. When reading a paper for your own enrichment, your goal is to gather information as quickly as possible. In this case, you are a scientist who seeks to understand the context and content of existing work, to (for example) better understand how your own research might fit into the bigger picture or learn about techniques that might apply to your own work. The goal of reviewing is different. A reviewer’s goal is to first and foremost determine the suitability of a paper for some conference and second, to provide feedback to the authors to help them improve the paper in subsequent revisions. Remember that the reviewer’s primary goal trumps all other objectives: A reviewer often has a large number of papers to process and is typically not deeply devoted to improving the content of any particular paper. If you are lucky, you will get a diligent, thoughtful reviewer who provides thorough feedback, but do not be surprised if a review is not as thorough as you would have liked, or if the review “misses” some point you were trying to make. We would all like reviewers to make three passes through your paper submission—and, these are the instructions I would give, too, in an ideal world. Unfortunately, however, you will be lucky in many cases to get two thorough reads. The reviewer’s main goal is to determine the paper’s suitability for publication. As an author, you shouldn’t be surprised if some of the comments seem trivial: there may be underlying issues of taste that drove the reviewer’s opinion on your paper that a reviewer may not explicitly state. Whenever I read reviews I receive for a rejected paper, I try to look past specific detailed quibbles (or “excuses” for rejecting the paper) and figure out the big picture: the reviewer couldn’t find a reason to accept the paper.
Calibration: Reviewing one paper vs. reviewing many papers. The paper review process can differ depending on who, exactly, is reviewing the paper. For example, as a Ph.D. student, you may review one or two papers at a time, as an “external reviewer” for a conference or journal. Journal editors and program committee chairs often seek the help of external reviewers if they need a particular subject-matter expert to review a paper. Later in your Ph.D. career, you may have established yourself as an expert on a particular topic and find yourself reviewing a paper here and there on a handful of topics. Sometimes a member of the program committee (e.g., your advisor) might ask you to help review a particular paper. As you progress in your career, you will be asked to serve on program committees yourself, whereupon you’ll find yourself with tens of papers to review over the course of a couple of months. Ironically, it is sometimes easier to review a group of papers than a single (or a few) papers, because seeing a group of papers helps you “calibrate” your scores and rankings of papers according to the general quality of papers that have been submitted to the conference. If you have been asked to review a single paper for a conference, you should either figure out how to calibrate your assessment with respect to other papers that might have been submitted, or simply review the paper on its merits while reserving judgement as to the paper’s ultimate disposition.
Does the Paper Realize a Great Idea?
Look for a reason to accept the paper. Does it realize a great contribution or idea? Every paper is imperfect. The paper may have made an incorrect or imperfect assumption. The experiments may not have been as thorough as you liked. The graphs may be difficult to read. Parts of the paper may be difficult to understand. These types of issues certainly reflect problems with a paper, but they do not necessarily constitute a reason to reject a paper if they do not affect the correctness or significance of the main underlying conclusion or contribution of the paper. Therefore, the first two questions I ask myself when reviewing a paper are: (1) Does the paper have a great idea?; and (2) Does it realize the great idea? (or, alternatively, to what extent does it realize that great idea, since typically no paper is water-tight).
What makes an idea “great”?Judging a paper’s contribution turns out to be highly subjective, which is why the review process remains so uncertain. A paper isn’t judged on a set of fixed checkboxes, a grading “key”, or any notion of absolute correctness. Reviewers often reserve considerable judgment based on “taste“, and reasonable people will disagree as to the merits of the main contribution or idea in a paper. In fact, there has been a fair amount of documentation that, as reviewers, we are often quite terrible at predicting the merits of a particular piece of submitted work: There’s a great article on this topic, as well as some parodies to illustrate the subjective nature of the process. Many fields have also introduced a “test of time” award to papers from past decades, to recognize accepted papers that have truly had long-term positive impact (implicitly acknowledging that this is almost impossible to assess when a paper is first published). Due to the subjective nature of this judgment, it is all the more important that your writing is clear, and well-matched to what a reviewer is looking for (i.e., the contributions and ideas).
Invariant questions. Different conferences may have different value structures, and the chairs of any given conference may ask the reviewers to focus on different criteria when judging a paper. Regardless, there are some invariant questions that most reviewers would (or at least should) always consider, including:
- Is the problem important? What problem is the paper trying to solve, and is it important? Seek to summarize the paper’s contribution in one sentence. Make this short summary the beginning of your review, as well. Try to convince yourself (by reading the paper or otherwise) that a solution to the problem that the paper is proposing would advance knowledge or significantly improve the state of affairs for some group of people. Note that you may not care about the problem, but also ask yourself whether you can imagine some group of readers who will be interested in the solution to the problem. When asking yourself this question about a paper, try to divorce your own taste about the problem’s importance from the more general question concerning whether there is some group of people who would be interested in the problem the paper is addressing and solving.
- To what extent does the paper solve the problem it describes? A single paper very rarely closes the book on a single problem, but it may take an important step towards solving the problem. It might solve the problem for an important set of operating conditions or under a new set of assumptions. Or, if the problem area is completely new, perhaps the paper doesn’t really solve the problem at all, but simply articulating a new problem area for follow-on work is a significant contribution.
- What is the “intellectual nugget”? As a reviewer, I try to identify whether a paper has a particular intellectual kernel that lies at the heart of the solution. This kernel is often what separates an important research contribution from a simple matter of engineering. This intellectual nugget might be the application (or invention) of a particular technique, a proof of correctness (where one previously did not exist), or an attempt to put the solution into a broader intellectual context. In other words, the intellectual contribution might be to take a general problem and tackle a specific sub-problem (e.g., under certain assumptions or conditions), or to take a specific problem and generalize it (e.g., develop a general theory, proof of correctness, or taxonomy). Looking through the paper for applications of specific research patterns can help identify an intellectual nugget, if one exists.
- What is the main contribution or conclusion? Is it important? As a reviewer, I try to concisely articulate the paper’s main contribution (or small number of contributions). Often, a paper will helpfully summarize those contributions somewhere in the introduction (Jim Kurose’s advice on writing paper introductions advises the writer to explicitly do so). The reviewer’s job is then to assess whether those contributions are significant or important enough to warrant a publication. The significance of those contributions often depends on the perceived increment over previous work. All work is incremental to some degree, as everything builds on past work. The author’s job is to convince the reviewer that the increment is important, and the reviewer’s job is to assess the author’s claims of significance.
- Does the content support the conclusion? An introduction may make broad (or wild) claims, and it is important to dig into the paper to determine whether the content of the paper supports the conclusion. Are the experiments run correctly? Are they based on the correct set of assumptions? If the conclusion involves comparison to previous work, is the comparison performed in a controlled manner, using an equivalent (or at least fair) experimental setup? If applicable, have the authors released their code and data so that you (or others) can check the claims yourself?
Preparing Your Review
Consider the audience. Not every publication venue is the same. Some venues are explicitly geared towards acceptance of early, incomplete work that is likely to generate discussion (many workshops use this criterion for acceptance). Other venues favor contributions that constitute well-executed, smaller increments. When reviewing a paper, either externally or as a member of a committee, your first question should be to consider the audience for the conference, workshop, or journal, and whether the likely audience for the venue would benefit from reading the paper. The question of audience involves that of both the “bar” for acceptance (Does the paper meet the audience’s standards for something that is worth reading?) and the “scope” of the venue (is the paper on-topic for the venue?). Often, scope can be (and is) broadly construed, so the key question really boils down to whether the likely audience for the paper will benefit from reading it.
Consider the standards. Your standards will (and should) vary depending on the venue for which you are reviewing a paper submission. Workshops are typically more permissive as far as accepting “vision” papers that outline a new problem or problem area or papers that “foster discussion” than conferences, which typically aim to accept more complete pieces of work. Nevertheless, even the standards for a conference review process will vary depending on both the conference itself, the program committee chair’s instructions about how permissive to be, and the relative quality of the group of papers that you are reviewing. A good way to get a sense for the standards of a conference for which you are reviewing is to read through the complete set of papers that you have been asked to review and rank them, before writing a single review. This will ensure some level of calibration, although it is still biased based on the set of papers that you are reviewing. Reading past proceedings of the particular journal or conference can also help you determine the appropriate standard to set for acceptance.
Consider the purpose. Different papers serve different purposes. Multiple paper submissions to the same venue might in fact have quite different purposes, and it is important to establish what the paper is contributing (or attempting to contribute) before passing judgement. For example, a paper might be a complete piece of work, but it might also be a survey, a tutorial, or simply a proposal. If the paper is one of the latter types, your first questions as a reviewer should concern whether the audience would benefit from the survey, tutorial, or proposal, and whether such a paper meets the standards for the conference. If the answers to those questions are “yes”, then your evaluation should be tailored to the paper’s purpose. If the paper is a survey, your assessment should be based on the completeness of the survey, with respect to the area that the paper is claiming to summarize. If the paper is a tutorial, is the description correct and clearly described? If the paper is a proposal, does the proposed research agenda make sense, and is the outcome (if the proposal is successful) worthwhile?
Consider the big picture. Every paper can be rejected. It is always easy to find reasons to reject a paper. The reviewer’s goal should not be to identify the reasons to reject a paper, but rather to determine whether there are any reasons to accept the paper. If the answer to that question is negative, then it is always easy to find “excuses” to reject a paper (recall the discussion above). You should be aiming to figure out whether the paper has important contributions that the audience will benefit from knowing about, and whether the paper supports those contributions and conclusions to the level of standard that is commensurate with the standard of the audience and the venue. One litmus test I use to ensure that a negative aspect of a paper does not condemn it is to ask myself whether the problem (1) affects the main conclusion or contribution of the paper; and (2) can be fixed easily in a revision. If the problem doesn’t affect the main contribution or conclusion, and if it can be easily fixed, then it should not negatively affect a paper’s review.
Writing Your Review
Start with a summary of the paper and its contributions. A short, one-paragraph summary describing the paper’s main contribution(s) demonstrates to the authors (and to you!) that you understand the main point of the paper. This helps you as a reviewer articulate the main contributions and conclusions of the paper for the purposes of your own evaluation. Try to address the type of paper it is (is it a survey paper, for example?), the context for the paper (i.e., how it builds on or relates to previous work), its overall correctness, and its contributions. If you cannot concisely summarize the paper, then the paper is not in good shape, and you can reflect this assessment in the review, as well. These summaries are very helpful to authors, since they may not match the authors’ views of the main contribution! For example, as an author, you can easily figure out if you’ve “missed the mark” or whether the reviewer fundamentally misunderstood the paper by reading a reviewer’s summary of your own work. If the summary of the contribution does not match your own view of the paper’s contribution, then you know that you have some work to do in writing and presentation.
Assess whether the paper delivers on the main claims and contributions. You should provide an assessment, for each of the paper’s main claims and contributions, whether it delivers on that claim. If the main contribution of the paper is flawed, you should indicate whether you think a flaw is “fatal”, or whether the authors could simply fix the flaw in a revision if the paper is accepted. Sometimes flaws (e.g., inconsistent terminology) are fixable. Other flaws (e.g., a questionable experimental setup) may or may not be fixable. While it might seem that a broken experimental setup is “fatal”, ask yourself as a reviewer whether the conclusions from the paper’s experiments as is are still meaningful, even if the authors have not interpreted the results correctly. If the conclusions from the experiments can be restated and still turn out to be meaningful contributions—or, if the flaw in an experiment doesn’t affect the main contribution or conclusion—then even a flaw in experiments can likely be fixed in revision. Occasionally, however, experiments may need to be completely redesigned because they don’t support any meaningful conclusion. Or, the content of the paper may simply be incorrect; sometimes correctness issues are difficult for a reviewer to spot, so a paper isn’t necessarily “correct” simply because a reviewer has validated the paper. Regardless, if there are correctness issues that affect the main contribution of the paper that call into question whether the main result or contribution is correct in the first place, the paper’s review should reflect these concerns and likely cannot be accepted.
Discuss positive aspects of the paper; always try to find something positive, even in “bad” papers. It is easy to identify problems with a paper. It can be much trickier (especially with “average” papers) to identify the positive aspects and contributions, but most papers typically have at least some small kernel of goodness. Even for particularly bad papers, there might be one sentence in the introduction, discussion, or future work section that makes an interesting point or highlights a possibility for interesting contributions. In a pinch, if you can’t find anything positive, those are good places to look. As a reviewer, you can remark that those observations are interesting, and that you would really like to see those parts of the work further developed. These positive comments aren’t just for author morale (although that’s important, too): They give the author a direction to move forward. The worst reviews are those that reject a paper but don’t provide any specific action for moving forward. The best reviews are those that highlight the positive aspects of the work, while identifying weaknesses and areas where the work could be further developed to address weaknesses or build on the paper’s existing strengths.
Criticize the paper, not the authors. When writing your review, consider the type of review that you would like to receive. Always be polite, respectful, and positive. Don’t be personal. Choose your language carefully, as it will help convey your message. For example, if you say “the authors don’t consider the related work”, that is a much more personal statement than “the paper doesn’t consider the related work”. (In fact, you don’t know if the authors considered a particular piece of related work anyway; they may have simply chosen not to include it in the writeup!) Talking about “the authors” gets personal, and it will put the authors themselves on the defensive when reading your review. Instead, focus on “the paper” and frame your critique around “suggestions for improvement”. Never, ever insult the authors; don’t accuse the authors of being sloppy or unethical researchers. As a reviewer, you don’t always know the full context, so limit your judgement to what you can directly conclude by reading the paper.
Consider the type of feedback you would like to receive. Receiving reviews for rejected papers is a part of the research process, but it is never fun for the authors (particularly new Ph.D. students). Do your part to contribute positively to the process by suggesting changes that you’d like to see if you had to review the paper again. In all likelihood, you may see the paper again in the form of a revision!
An interactive workshop on ‘The Critical Steps for Successful Research: The Research Proposal and Scientific Writing’ was conducted in conjunction with the 64th Annual Conference of the Indian Pharmaceutical Congress-2012 at Chennai, India. In essence, research is performed to enlighten our understanding of a contemporary issue relevant to the needs of society. To accomplish this, a researcher begins search for a novel topic based on purpose, creativity, critical thinking, and logic. This leads to the fundamental pieces of the research endeavor: Question, objective, hypothesis, experimental tools to test the hypothesis, methodology, and data analysis. When correctly performed, research should produce new knowledge. The four cornerstones of good research are the well-formulated protocol or proposal that is well executed, analyzed, discussed and concluded. This recent workshop educated researchers in the critical steps involved in the development of a scientific idea to its successful execution and eventual publication.
Keywords: Research protocol, scientific writing, creativity, publication ethics
Creativity and critical thinking are of particular importance in scientific research. Basically, research is original investigation undertaken to gain knowledge and understand concepts in major subject areas of specialization, and includes the generation of ideas and information leading to new or substantially improved scientific insights with relevance to the needs of society. Hence, the primary objective of research is to produce new knowledge. Research is both theoretical and empirical. It is theoretical because the starting point of scientific research is the conceptualization of a research topic and development of a research question and hypothesis. Research is empirical (practical) because all of the planned studies involve a series of observations, measurements, and analyses of data that are all based on proper experimental design.[1–9]
The subject of this report is to inform readers of the proceedings from a recent workshop organized by the 64th Annual conference of the ‘Indian Pharmaceutical Congress’ at SRM University, Chennai, India, from 05 to 06 December 2012. The objectives of the workshop titled ‘The Critical Steps for Successful Research: The Research Proposal and Scientific Writing,’ were to assist participants in developing a strong fundamental understanding of how best to develop a research or study protocol, and communicate those research findings in a conference setting or scientific journal. Completing any research project requires meticulous planning, experimental design and execution, and compilation and publication of findings in the form of a research paper. All of these are often unfamiliar to naïve researchers; thus, the purpose of this workshop was to teach participants to master the critical steps involved in the development of an idea to its execution and eventual publication of the results (See the last section for a list of learning objectives).
THE STRUCTURE OF THE WORKSHOP
The two-day workshop was formatted to include key lectures and interactive breakout sessions that focused on protocol development in six subject areas of the pharmaceutical sciences. This was followed by sessions on scientific writing. DAY 1 taught the basic concepts of scientific research, including: (1) how to formulate a topic for research and to describe the what, why, and how of the protocol, (2) biomedical literature search and review, (3) study designs, statistical concepts, and result analyses, and (4) publication ethics. DAY 2 educated the attendees on the basic elements and logistics of writing a scientific paper and thesis, and preparation of poster as well as oral presentations.
The final phase of the workshop was the ‘Panel Discussion,’ including ‘Feedback/Comments’ by participants. There were thirteen distinguished speakers from India and abroad. Approximately 120 post-graduate and pre-doctoral students, young faculty members, and scientists representing industries attended the workshop from different parts of the country. All participants received a printed copy of the workshop manual and supporting materials on statistical analyses of data.
THE BASIC CONCEPTS OF RESEARCH: THE KEY TO GETTING STARTED IN RESEARCH
A research project generally comprises four key components: (1) writing a protocol, (2) performing experiments, (3) tabulating and analyzing data, and (4) writing a thesis or manuscript for publication.
Fundamentals in the research process
A protocol, whether experimental or clinical, serves as a navigator that evolves from a basic outline of the study plan to become a qualified research or grant proposal. It provides the structural support for the research. Dr. G. Jagadeesh (US FDA), the first speaker of the session, spoke on ‘Fundamentals in research process and cornerstones of a research project.’ He discussed at length the developmental and structural processes in preparing a research protocol. A systematic and step-by-step approach is necessary in planning a study. Without a well-designed protocol, there would be a little chance for successful completion of a research project or an experiment.
The first and the foremost difficult task in research is to identify a topic for investigation. The research topic is the keystone of the entire scientific enterprise. It begins the project, drives the entire study, and is crucial for moving the project forward. It dictates the remaining elements of the study [Table 1] and thus, it should not be too narrow or too broad or unfocused. Because of these potential pitfalls, it is essential that a good or novel scientific idea be based on a sound concept. Creativity, critical thinking, and logic are required to generate new concepts and ideas in solving a research problem. Creativity involves critical thinking and is associated with generating many ideas. Critical thinking is analytical, judgmental, and involves evaluating choices before making a decision. Thus, critical thinking is convergent type thinking that narrows and refines those divergent ideas and finally settles to one idea for an in-depth study. The idea on which a research project is built should be novel, appropriate to achieve within the existing conditions, and useful to the society at large. Therefore, creativity and critical thinking assist biomedical scientists in research that results in funding support, novel discovery, and publication.[1,4]
Elements of a study protocol
The next most crucial aspect of a study protocol is identifying a research question. It should be a thought-provoking question. The question sets the framework. It emerges from the title, findings/results, and problems observed in previous studies. Thus, mastering the literature, attendance at conferences, and discussion in journal clubs/seminars are sources for developing research questions. Consider the following example in developing related research questions from the research topic.
Hepatoprotective activity of Terminalia arjuna and Apium graveolens on paracetamol-induced liver damage in albino rats.
How is paracetamol metabolized in the body? Does it involve P450 enzymes? How does paracetamol cause liver injury? What are the mechanisms by which drugs can alleviate liver damage? What biochemical parameters are indicative of liver injury? What major endogenous inflammatory molecules are involved in paracetamol-induced liver damage?
A research question is broken down into more precise objectives. The objectives lead to more precise methods and definition of key terms. The objectives should be SMART-Specific, Measurable, Achievable, Realistic, Time-framed, and should cover the entire breadth of the project. The objectives are sometimes organized into hierarchies: Primary, secondary, and exploratory; or simply general and specific. Study the following example:
To evaluate the safety and tolerability of single oral doses of compound X in normal volunteers.
To assess the pharmacokinetic profile of compound X following single oral doses.
To evaluate the incidence of peripheral edema reported as an adverse event.
The objectives and research questions are then formulated into a workable or testable hypothesis. The latter forces us to think carefully about what comparisons will be needed to answer the research question, and establishes the format for applying statistical tests to interpret the results. The hypothesis should link a process to an existing or postulated biologic pathway. A hypothesis is written in a form that can yield measurable results. Studies that utilize statistics to compare groups of data should have a hypothesis. Consider the following example:
The hepatoprotective activity of Terminalia arjuna is superior to that of Apium graveolens against paracetamol-induced liver damage in albino rats.
All biological research, including discovery science, is hypothesis-driven. However, not all studies need be conducted with a hypothesis. For example, descriptive studies (e.g., describing characteristics of a plant, or a chemical compound) do not need a hypothesis.
Relevance of the study
Another important section to be included in the protocol is ‘significance of the study.’ Its purpose is to justify the need for the research that is being proposed (e.g., development of a vaccine for a disease). In summary, the proposed study should demonstrate that it represents an advancement in understanding and that the eventual results will be meaningful, contribute to the field, and possibly even impact society.
A literature search may be defined as the process of examining published sources of information on a research or review topic, thesis, grant application, chemical, drug, disease, or clinical trial, etc. The quantity of information available in print or electronically (e.g., the internet) is immense and growing with time. A researcher should be familiar with the right kinds of databases and search engines to extract the needed information.[3,6]
Dr. P. Balakumar (Institute of Pharmacy, Rajendra Institute of Technology and Sciences, Sirsa, Haryana; currently, Faculty of Pharmacy, AIMST University, Malaysia) spoke on ‘Biomedical literature: Searching, reviewing and referencing.’ He schematically explained the basis of scientific literature, designing a literature review, and searching literature. After an introduction to the genesis and diverse sources of scientific literature searches, the use of PubMed, one of the premier databases used for biomedical literature searches world-wide, was illustrated with examples and screenshots. Several companion databases and search engines are also used for finding information related to health sciences, and they include Embase, Web of Science, SciFinder, The Cochrane Library, International Pharmaceutical Abstracts, Scopus, and Google Scholar. Literature searches using alternative interfaces for PubMed such as GoPubMed, Quertle, PubFocus, Pubget, and BibliMed were discussed. The participants were additionally informed of databases on chemistry, drugs and drug targets, clinical trials, toxicology, and laboratory animals (reviewed in ref).
Referencing and bibliography are essential in scientific writing and publication. Referencing systems are broadly classified into two major types, such as Parenthetical and Notation systems. Parenthetical referencing is also known as Harvard style of referencing, while Vancouver referencing style and ‘Footnote’ or ‘Endnote’ are placed under Notation referencing systems. The participants were educated on each referencing system with examples.
Dr. Raj Rajasekaran (University of California at San Diego, CA, USA) enlightened the audience on ‘bibliography management’ using reference management software programs such as Reference Manager®, Endnote®, and Zotero® for creating and formatting bibliographies while writing a manuscript for publication. The discussion focused on the use of bibliography management software in avoiding common mistakes such as incomplete references. Important steps in bibliography management, such as creating reference libraries/databases, searching for references using PubMed/Google scholar, selecting and transferring selected references into a library, inserting citations into a research article and formatting bibliographies, were presented. A demonstration of Zotero®, a freely available reference management program, included the salient features of the software, adding references from PubMed using PubMed ID, inserting citations and formatting using different styles.
Writing experimental protocols
The workshop systematically instructed the participants in writing ‘experimental protocols’ in six disciplines of Pharmaceutical Sciences.: (1) Pharmaceutical Chemistry (presented by Dr. P. V. Bharatam, NIPER, Mohali, Punjab); (2) Pharmacology (presented by Dr. G. Jagadeesh and Dr. P. Balakumar); (3) Pharmaceutics (presented by Dr. Jayant Khandare, Piramal Life Sciences, Mumbai); (4) Pharmacy Practice (presented by Dr. Shobha Hiremath, Al-Ameen College of Pharmacy, Bengaluru); (5) Pharmacognosy and Phytochemistry (presented by Dr. Salma Khanam, Al-Ameen College of Pharmacy, Bengaluru); and (6) Pharmaceutical Analysis (presented by Dr. Saranjit Singh, NIPER, Mohali, Punjab). The purpose of the research plan is to describe the what (Specific Aims/Objectives), why (Background and Significance), and how (Design and Methods) of the proposal.
The research plan should answer the following questions: (a) what do you intend to do; (b) what has already been done in general, and what have other researchers done in the field; (c) why is this worth doing; (d) how is it innovative; (e) what will this new work add to existing knowledge; and (f) how will the research be accomplished?
In general, the format used by the faculty in all subjects is shown in Table 2.
Elements of a research protocol
Biostatistics is a key component of biomedical research. Highly reputed journals like The Lancet, BMJ, Journal of the American Medical Association, and many other biomedical journals include biostatisticians on their editorial board or reviewers list. This indicates that a great importance is given for learning and correctly employing appropriate statistical methods in biomedical research. The post-lunch session on day 1 of the workshop was largely committed to discussion on ‘Basic biostatistics.’ Dr. R. Raveendran (JIPMER, Puducherry) and Dr. Avijit Hazra (PGIMER, Kolkata) reviewed, in parallel sessions, descriptive statistics, probability concepts, sample size calculation, choosing a statistical test, confidence intervals, hypothesis testing and ‘P’ values, parametric and non-parametric statistical tests, including analysis of variance (ANOVA), t tests, Chi-square test, type I and type II errors, correlation and regression, and summary statistics. This was followed by a practice and demonstration session. Statistics CD, compiled by Dr. Raveendran, was distributed to the participants before the session began and was demonstrated live. Both speakers worked on a variety of problems that involved both clinical and experimental data. They discussed through examples the experimental designs encountered in a variety of studies and statistical analyses performed for different types of data. For the benefit of readers, we have summarized statistical tests applied frequently for different experimental designs and post-hoc tests [Figure 1].
Conceptual framework for statistical analyses of data. Of the two kinds of variables, qualitative (categorical) and quantitative (numerical), qualitative variables (nominal or ordinal) are not normally distributed. Numerical data that come from normal...
Research and publication ethics
The legitimate pursuit of scientific creativity is unfortunately being marred by a simultaneous increase in scientific misconduct. A disproportionate share of allegations involves scientists of many countries, and even from respected laboratories. Misconduct destroys faith in science and scientists and creates a hierarchy of fraudsters. Investigating misconduct also steals valuable time and resources. In spite of these facts, most researchers are not aware of publication ethics.
Day 1 of the workshop ended with a presentation on ‘research and publication ethics’ by Dr. M. K. Unnikrishnan (College of Pharmaceutical Sciences, Manipal University, Manipal). He spoke on the essentials of publication ethics that included plagiarism (attempting to take credit of the work of others), self-plagiarism (multiple publications by an author on the same content of work with slightly different wordings), falsification (manipulation of research data and processes and omitting critical data or results), gift authorship (guest authorship), ghostwriting (someone other than the named author (s) makes a major contribution), salami publishing (publishing many papers, with minor differences, from the same study), and sabotage (distracting the research works of others to halt their research completion). Additionally, Dr. Unnikrishnan pointed out the ‘Ingelfinger rule’ of stipulating that a scientist must not submit the same original research in two different journals. He also advised the audience that authorship is not just credit for the work but also responsibility for scientific contents of a paper. Although some Indian Universities are instituting preventive measures (e.g., use of plagiarism detecting software, Shodhganga digital archiving of doctoral theses), Dr. Unnikrishnan argued for a great need to sensitize young researchers on the nature and implications of scientific misconduct. Finally, he discussed methods on how editors and peer reviewers should ethically conduct themselves while managing a manuscript for publication.
SCIENTIFIC COMMUNICATION: THE KEY TO SUCCESSFUL SELLING OF FINDINGS
Research outcomes are measured through quality publications. Scientists must not only ‘do’ science but must ‘write’ science. The story of the project must be told in a clear, simple language weaving in previous work done in the field, answering the research question, and addressing the hypothesis set forth at the beginning of the study. Scientific publication is an organic process of planning, researching, drafting, revising, and updating the current knowledge for future perspectives. Writing a research paper is no easier than the research itself. The lectures of Day 2 of the workshop dealt with the basic elements and logistics of writing a scientific paper.
An overview of paper structure and thesis writing
Dr. Amitabh Prakash (Adis, Auckland, New Zealand) spoke on ‘Learning how to write a good scientific paper.’ His presentation described the essential components of an original research paper and thesis (e.g., introduction, methods, results, and discussion [IMRaD]) and provided guidance on the correct order, in which data should appear within these sections. The characteristics of a good abstract and title and the creation of appropriate key words were discussed. Dr. Prakash suggested that the ‘title of a paper’ might perhaps have a chance to make a good impression, and the title might be either indicative (title that gives the purpose of the study) or declarative (title that gives the study conclusion). He also suggested that an abstract is a succinct summary of a research paper, and it should be specific, clear, and concise, and should have IMRaD structure in brief, followed by key words. Selection of appropriate papers to be cited in the reference list was also discussed. Various unethical authorships were enumerated, and ‘The International Committee of Medical Journal Editors (ICMJE) criteria for authorship’ was explained (http://www.icmje.org/ethical_1author.html; also see Table 1 in reference #9). The session highlighted the need for transparency in medical publication and provided a clear description of items that needed to be included in the ‘Disclosures’ section (e.g., sources of funding for the study and potential conflicts of interest of all authors, etc.) and ‘Acknowledgements’ section (e.g., writing assistance and input from all individuals who did not meet the authorship criteria). The final part of the presentation was devoted to thesis writing, and Dr. Prakash provided the audience with a list of common mistakes that are frequently encountered when writing a manuscript.
The backbone of a study is description of results through Text, Tables, and Figures. Dr. S. B. Deshpande (Institute of Medical Sciences, Banaras Hindu University, Varanasi, India) spoke on ‘Effective Presentation of Results.’ The Results section deals with the observations made by the authors and thus, is not hypothetical. This section is subdivided into three segments, that is, descriptive form of the Text, providing numerical data in Tables, and visualizing the observations in Graphs or Figures. All these are arranged in a sequential order to address the question hypothesized in the Introduction. The description in Text provides clear content of the findings highlighting the observations. It should not be the repetition of facts in tables or graphs. Tables are used to summarize or emphasize descriptive content in the text or to present the numerical data that are unrelated. Illustrations should be used when the evidence bearing on the conclusions of a paper cannot be adequately presented in a written description or in a Table. Tables or Figures should relate to each other logically in sequence and should be clear by themselves. Furthermore, the discussion is based entirely on these observations. Additionally, how the results are applied to further research in the field to advance our understanding of research questions was discussed.
Dr. Peush Sahni (All-India Institute of Medical Sciences, New Delhi) spoke on effectively ‘structuring the Discussion’ for a research paper. The Discussion section deals with a systematic interpretation of study results within the available knowledge. He said the section should begin with the most important point relating to the subject studied, focusing on key issues, providing link sentences between paragraphs, and ensuring the flow of text. Points were made to avoid history, not repeat all the results, and provide limitations of the study. The strengths and novel findings of the study should be provided in the discussion, and it should open avenues for future research and new questions. The Discussion section should end with a conclusion stating the summary of key findings. Dr. Sahni gave an example from a published paper for writing a Discussion. In another presentation titled ‘Writing an effective title and the abstract,’ Dr. Sahni described the important components of a good title, such as, it should be simple, concise, informative, interesting and eye-catching, accurate and specific about the paper's content, and should state the subject in full indicating study design and animal species. Dr. Sahni explained structured (IMRaD) and unstructured abstracts and discussed a few selected examples with the audience.
Language and style in publication
The next lecture of Dr. Amitabh Prakash on ‘Language and style in scientific writing: Importance of terseness, shortness and clarity in writing’ focused on the actual sentence construction, language, grammar and punctuation in scientific manuscripts. His presentation emphasized the importance of brevity and clarity in the writing of manuscripts describing biomedical research. Starting with a guide to the appropriate construction of sentences and paragraphs, attendees were given a brief overview of the correct use of punctuation with interactive examples. Dr. Prakash discussed common errors in grammar and proactively sought audience participation in correcting some examples. Additional discussion was centered on discouraging the use of redundant and expendable words, jargon, and the use of adjectives with incomparable words. The session ended with a discussion of words and phrases that are commonly misused (e.g., data vs. datum, affect vs. effect, among vs. between, dose vs. dosage, and efficacy/efficacious vs. effective/effectiveness) in biomedical research manuscripts.
Working with journals
The appropriateness in selecting the journal for submission and acceptance of the manuscript should be determined by the experience of an author. The corresponding author must have a rationale in choosing the appropriate journal, and this depends upon the scope of the study and the quality of work performed. Dr. Amitabh Prakash spoke on ‘Working with journals: Selecting a journal, cover letter, peer review process and impact factor’ by instructing the audience in assessing the true value of a journal, understanding principles involved in the peer review processes, providing tips on making an initial approach to the editorial office, and drafting an appropriate cover letter to accompany the submission. His presentation defined the metrics that are most commonly used to measure journal quality (e.g., impact factor™, Eigenfactor™ score, Article Influence™ score, SCOPUS 2-year citation data, SCImago Journal Rank, h-Index, etc.) and guided attendees on the relative advantages and disadvantages of using each metric. Factors to consider when assessing journal quality were discussed, and the audience was educated on the ‘green’ and ‘gold’ open access publication models. Various peer review models (e.g., double-blind, single-blind, non-blind) were described together with the role of the journal editor in assessing manuscripts and selecting suitable reviewers. A typical checklist sent to referees was shared with the attendees, and clear guidance was provided on the best way to address referee feedback. The session concluded with a discussion of the potential drawbacks of the current peer review system.
Poster and oral presentations at conferences
Posters have become an increasingly popular mode of presentation at conferences, as it can accommodate more papers per meeting, has no time constraint, provides a better presenter-audience interaction, and allows one to select and attend papers of interest. In Figure 2, we provide instructions, design, and layout in preparing a scientific poster. In the final presentation, Dr. Sahni provided the audience with step-by-step instructions on how to write and format posters for layout, content, font size, color, and graphics. Attendees were given specific guidance on the format of text on slides, the use of color, font type and size, and the use of illustrations and multimedia effects. Moreover, the importance of practical tips while delivering oral or poster presentation was provided to the audience, such as speak slowly and clearly, be informative, maintain eye contact, and listen to the questions from judges/audience carefully before coming up with an answer.
Guidelines and design to scientific poster presentation. The objective of scientific posters is to present laboratory work in scientific meetings. A poster is an excellent means of communicating scientific work, because it is a graphic representation...
PANEL DISCUSSION: FEEDBACK AND COMMENTS BY PARTICIPANTS
After all the presentations were made, Dr. Jagadeesh began a panel discussion that included all speakers. The discussion was aimed at what we do currently and could do in the future with respect to ‘developing a research question and then writing an effective thesis proposal/protocol followed by publication.’ Dr. Jagadeesh asked the following questions to the panelists, while receiving questions/suggestions from the participants and panelists.
Does a Post-Graduate or Ph.D. student receive adequate training, either through an institutional course, a workshop of the present nature, or from the guide?
Are these Post-Graduates self-taught (like most of us who learnt the hard way)?
How are these guides trained? How do we train them to become more efficient mentors?
Does a Post-Graduate or Ph.D. student struggle to find a method (s) to carry out studies? To what extent do seniors/guides help a post graduate overcome technical difficulties? How difficult is it for a student to find chemicals, reagents, instruments, and technical help in conducting studies?
Analyses of data and interpretation: Most students struggle without adequate guidance.
Thesis and publications frequently feature inadequate/incorrect statistical analyses and representation of data in tables/graphs. The student, their guide, and the reviewers all share equal responsibility.
Who initiates and drafts the research paper? The Post-Graduate or their guide?
What kind of assistance does a Post-Graduate get from the guide in finalizing a paper for publication?
Does the guide insist that each Post-Graduate thesis yield at least one paper, and each Ph.D. thesis more than two papers, plus a review article?
The panelists and audience expressed a variety of views, but were unable to arrive at a decisive conclusion.
WHAT HAVE THE PARTICIPANTS LEARNED?
At the end of this fast-moving two-day workshop, the participants had opportunities in learning the following topics:
Sequential steps in developing a study protocol, from choosing a research topic to developing research questions and a hypothesis.
Study protocols on different topics in their subject of specialization
Searching and reviewing the literature
Appropriate statistical analyses in biomedical research
Scientific ethics in publication
Writing and understanding the components of a research paper (IMRaD)
Recognizing the value of good title, running title, abstract, key words, etc
Importance of Tables and Figures in the Results section, and their importance in describing findings
Evidence-based Discussion in a research paper
Language and style in writing a paper and expert tips on getting it published
Presentation of research findings at a conference (oral and poster).
Overall, the workshop was deemed very helpful to participants. The participants rated the quality of workshop from “satisfied” to “very satisfied.” A significant number of participants were of the opinion that the time allotted for each presentation was short and thus, be extended from the present two days to four days with adequate time to ask questions. In addition, a ‘hands-on’ session should be introduced for writing a proposal and manuscript. A large number of attendees expressed their desire to attend a similar workshop, if conducted, in the near future.
We gratefully express our gratitude to the Organizing Committee, especially Professors K. Chinnasamy, B. G. Shivananda, N. Udupa, Jerad Suresh, Padma Parekh, A. P. Basavarajappa, Mr. S. V. Veerramani, Mr. J. Jayaseelan, and all volunteers of the SRM University. We thank Dr. Thomas Papoian (US FDA) for helpful comments on the manuscript.
The opinions expressed herein are those of Gowraganahalli Jagadeesh and do not necessarily reflect those of the US Food and Drug Administration
Source of Support: Nil
Conflict of Interest: None declared.
1. Jagadeesh G, Murthy S, Gupta YK, Prakash A, editors. 1st ed. Philadelphia: Wolters Kluwer Health-Lippincott Williams and Wilkins; 2010. Biomedical Research-From Ideation to Publication.
2. Balakumar P. Letter to the Editor. Biomedical Research. From Ideation to Publication. In: Jagadeesh G, Murthy S, Gupta YK, Prakash A, editors. J Nat Sci Biol Med. 1st ed. Vol. 3. Philadelphia: Wolters Kluwer Health-Lippincott Williams and Wilkins: 2010; 2012. p. 212.
3. Balakumar P, Marcus SJ, Jagadeesh G. Navigating your way through online resources for biomedical research. RGUHS J Pharm Sci. 2012;2:5–27.
4. Reisman F. Creative, critical thinking and logic in research. RGUHS J Pharm Sci. 2011;1:97–102.
5. Bartz CC. Getting Started with Research: Ideas to Research Process. RGUHS J Pharm Sci. 2011;1:176–9.
6. Kuberappa YV, Kumar AH. Knowing the known to understand the unknown-A systematic approach to reviewing the scientific literature. RGUHS J Pharm Sci. 2012;2:1–7.
7. Neville C. Referencing: Principles, Practice and Problems. RGUHS J Pharm Sci. 2012;2:1–8.
8. Balakumar P, Murthy S, Jagadeesh G. The basic concepts of scientific research and communication. Indian J Pharmacol. 2007;39:303–6.
9. Balakumar P, Jagadeesh G. The basic concepts of scientific research and scientific communication. J Pharmacol Pharmacother. 2012;3:178–82.[PMC free article][PubMed]
10. Kalmund P. Setting thesis research objectives. M.Sc. programme. Cited by Jenicek M. In: Jagadeesh G, Murthy S, Gupta YK, Prakash A, editors. Biomedical Research-From Ideation to Publication. 1st ed. Philadelphia: Wolters Kluwer Health-Lippincott Williams and Wilkins; 2010. pp. 27–34. Ch 3.