Automated Software Engineering Research Group @NCSU

On reviewing a student's paper drafts

We will have three reviewing phases of your paper drafts:

Phase 1: Submit your draft to me for me to review the ideas (in the phase, I won't fix your writing but focus on your ideas, just like the way a PC member reviews your draft). After you receive my feedback, you revise your draft based on my feedback. Then you repeat Phase 1 till I explicitly tell you to do a peer review (moving on to Phase 2). When you send me a new version via iterations, you need to highlight the changed parts that need my attention.

Phase 2: Submit your draft to a peer student for the student to review both the writing and idea. After you receive the feedback from the student, you revise your draft based on the  feedback. Move on to Phase 3.

Phase 3: Submit your draft to me for me to review both the writing and ideas. Note that I will review your writing only after you have gone through Phase 2.

This policy is also applicable to the intermediate writing during the research development besides the final writing before the submission deadline.

I want to point out another thing on intermediate writing during research development. Many students sit too long on a writing phase before actual tool development or evaluation. I expect the turnaround time for submitting your writing to me in a cycle of one week or two weeks. But some students take multiple weeks to prepare the writing to submit to me. Doing so loses the point of the original intention: getting in-time and early feedback on your work.

Here are my more explicit guideline on intermediate writing during the life cycle of research development:

(1) Prepare the abstract, introduction, example, and related work sections and submit them to me.

--> Start tool development

(2) Prepare the approach section and submit it to me

--> Continue tool development

(3) Refine the approach section and prepare the evaluation design section and submit them to me

--> Finish tool development
--> Conduct evaluation

(4) Prepare the evaluation results section and submit them to me

--> Now you shall have a relatively complete draft after you add the conclusion section and discussion section if needed.

Additional points:

*. In each later phase, you should refine and improve sections written during early phases.

*. Before you write down your ideas (on your approach or evaluation design, etc.) in your paper as formal writing, you need to discuss with me to get my OK stamp on the things that you are going to write down. I had been through some cases where a student spent a lot of time in writing down what the student thought to be a reasonable idea before discussing with me on the idea. After I read the writing, I rejected the idea with good reasons. Then the student basically had to discard the old writing (with a lot of time previously being wasted). That is, don't rely on only the formal writing as the only media for communicating with me with your ideas or work. You need to fully discuss with me with what you plan to write down in your formal writing.

Again, remember that you shouldn't sit on the writing for one of the above phases for too long.

Posted at 10:53PM Oct 18, 2008 by XIE, TAO in Technical Writing  |  Comments[0]

Why shall we (research advisors) avoid directly writing on students' research papers?

When students started their research in my research group, I told them that "Don't expect me to do tool implementation for your projects", "Don't expect me to do experiments for your projects", and "Don't expect me to write sections of your papers for you". Indeed, I didn't say that "Don't expect me to give you research ideas to work on" since it is very tough for students to come up with good research ideas to work on when they first start their research. At the same time, I have tried different ways of giving students space and opportunities to think about their own research ideas. Such ways are taking good effect and many students are improving themselves in coming up with good research ideas. That is a separate topic here and deserves another post of discussion.

Why shall I (as a research advisor) avoid directly writing on students' research papers?

Two main reasons. First, if I write for my students, they never get a chance to learn how to write themselves. If they don't know how to write papers, how can they write their own dissertation? The advisor cannot write the dissertation for them! How can they write their research papers after they graduate? The advisor cannot accompany the students for all their professional career.

Second, a bit selfish reason. I don't have much time to write papers for students: I have proposals to write (so far few of my students can write proposals for me but I have been trying to train them to have the capability when they get close to graduation), I have professional services to serve, I have a good number of students to work with, advise, and train, ...

Although I told students that I didn't write for them, I spent much effort in training them how to write and how to improve their writing. Below are several things that I do:

*. I mark my revisions and suggestions on hardcopy of their writing submitted for my review and revision; I don't directly write on the LaTeX source files or Word files (whatever format is used in paper writing). Doing so can "force" students to at least recognize and understand what I marked and commented.

*. I walk through with the students in person on some major issues being marked and tell them the reasons for my revisions. These reasons may not be written down on the hardcopy as comments. Basically, I want them to learn the corrections not only for the particular mistakes that the students made but also for future similar types of mistakes that the students shall avoid making. At the same time, I ask the students whether they have any doubts or questions on my marks and I then address their doubts and questions if any. Smart students will always take opportunities to ask me the reasons for revision places that the students don't understand.

*. Before requesting my review or revision of a student's paper, the student should ask a peer student to review and revise the student's paper. Doing so not only allows other students to be trained with review skills but also reduce my revision load on picking up those low-level, easy-to-spot writing issues. When I finish my revisions on the hardcopy, I expect that the peer student who reviewed the paper also look at my revisions and asked themselves "why didn't I catch these issues?" Another major motivation for reducing my revision load is that when a paper includes too many low-level writing issues, I would be busy in correcting those low-level writing issues and forget about the logical flows and high level technical content issues. That is, the student, the author of the paper, will lose much of the benefit given by me in improving his or her paper. Including too many low-level writing issues in a student's paper before submitting it for my revision would just do harm to the student, not being able to take full advantage of my advice on their high level, more important, writing issues and research issues.

*. I will pay much attention to the abstract and introduction sections, whose writing is very important and students often don't do a good job. These parts often require much more iterations of my revisions than other parts of the paper. Students often don't do a good job in organizing the logical flow and the organization or reasoning of the motivations.

*. I will ask students to turn their common mistakes into some regular expression patterns, and then use a tool called style-check developed by Neil Spring so that the students can guard against some low-level issues themselves (if they cannot guard against these issues manually easily). Frankly, I found that several of my students still make many low-level mistakes repeatedly and they don't use style-check as I suggested. I hope these students can take full advantage of tools to help address the low-level issues of their writing, and exploit and focus my guidance on the high-level issues of their writing.

Next I discuss what I do to help students to write early and write enough so that I can train their writing.

Many of you may wonder: when deadlines are coming and students are lagging behind in their writing, we would have to jump in to make these deadlines by writing the papers for the students since we really don't want to miss these important deadlines. Indeed, I had been through these situations.

One possible way to deal with that is to really impose strict policies to students: if they don't prepare their papers early enough for us (advisors) to review or revise, let's just bypass the deadlines. In principle, bypassing an important deadline will do more harm to students than the advisor: the advisor has multiple students to work with and have multiple papers to submit, but the students individually may just have a limited number of papers to submit yearly or in their whole graduate program. Each year, there exist only several top conference paper deadlines and a small number of good or right conferences for the work. Missing some important deadlines produces very negative impact on students' research record building (indeed as well as the advisor's), given that often the time papers will be rejected by top conferences.

But the above way often doesn't work (either for the advisor or the student). The advisor doesn't want to miss important deadlines either. The students also want to make the deadlines but their deadline-making or engineering skills (c.f. my talk slides on research skills) are often not good. Often the time, the students postpone their paper writing to the last minute; they often don't like paper writing.

Then how can I help students to write early? Here is what I do.

*. After a student (or I, or both) has some promising research idea, and both I and the student feel that the idea could have a high chance of leading to a good research project, before the student goes ahead to carry out the implementation of the approach, I will ask the student to write down the following sections (see my slides on writing papers for a paper's structure):
   -- Abstract
   -- Introduction
   -- Example
   -- Approach (the description of the approach could be high level)
    (No implementation section or evaluation section is required at this stage)
   -- Related work
   -- Conclusion

If the student can produce some preliminary results (e.g., by doing some manual walk-through of the approach), the student can also put in a "preliminary results" section. The resulting draft would form a fine workshop submission draft (but note that it is not necessary for us to submit it as a workshop submission unless really needed in some cases to gather early feedback on the work from the community).

Basically, the student presents to me a "design" document (similar to one in software development) before the student starts implementation. Then I can revise and understand what the student is planning to implement, and brainstorm with the student on possible new ideas or improvement of the existing ideas based on the writing. Doing so can avoid some of the past issues on a student's approach or research project being a bit "black-box" to me and I found out issues when reading the student's draft when being very close to the deadline and then the student couldn't fix the issues that I identified.

Note that, at this stage, the student must identify a convincing motivating example and describe it in the example section before moving on to implement the approach. I had one past bad case where a student couldn't find a convincing motivating example for his approach and then moved on to the implementation of the approach, and ended up with an unsuccessful project.

*. After the student implements the approach, the student expands and refines the approach section and adds the implementation section. I will review and revise these sections. Note that in this step when the student does some hands-on stuffs, new ideas may come up and the originally written down approach can be revised and improved. I am very supportive on the student's hands-on experience and generating new ideas from hands-on experience. But that doesn't mean that the student should directly dive into tool implementations without thinking enough or finishing  the first step of paper writing described above. In my belief, good new ideas can come from both creative thinking before tool implementation and during/after tool implementation.

*. Before the student carries out the evaluation, the student needs to write the experimental design subsection and other subsections in the evaluation section that can be written before the experimental results are available. Doing so can help me make sure that the student is doing the right things in the evaluation.

*. After the student finishes the evaluation, the student fills in the experimental results subsection. I will review the new subsections as well as the whole draft before submission.

Doing this style of phased writing, students benefit in several major ways. (1). They don't rely on me to write for them before the deadline (since I won't). (2). They allow me to have opportunities to iterate with them on their writing and improve their writing since in the early phases, we don't have the deadline pressure. (3). They can gather feedback from me early on to help them to avoid making wrong decisions or misinterpreting my suggestions in their research development. (4). Our remote research collaborators (if any) can read through the early formal writing to give us feedback and suggestions on the research development.

So far I have quite good experience in training and educating students to write research papers with the above ways. In the end, I indeed enforce what I told my students "Don't expect me to write sections of your papers for you". More importantly, most of the students (after working with me for a short period of time) in my group can independently write whole papers without my writing there (with my helps only in a way of my not-very-significant revising on hardcopies of their papers). I am very glad to see that happen while knowing that many of my fellow junior faculty members are still busy writing papers for their students.

Hope my experiences described above can further help my students to understand why I am doing so and further improve what they are doing while working with me, and help other advisors at least to think how to do better along this line of advising students in writing papers...

Posted at 10:18PM Oct 12, 2008 by XIE, TAO in Technical Writing  |  Comments[1]

More on formal writing before one-on-one meetings

Here is the definition on formal writing:

 

(1). The formal writing includes the text that you can turn into a part of your future paper submission directly or with minor polishing. If you just write in some high-level bulleted points like those in slides, this type of writing is not formal and not acceptable in terms of formal writing.

 

(2). Because our group uses LaTeX as the format of writing papers, your formal writing needs to be in the LaTeX format. If you don?t know how to use LaTeX in writing papers, take a look at

http://people.engr.ncsu.edu/txie/publications/writingtools.html

Especially on which software packages to use for editing and compiling LaTeX source files.

 

(3). Because our group uses CVS to keep track of revisions and allow collaborative writing, your formal writing needs to be put in our research server?s CVS repository. Basically after you set up CVS, you can create a subdirectory under /cvs/root/papers/ with the naming convention of ?lastname-conferenceorworkshopname? (e.g., acharya-FSE07). If there is no specific conference or workshop to aim at currently, you can put the name of your project/tool/topic in the place of ?conferencworkshopname?. For info on how to set up CVS and use Eclipse to checkout CVS, take a look at:

http://ase.csc.ncsu.edu/server.html#cvssetup

 

Then your submission of your formal writing is an email including some words like ?My formal writing so far is included in the CVS directory XXXXXX. You can check it out.?

 

Basically you can view the formal writing that you submit before our one-on-one meetings as a portion of the paper that you are going to submit eventually. Week after week, you will expand the draft by filling in additional text that describes what you have done in the preceding week(s) and in the upcoming week(s).

 

Note that initially or early in the phase of your formal writing, you shall write the abstract, introduction, example sections early on. In addition, you may also start writing the related work section when you read other researchers? papers early on.  Writing these preceding sections doesn?t require any tool implementation or experiment. Then along the way of week-by-week work, you fill in the approach/implementation sections when you have more implementation details figured out and more development work done, you fill in the experiment setup and design sections when you try to set up your experiment, and you fill in the experimental results section when you finish producing experimental results, ?

 

This mechanism is to fix several issues being faced nowadays.

 

(a). students tend not to write serious/formal text along the way but put a lot of efforts in formal writing immediately before a submission deadline. Then the students cannot get helps from me on their writing early on.

 

(b). students tend not to disclose sufficient technical details or progresses of their projects along the way during one-on-one meetings week by week. Often immediately before the deadline, some students gave me ?surprises?, disclosing to me that they didn?t do some part that they were expected (by me) to do or they did something in an un-optimized or incorrect way; then it is often too late to fix these issues when getting too close to a deadline.

 

(c). when students don?t write things down in formal writing, they don?t have good feeling in the approach/tool design, experiment design, ? I often come up with good new ideas when I formally write down ideas in my proposals and I expect students to enjoy similar benefits by doing formal writing along the way.

Posted at 07:10PM Oct 24, 2007 by XIE, TAO in Technical Writing  |  Comments[0]

Written materials prepared before one-on-one weekly meetings

I recently talked to a colleague, Dr. Nagiza Samatova, who kindly shared her experience in training students' writing, and inspired by her way of training, I have tried to install a similar mechanism in my research group. I suspect that it will solve some students' issues in delaying writing in the last minute and turning their research as a black/grey box to me. Below is adapted from my email sent to some students in my group who have already had some concrete research projects ongoing:

Before one-on-one student meetings, the advisor requires the student to bring formal technical writing on the things to be discussed: the written materials later will be turned into a part of a paper submission so it is not wasteful or just specific for being used in one-on-one meetings.

For example,

-- if you plan to discuss a new idea that you may have, write paragraphs describing it, which can be turned into the introduction section, example section, or approach overview section of your future paper.

-- if you plan to discuss about design and implementation of your approach, write paragraphs describing these designs or implementations, which can be turned into the approach and implementation sections of your future paper.

-- If you plan to discuss about your evaluation, write paragraphs describing your experiments (either experiment setup, design, subjects, or results), which can be turned into the experiment section of your future paper.

-- If you plan to discuss other related papers that you read, write paragraphs describing them and the differences of them with your own approach, which can be turned into the related work section of your future paper.

In any case, you shall prepare your writing and present it to me along the way of weekly one-on-one meetings rather than a big bang in the end immediately before the deadline. Doing so can allow me to (1) give you early feedback on your work and writing and to (2) keep track of your work since currently your work?s technical progress is more a black/grey box to me.

 

In addition, this mechanism would be also very helpful to yourself in keeping yourself in having the habit of writing things down more formally (when you try to write things down more formally, you can have a better idea and generate new good ideas).

 

I expect you to send me an email telling me the sections/paragraphs in LaTeX in your paper in a specific CVS paper directory ** not later than the same morning ** of an afternoon one-on-one meeting. I don?t accept informal writing being put in the body of an email message or any way other than the preceding specified way.

 

If you cannot prepare such writing before a one-on-one meeting, I would suggest you to cancel or postpone that week?s one-on-one meeting with me. If you cancel or postpone too many weeks? meetings, the implication of reflecting your work progress/performance is self-evident.

Posted at 02:36PM Oct 13, 2007 by XIE, TAO in Technical Writing  |  Comments[0]

One pitfall in writing the related work section

In the related work section or elsewhere in a paper, many people tend to compare the proposed approach with the existing approaches in a shallow way, making the justification weak.

For example, your approach may be a static analysis approach and when you compare with previous dynamic analysis approaches, it is bad for you just say "Our approach based on static analysis are different from the existing approaches because they are based on dynamic analysis." It is not enough. You should describe what observable inputs/outputs (benefits) that the internal technical differences in your approach could bring. Here, it is better to say "Our approach based on static analysis are different from the existing dynamic analysis approaches because they require runtime setup and good sets of test suites, which are not required by our approach." You can see that the difference is on the observable inputs.

When you compare your dynamic analysis approach with existing static analysis approaches. You can focus on the difference of the observable outputs. You may say "... because our approach uses precise runtime information, reducing false positives that are produced by existing static
analysis approaches."

This issue has been raised in reviews for one of our past submissions.

"you omit effectiveness rates of a lot of tools in favor of "ours technically works differently on the inside". That is not quite good enough;
the comparison should be on result, not just on "we are technically different"."

You should go through your related work section to see whether you described the differences in observable results/benefits outside beyond just
describing technical differences inside.

Posted at 02:32PM Jun 03, 2007 by XIE, TAO in Technical Writing  |  Comments[0]

How can you write "much"?

It is common that students cannot write much when they start writing their first several research papers. Usually conference papers shall have 10 or 11 pages IEEE or ACM conference format but students often don't know how to fill in writing for that many pages.

Based on my experience with improving students' writing, here are some tips for expanding their papers to be longer, richer, and more professional.

For advice in writing each section of a paper, see my slides at:
http://www.csc.ncsu.edu/faculty/xie/publications/writepapers.pdf
Next I am talking about some advice on expanding students' writing on each section.

*. Abstract
An abstract usually consists of the following parts: short motivation (problem), proposed solution, evaluation, and evaluation results. For more detailed guideline on writing an abstract, please see:
http://www.lightbluetouchpaper.org/2007/03/14/how-not-to-write-an-abstract/

*. Introduction
A short introduction can be expanded by adding more background on the problem and how related work addresses (or doesn't address) the problem, the overview of your approach, the evaluation of your approach, and the evaluation results. You can list the main contributions of the paper there and lay out the structure of the rest of the paper in the end of the introduction section.

*. Example
Put an example section in your paper!! Some students still didn't put such a section there. Explain the details of the example and how your approach would work on the example. You can have a background section before the example section if there is much background info to present. But you can also consider to use your example to illustrate the background info.

*. Framework (or Approach)
First give a high-level overview of the approach (no need to turn it into a subsection but just put it as the text immediately after the framework section and before the first subsection. Use a diagram to show the components of the framework. You may have some bullet points to summarize the key features of each component. Then you can put each subsection for each component of the framework. You can use examples (which can be the same as the one described in the example section) to illustrate some key points of each component.

Many students initially used just a few sentences to describe the techniques in each component. When you see that your description is too short, you may see whether a reader could understand the details. Another way to think about it is to see if you provide enough details so that readers can reimplement your approach (not necessarily with the tools or libraries that you will describe in the implementation section).

Sometimes some components may be third-party tools. Many students have the perceptions that they don't need to describe these tools in details but refer the readers to read the cited papers. This is wrong. Your paper needs to be self-contained: if some techniques from third-party are used as an important part of a component, you need to provide enough details. Of course, you need to explicitly cite related papers when these components are not developed by you for the work in this paper.

*. Implementation
Usually this section is not too long. Some common mistakes that students make are to repeat many things that have been described in the framework section. You don't need to. You can list the components and then describe what third-party tools you use or how you implement them. But don't make your implementation section too short. You can double check to see whether you provide enough details so that readers can reimplement your approach exactly as the implementation that you have developed.

*. Experiment
Many students forgot to put in the necessary subsections there such as Objectives, Metrics, and Experiment Setup (including subjects and procedure, etc.). Many students simply put "Experimental Results" there. Even when they put experimental results there, they simply put figures or tables for the results and use a few sentences to describe the results. Students should describe in details what figure x/y axis mean, what table columns mean, etc. Many students rely on readers to read the figures or tables directly without using text to describe them.

In addition, students need to summarize how the experimental results answer the question/objective posed in the beginning of the section.

*. Discussion
Many students don't have a discussion section. Sometimes there are complications or future work, and this discussion section could be good to include these things.

*. Related work
Many students simply list the names of related projects/tools or just one sentence to describe related projects/tools. Students should describe more details about these related projects/tools. One paragraph for one related project/tool is even acceptable for a very related project/tool. In addition, students often forget to compare the difference between the discussed related project/tool with the new project/tool. Usually, the related work section can be easily substantially expanded (of course, when you go beyond your page limit, you can come back to cut short your related work section).

*. Conclusion
Conclusion is usually not long but not too short. Many students have a very short conclusion. It can have similar structure as the abstract: you need to say the motivation/problem, how your tool addresses the problem, the components that you tool has, the evaluation that you conducted, the evaluation results that you got, etc. Optionally you can state your future work here.

*. References
Students tend to cite too few papers, partly because they include too few related projects/tools in the related work section, without knowing enough relevant literature. To fix that, of course, students need to collect more related papers. One way is to upload their PDF draft to http://mondego.calit2.uci.edu/ra/ to collect relevant papers based on text or keyword similarity. Of course searching google with some keywords is also very effective. More other tips are described in my slides on "advice on writing research papers" whose link is listed earlier.

Another way to expand references is to cite representative papers on the same (big) field (e.g., regression testing, test generation, static bug detection, ...) in the introduction.

Final note:

One common bad writing style that I found among students in their early writing is over-use of bullet points. They tend to list things in bullet points. Normally in my papers, except for the main contributions that I want to list in the introduction section, I don't use bullet points often.  Too many bullet points make your writing not so professional. A similar listing effect can be achieved well by using some highlight (bolded) keywords in the beginning of paragraphs (but not bullet points). Sometimes when I try to list things, I simply use "First, ..... Second, .... Third, ....". When these different parts are long enough, I will make paragraphs and start these paragraphs with "First, .." "Second, ...." "Third, ....".
 

Posted at 02:06AM Apr 24, 2007 by XIE, TAO in Technical Writing  |  Comments[0]