How to run our seminars

Purpose of seminars

The purpose of a seminar is to read and discuss papers critically:

  • Distinguish main contributions
  • Comment on the appropriateness of methods
  • Discuss the validity and generalizability of the results
  • Generate questions and discuss potential answers
  • Draw inferences and think of future work
  • Understand (retrospectively!) the paper’s importance

Discussing a paper: Moderator’s duties

  1. Have the audience read the paper and come up with a list of questions
  2. Prepare a short presentation of the paper
  3. Prepare a list of questions about the paper
  4. While the discussion is running:
    • Ensure that everyone participates
    • Ensure that everyone’s opinion is being heard
    • Keep track of time per participant
    • Take notes of opinions and discussion points
  5. After the discussion: summarize the main points discussed

Discussing a paper: participant’s duties

  1. Read the paper before the discussion and come up with a set of questions or discussion points
  2. Annotate parts of the paper (e.g., underline important sentences), especially those that are surprising to you
  3. Write a short review / commentary about the paper
  4. Ask clarification questions to the moderator
  5. Actively think about, take notes and participate to the discussion

What is a good discussion question?

A good question:

  • Does not have a binary (i.e., yes/no) or quantitative (i.e., 15) answer
  • Encourages participants to explain “how”
  • Motivates participants to make connections to things they know already

Types of good questions:

  • Comparative: Contrast various approaches / aspects / opinions
  • Connective: Link examples, facts, theories etc
  • Critical: Break down an argument, provide counter arguments

Example questions

D Characterize the following questions:

  • What aspects of software complexity does the paper capture?
  • Using what techniques could the author have analyzed the results better?
  • How many projects did the author analyze?
  • Is statistical test X suitable for this paper?
  • How would you replicate this paper?

Role playing

In some cases, actually assigning roles may help the discussion:

  • Problem or Dilemma poser: The moderator
  • Reflective analyst: Keeps track of the discussion and periodically sums it up
  • Devil’s advocate: When consensus emerges, formulates an alternative view
  • Detective: Listens carefully for unacknowledged biases that seem to be emerging

Another form of role playing is debating. Groups are assigned two opposing views and try to defend them until an agreement is reached.

Tips for being moderator

  • Encourage everyone to participate:

    • Make arrangements so that everyone is heard
    • Ask questions to specific people
    • Never criticise any opinion. Welcome inadequate answers.

  • Help participants summarize and articulate what they ’ve learned

  • Be honest, open and inviting.

  • Most importantly: Keep notes!

Our reading sessions

Befor each session:

  • Everybody reads the paper for the reading session
  • A group presents a short introduction to the paper
  • Everybody participates in the discussion

At the end of each session:

  • This document is returned to the teachers, in Rmd format, for inclusion in the course web page.

A seminar example

The paper

We will discuss the paper by Pradel and Sen “DeepBugs: A Learning Approach to Name-based Bug Detection” [1]

  • Appeared at OOPSLA 2018
  • Cited 71 times
  • Importance: First convincing argument of using NLP techniques to find bugs

The authors

  • Michael Pradel: prof at U. Stuttgart; works on static analysis and (lately) on ML4Se
  • Koushik Sen: prof at UC Berkeley; works on static/dynamic analysis

Motivation

Most static analysis tools ignore identifier names, and this leads to simple bugs go unnoticed.

Q Can you spot the bug?

function setTimeout(callBack, delay){...};

browserSingleton.startPoller(100,
  function(delay, fn) {
    setTimeout(delay,fn);
  });

A The order of arguments in setTimeout is reversed.

Q Could a static analysis find it?

A Not without type hints

Idea: How about training an ML model to detect those kinds of bugs?

Research method

What does the paper do?

The paper trains an ML model to detect a single type of bugs.

  • Code is represented as embeddings
  • The ML model is trained on a binary classification task
    • It tries to predict whether a piece of code is correct or not
  • An pair of ML / data augmenters is created per bug type

ML4SE techniques used

How does the paper use ML techniques?

  • Identifier embeddings, to capture semantic similarities from similar usage contexts (e.g. len and count are similar)
  • Lightweight AST parsing
  • Simple feed forward net with 2 layers
  • Data augmentation: data is missing for buggy code, but this can be solved by injecting bugs

Results

What does the paper find?

  • Depending on the bug detector, accuracy scores between 89% – 94%
  • The trained bug detectors can find bugs on unseen before code

Implications

Why are the results important?

The paper shows that:

  • NL in source code contains rich semantic information that ML models can capture
  • Training bug detectors can be done even when ground truth data does not exist

Technical questions

3-4 questions to gauge the audience’s understanding of the paper

  • How is DeepBugs different from a static analysis tool?
  • Why do the authors use embeddings?
  • How do the authors inject “bugs” in the code?

Discussion points

3-4 questions to trigger general discussions about the paper

  • How would a different NN architecture change the results?
  • (How) Can we automate the creation of bug detectors in the DeepBugs style?
  • The authors use a fixed-size input vocabulary. What are the drawbacks of this approach?
  • What would be an approach to making this model a product?

Summary of the discussion

to be filled in after the discussion by the discussion group

References

[1]
M. Pradel and K. Sen, “DeepBugs: A learning approach to name-based bug detection,” Proc. ACM Program. Lang., vol. 2, no. OOPSLA, pp. 147:1–147:25, Oct. 2018.
[2]
S. A. Stuff, “Leading an interactive discussion: Elective seminar,” Case Western University, 2016.