CITS5501 lab 2 (week 3) – Data-driven tests and test design – solutions

1. Parameterized tests

Normal test methods in JUnit don’t take any parameters at all – for instance, the testAdd method from last week’s code:

  @Test
  public void testAdd() {
      Calculator c = new Calculator(3, 4);
      int result = c.add();
      assertEquals(result, 7, "result should be 7");
  }

But in last week’s lab, we briefly saw an example of a parameterized test in JUnit – these do take parameters. The parameterized test was addZeroHasNoEffect, which checks, for a range of int values, that using the Calculator to add 0 to the number gives back the original int:

  /** Adding zero to a number should always
    * give us the same number back.
    */
  @ParameterizedTest
  @ValueSource( ints = { -99, -1, 0, 1, 2, 101, 337  })
  void addZeroHasNoEffect(int num) {
      Calculator c = new Calculator(num, 0);
      int result = c.add();
      assertEquals(result, num, "result should be same as num");
  }

What do the @ParameterizedTest and @ValueSource annotations on this test method mean?

Adding the @ParameterizedTest tells JUnit that this test needs to be run in a special way. In effect, this annotation tells JUnit: “You need to call this test multiple times. Each time you should call it with a different int.” Where does Junit get the ints from? We supplied a @ValueSource annotation which tells JUnit “Here is the list of ints to use with the test”. (But there are other ways of supplying the list of ints besides writing them out explicitly as we’ve done here – you could put them in a .csv text file, or write a special method that returns a “stream” of ints.1)

So to summarize: the test method, addZeroHasNoEffect, takes one parameter, an int, and each time the method is called, it is passed a different int from the list given by @ValueSource.

The code for this week’s lab is the same as last week’s, but with some new test methods, so you can try out the addZeroHasNoEffect test if you haven’t already:

Exercises

  1. Run the tests in your IDE or editor to confirm that all those ints are used – how can you tell?
  2. Try changing them and/or adding to the list.
  3. addZeroHasNoEffect is a single method. But (based on the material from lectures and the textbooks) is it also a single test case? If not, how many test cases does it comprise?

Sample solutions:

  1. Most IDEs should show you exactly what tests cases have been run – check the documentation for your IDE if it is not clear.
  2. No solution needed here; the aim is just to practice working with JUnit-based code.
  3. Seven test cases, because there are seven ints in the list.

Hopefully the use of @ValueSource seems straightforward. But a question may now arise: the addZeroHasNoEffect(int num) test method takes only one parameter, but what if we were working with a method that took two parameters (a String and an int, say). Or what if we wish to specify not just the test values, but also the expected values (a very common situation)? How could we supply multiple sets of parameters in that case?

Below is an example of such a method with multiple parameters. It’s taken from this week’s code, which includes an additional test method when compared with last week’s: tableOfTests(). The tableOfTests() method is another parameterized test; it draws its test values and expected values from a method called additionTestCasesProvider():

  @ParameterizedTest
  @MethodSource("additionTestCasesProvider")
  /** Test Calculator.add() with multiple test values
   * and expected values.
   *
   * @param num1 First test value to be passed to Calculator.add()
   * @param num1 Second test value to be passed to Calculator.add()
   * @param expectedResult The expected result for Calculator.add()
   */
  void tableOfTests(int num1, int num2, int expectedResult) {
      Calculator c = new Calculator(num1, num2);
      int result = c.add();
      assertEquals(expectedResult, result, "result should be same as as expected result");
  }

In the previous example, we used the @ValueSource annotation, which is used when you have a straightforward list of literal values you want JUnit to iterate over. But in tableOfTests, we’re using a new annotation, @MethodSource. If we annotate our test method with @MethodSource("additionTestCasesProvider"), we are saying to JUnit, “Go and call the additionTestCasesProvider method in order to get a list of test values and expected values”. You can read more about MethodSources in the JUnit documentation on writing parameterized tests.

This sort of testing is called data-driven testing – we have basically the same test being run, but with different values each time; so it makes sense to write the logic for the test just once (rather than four times).2

Let’s look at the additionTestCasesProvider() method which supplies the parameters used by tableOfTests(). The additionTestCasesProvider() method uses Java’s Stream API, introduced in Java 8. For the moment, you don’t need to know the details of how the Stream API works – just that it when used with JUnit, it gives us a convenient way of constructing “lists of lists”. In the code below, the arguments() method will take any number of arguments we like (here, three ints, comprising two test values and an expected value) and group them together into a list-like object (Arguments); and the Stream.of() method will take any number of Arguments objects we like, and group those together in a list-like object (of type Stream<Arguments>):

  static Stream<Arguments> additionTestCasesProvider() {
      return Stream.of(
          // the arguments are:
          //    num1, num2, and expected result.
          arguments(1, 2, 3),
          arguments(3, 7, 10),
          arguments(3, 7, 11)
          arguments(99, 1, 100)
      );
  }

Exercises

  1. How many test cases would you say tableOfTests and additionTestCasesProvider comprise?

  2. Read through the JUnit documentation on writing parameterized tests. (You might find the baeldung.com “Guide to JUnit 5 Parameterized Tests” by Ali Dehghani helpful as well.) If you have time, try experimenting with the different features it describes.

  3. In Java, enum types are used to represent types that can take on values from only a distinct set. By convention, the values are given names in ALL CAPS with underscores to separate words (also called “SCREAMING_SNAKE_CASE”).

    For instance,

    public enum Weekday {
  MON, TUE, WED, THU, FRI, SAT, SUN
}

    or

    public enum Color {
  RED, ORANGE, YELLOW, BLUE, GREEN, INDIGO, VIOLET
}

    (For more on Java enums, including how to create enums with constructors and fields, see the Java documentation on enum types.)

    Suppose we need to run a test which should be passed each Weekday in turn. Which JUnit annotation should we use for this?

  4. If you have used testing frameworks in languages other than Java – how do they compare with JUnit? Do they offer facilities for creating data-driven tests? Are these more or less convenient than the way things are done with JUnit?

Sample solutions:

  1. Together, they comprise four test cases: tableOfTests will get invoked four times, each time with different test values and expected result.
  2. No solution needed; the aim is just to familiarize yourself with JUnit’s parameterized tests.
  3. The JUnit documentation explains how the @EnumSource annotation can be used with enum types. By default, an @EnumSource will iterate over all the possible values of an enum type, calling the test method once with each value.
  4. Writing data-driven tests tends to be more convenient (and less verbose) in languages which either are not statically type-checked (for instance, JavaScript, Python and Ruby and Lisp-family languages) or which have rich type systems and type inference3 (like Scala, Ocaml and Haskell).
    But as we have seen, it is certainly possible in Java, and the Streams API allows for more concise code than in earlier versions of Java.

Challenge exercise

Based on this example, try writing your own parameterized tests for other methods (for instance, subtraction).

(Challenge exercises aren’t compulsory, but may be interesting or good practice if you’ve completed all other work in the lab.)

2. Preconditions and postconditions

Work through following scenario and exercises. After you’ve done so, it’s suggested you drop in to a timetabled lab sessions to discuss and compare your answers with other students, and/or obtain feedback from the lab facilitator.

Consider the following scenario:

Enrolment database

A database has a table for students, a table for units being offered, and a table for enrolments. When a unit is removed as an offering, all enrolments relating to that unit must also be removed. The code for doing a unit removal currently looks like this:

/** Remove a unit from the system
 */
void removeUnit(String unitCode) {
  units.removeRecord(unitCode);
}

If possible, it’s recommended you discuss the following questions with a partner or in a small group, and come up with an answer for each. But if that is not feasible, spend several minutes thinking about the questions yourself before sharing ideas with the class.

Exercises

  1. What preconditions do you think there should be for calling removeUnit()?
  2. What postconditions should hold after it is called?
  3. Does the scenario give rise to any system invariants?
  4. Can you identify any problems with the code? Describe what defects, failures and erroneous states might exist as a consequence.

If you don’t recall what preconditions, postconditions, and invariants are, you might wish to review the week 1 readings.

Sample solutions:

a. Preconditions

Preconditions for removeUnit to complete properly, and bring about the postconditions, might include:

However, note that we would not necessarily mention all of these in the Javadoc comment for removeUnit():

Other preconditions we need not document:

Tip: If asked in assignments or the exam to specify preconditions for a method, we will nearly always be using the Java language. If you specify preconditions like “The parameter String s must be a String” you generally (a) will not get any marks for saying so, and (b) will make the markers think you don’t understand how Java works.

Alternative solutions:

If you are not clear on what preconditions are from the lecture slides, you might want to read chapter 3 of Object-Oriented Design and Patterns (2nd edn) by Cay S. Horstmann. (This can be found in the “Unit Readings” via the LMS.)

Sample solutions, cont’d:

b. Postconditions

The postcondition here could be stated as:

c. System invariants

Recall that invariants are assertions that should hold true before and after every method call of a class (or, if we are describing invariants for a whole subsystem or system, for all methods of classes in the subsystem or system).

From what we are told, we can infer that there is at least the following invariant (presumably, for the whole system):

There might be others as well.

d. Problems with the code

We are not told what units.removeRecord() does, exactly.

However, if we make the following assumption:

then there is a problem with the code: it should have updated the enrolments table, to remove any references to the deleted unit.

So a defect in the code is that it does not call whatever methods are necessary to delete records from the enrolments table. (This is a static property of the code.)

The system enters an erroneous state immediately after the removeUnit method call, because now one of the system invariants is not satisfied – the system is inconsistent.

As for failures: recall that these are ways in which the system observably departs from its specification. It is likely that no failure will occur directly after the removeUnit method call. But the next time someone queries the system to see what units a student was enrolled in: they may be recorded as being enrolled in a non-existent unit, which likely is a failure.

If you are not clear about the difference between faults, failures, and erroneous states, you might want to read chapter 11 of Bruegge and Dutoit, Object-Oriented Software Engineering Using UML, Patterns, and Java (3rd edn), particularly section 11.3, “Testing Concepts”.

Alternative solution:

3. Testability

Consider the following, each of which is supposed to be a requirement or specification. Do you think it would be straightforward to write tests for them? If not, why not? After answering these questions, you might like to drop in to a timetabled lab session to compare your answers with other students. Although model solutions will be provided, there is often no single definitive answers to software engineering questions such as these – different students may come up with slightly different answers.

  1. The flight booking system should be easy for travel agents to use.
  2. The int String.indexOf(char ch) method should return a -1 if ch does not appear in the receiver string, or the index at which it appears, if it does.
  3. Internet-aware Toast-O-Matic toasters should have a mean time between failure of 6 months.

Sample solutions:

System requirements:

a. “Easiness of use” requirement

This would be difficult to test.

If you are not clear about what makes a good requirement, you might want to review the chapter from the Pressman textbook on “Understanding Requirements” (in the 9th edition) or “Requirements Engineering” (in earlier editions).
There is also a quick summary (taken from documentation for an IBM requirements management product) available here.

  1. You can find a list of all the possible annotations you can use for supplying parameters to parameterized tests in the JUnit API documentation – look down the bottom of the page for class names ending in “Source”. The API documentation is fairly terse, though – if you’d like to see examples of how the different sources are used in practice, check out the “Guide to JUnit 5 Parameterized Tests” by Ali Dehghani.↩︎

  2. This is an example of the “DRY” principle (“Don’t Repeat Yourself”) in action. Writing out the same test method four times, but each time with different test values and expected values, would just lead to unmaintainable code.↩︎

  3. Type inference is a feature of some programming languages which allows the programmer to omit types where compiler can work out what they should be.
    The Java language has only a very limited form of type inference. It allows you to omit type arguments to generic types, in some cases – e.g. Map<String, List<String>> myMap = new HashMap<>() doesn’t need type arguments after new HashMap – and (since Java 10) allows you to omit local variable declarations, instead using var. See the Oracle documentation on generics type inference and on Java 10 local variable type inference↩︎