BigBinary Blog

Ruby 2.5 supports measuring branch and method coverages

Ruby comes with Coverage, a simple standard library for test coverage measurement for a long time.

Before Ruby 2.5

Before Ruby 2.5, we could measure just the line coverage using Coverage.

Line coverage tells us whether a line is executed or not. If executed, then how many times that line was executed.

We have a file called score.rb.

1score = 33
2
3if score >= 40
4  p :PASSED
5else
6  p :FAILED
7end

Now create another file score_coverage.rb.

1require "coverage"
2
3Coverage.start
4load "score.rb"
5p Coverage.result

We used Coverage#start method to measure the coverage of score.rb file. Coverage#result returns the coverage result.

Let's run it with Ruby 2.4.

1$ RBENV_VERSION=2.4.0 ruby score_coverage.rb
2:FAILED
3{ "score.rb"=> [1, nil, 1, 0, nil, 1, nil] }

Let's look at the output. Each value in the array [1, nil, 1, 0, nil, 1, nil] denotes the count of line executions by the interpreter for each line in score.rb file.

This array is also called the "line coverage" of score.rb file.

A nil value in line coverage array means coverage is disabled for that particular line number or it is not a relevant line. Lines like else, end and blank lines have line coverage disabled.

Here's how we can read above line coverage result.

• Line number 1 (i.e. 0th index in the above result array) was executed once.
• Coverage was disabled for line number 2 (i.e. index 1) as it is blank.
• Line number 3 (i.e. index 2) was executed once.
• Line number 4 did not execute.
• Coverage was disabled for line number 5 as it contains only else clause.
• Line number 6 was executed once.
• Coverage was disabled for line number 7 as it contains just end keyword.

After Ruby 2.5

There was a pull request opened in 2014 to add method coverage and decision coverage metrics in Ruby. It was rejected by Yusuke Endoh as he saw some issues with it and mentioned that he was also working on a similar implementation.

In Ruby 2.5, Yusuke Endoh added branch coverage and method coverage feature to the Coverage library.

Let's see what's changed in Coverage library in Ruby 2.5.

Line Coverage

If we execute above example using Ruby 2.5, we will see no change in the result.

1$ RBENV_VERSION=2.5.0 ruby score_coverage.rb
2:FAILED
3{ "score.rb" => [1, nil, 1, 0, nil, 1, nil] }

This behavior is maintained to ensure that the Coverage#start API stays 100% backward compatible.

If we explicitly enable lines option on Coverage#start method in the above score_coverage.rb file, the coverage result will be different now.

1require "coverage"
2
3Coverage.start(lines: true)
4load "score.rb"
5p Coverage.result

1$ RBENV_VERSION=2.5.0 ruby score_coverage.rb
2:FAILED
3{ "score.rb" => {
4    :lines => [1, nil, 1, 0, nil, 1, nil]
5  }
6}

We can see that the coverage result is now a hash which reads that the score.rb file has lines coverage as [1, nil, 1, 0, nil, 1, nil].

Branch Coverage

Branch coverage helps us identify which branches are executed and which ones are not executed.

Let's see how to get branch coverage.

We will update the score_coverage.rb by enabling branches option.

1require "coverage"
2
3Coverage.start(branches: true)
4load "score.rb"
5p Coverage.result

1$ RBENV_VERSION=2.5.0 ruby score_coverage.rb
2:FAILED
3{ "score.rb" =>
4  { :branches => {
5      [:if, 0, 3, 0, 7, 3] => {
6        [:then, 1, 4, 2, 4, 15] => 0,
7        [:else, 2, 6, 2, 6, 15] => 1
8      }
9    }
10  }
11}

Here is how to read the data in array.

1[
2  BRANCH_TYPE,
3  UNIQUE_ID,
4  START_LINE_NUMBER,
5  START_COLUMN_NUMBER,
6  END_LINE_NUMBER,
7  END_COLUMN_NUMBER
8]

Please note that column numbers start from 0 and line numbers start from 1.

Let's try to read above printed branch coverage result.

[:if, 0, 3, 0, 7, 3] reads that if statement starts at line 3 & column 0 and ends at line 7 & column 3.

[:then, 1, 4, 2, 4, 15] reads that then clause starts at line 4 & column 2 and ends at line 4 & column 15.

Similarly, [:else, 2, 6, 2, 6, 15] reads that else clause starts at line 6 & column 2 and ends at line 6 & column 15.

Most importantly as per the branch coverage format, we can see that the branch from if to then was never executed since COUNTER is 0. The another branch from if to else was executed once since COUNTER is 1.

Method Coverage

Measuring method coverage helps us identify which methods were invoked and which were not.

We have a file grade_calculator.rb.

1students_scores = { "Sam" => [53, 91, 72],
2                    "Anna" => [91, 97, 95],
3                    "Bob" => [33, 69, 63] }
4
5def average(scores)
6  scores.reduce(&:+)/scores.size
7end
8
9def grade(average_score)
10  case average_score
11  when 90.0..100.0 then :A
12  when 80.0..90.0 then :B
13  when 70.0..80.0 then :C
14  when 60.0..70.0 then :D
15  else :F
16  end
17end
18
19def greet
20  puts "Congratulations!"
21end
22
23def warn
24  puts "Try hard next time!"
25end
26
27
28students_scores.each do |student_name, scores|
29  achieved_grade = grade(average(scores))
30
31  puts "#{student_name}, you've got '#{achieved_grade}' grade."
32
33  if achieved_grade == :A
34    greet
35  elsif achieved_grade == :F
36    warn
37  end
38
39  puts
40end

To measure method coverage of above file, let's create grade_calculator_coverage.rb by enabling methods option on Coverage#start method.

1require "coverage"
2
3Coverage.start(methods: true)
4load "grade_calculator.rb"
5p Coverage.result

Let's run it using Ruby 2.5.

1$ RBENV_VERSION=2.5.0 ruby grade_calculator_coverage.rb
2Sam, you've got 'C' grade.
3
4Anna, you've got 'A' grade.
5Congratulations!
6
7Bob, you've got 'F' grade.
8Try hard next time!
9
10{ "grade_calculator.rb" => {
11    :methods => {
12      [Object, :warn, 23, 0, 25, 3] => 1,
13      [Object, :greet, 19, 0, 21, 3] => 1,
14      [Object, :grade, 9, 0, 17, 3] => 3,
15      [Object, :average, 5, 0, 7, 3] => 3
16    }
17  }
18}

The format of method coverage result is defined as shown below.

1[ CLASS_NAME,
2  METHOD_NAME,
3  START_LINE_NUMBER,
4  START_COLUMN_NUMBER,
5  END_LINE_NUMBER,
6  END_COLUMN_NUMBER ]

Therefore, [Object, :grade, 9, 0, 17, 3] => 3 reads that the Object#grade method which starts from line 9 & column 0 to line 17 & column 3 was invoked 3 times.

Conclusion

We can measure all coverages at once also.

1Coverage.start(lines: true, branches: true, methods: true)

What's the use of these different types of coverages anyway?

Well, one use case is to integrate this in a test suite and to determine which lines, branches and methods are executed and which ones are not executed by the test. Further, we can sum up these and evaluate total coverage of a test suite.

Author of this feature, Yusuke Endoh (mame) has released coverage-helpers gem which allows further advanced manipulation and processing of coverage results obtained using Coverage#result.