BigBinary Blog
We write about Ruby on Rails, React.js, React Native, remote work, open source, engineering and design.
Ruby comes with Coverage, a simple standard library for test coverage measurement for a long time.
Before Ruby 2.5,
we could measure just the line coverage using Coverage
.
Line coverage tells us whether a line is executed or not. If executed, then how many times that line was executed.
We have a file called score.rb
.
score = 33
if score >= 40
p :PASSED
else
p :FAILED
end
Now create another file score_coverage.rb
.
require "coverage"
Coverage.start
load "score.rb"
p Coverage.result
We used Coverage#start
method
to measure the coverage of score.rb
file.
Coverage#result
returns the coverage result.
Let's run it with Ruby 2.4.
$ RBENV_VERSION=2.4.0 ruby score_coverage.rb
:FAILED
{ "score.rb"=> [1, nil, 1, 0, nil, 1, nil] }
Let's look at the output.
Each value in the array [1, nil, 1, 0, nil, 1, nil]
denotes the count of line executions by the interpreter
for each line in score.rb
file.
This array is also called the "line coverage"
of score.rb
file.
A nil
value in line coverage array means
coverage is disabled for that particular line number
or it is not a relevant line.
Lines like else
, end
and blank lines
have line coverage disabled.
Here's how we can read above line coverage result.
else
clause.end
keyword.There was a pull request opened in 2014 to add method coverage and decision coverage metrics in Ruby. It was rejected by Yusuke Endoh as he saw some issues with it and mentioned that he was also working on a similar implementation.
In Ruby 2.5,
Yusuke Endoh added branch coverage and method coverage feature
to the Coverage
library.
Let's see what's changed in Coverage
library in Ruby 2.5.
If we execute above example using Ruby 2.5, we will see no change in the result.
$ RBENV_VERSION=2.5.0 ruby score_coverage.rb
:FAILED
{ "score.rb" => [1, nil, 1, 0, nil, 1, nil] }
This behavior is maintained
to ensure that the Coverage#start
API
stays 100% backward compatible.
If we explicitly enable lines
option
on Coverage#start
method
in the above score_coverage.rb
file,
the coverage result will be different now.
require "coverage"
Coverage.start(lines: true)
load "score.rb"
p Coverage.result
$ RBENV_VERSION=2.5.0 ruby score_coverage.rb
:FAILED
{ "score.rb" => {
:lines => [1, nil, 1, 0, nil, 1, nil]
}
}
We can see that the coverage result is now a hash
which reads that the score.rb
file
has lines
coverage as [1, nil, 1, 0, nil, 1, nil]
.
Branch coverage helps us identify which branches are executed and which ones are not executed.
Let's see how to get branch coverage.
We will update the score_coverage.rb
by enabling branches
option.
require "coverage"
Coverage.start(branches: true)
load "score.rb"
p Coverage.result
$ RBENV_VERSION=2.5.0 ruby score_coverage.rb
:FAILED
{ "score.rb" =>
{ :branches => {
[:if, 0, 3, 0, 7, 3] => {
[:then, 1, 4, 2, 4, 15] => 0,
[:else, 2, 6, 2, 6, 15] => 1
}
}
}
}
Here is how to read the data in array.
[
BRANCH_TYPE,
UNIQUE_ID,
START_LINE_NUMBER,
START_COLUMN_NUMBER,
END_LINE_NUMBER,
END_COLUMN_NUMBER
]
Please note that column numbers start from 0 and line numbers start from 1.
Let's try to read above printed branch coverage result.
[:if, 0, 3, 0, 7, 3]
reads that
if
statement
starts at line 3 & column 0
and
ends at line 7 & column 3.
[:then, 1, 4, 2, 4, 15]
reads that
then
clause
starts at line 4 & column 2
and
ends at line 4 & column 15.
Similarly,
[:else, 2, 6, 2, 6, 15]
reads that
else
clause
starts at line 6 & column 2
and
ends at line 6 & column 15.
Most importantly
as per the branch coverage format,
we can see that
the branch from if
to then
was never executed
since COUNTER
is 0
.
The another branch from if
to else
was executed once
since COUNTER
is 1
.
Measuring method coverage helps us identify which methods were invoked and which were not.
We have a file grade_calculator.rb
.
students_scores = { "Sam" => [53, 91, 72],
"Anna" => [91, 97, 95],
"Bob" => [33, 69, 63] }
def average(scores)
scores.reduce(&:+)/scores.size
end
def grade(average_score)
case average_score
when 90.0..100.0 then :A
when 80.0..90.0 then :B
when 70.0..80.0 then :C
when 60.0..70.0 then :D
else :F
end
end
def greet
puts "Congratulations!"
end
def warn
puts "Try hard next time!"
end
students_scores.each do |student_name, scores|
achieved_grade = grade(average(scores))
puts "#{student_name}, you've got '#{achieved_grade}' grade."
if achieved_grade == :A
greet
elsif achieved_grade == :F
warn
end
puts
end
To measure method coverage of above file,
let's create grade_calculator_coverage.rb
by enabling methods
option
on Coverage#start
method.
require "coverage"
Coverage.start(methods: true)
load "grade_calculator.rb"
p Coverage.result
Let's run it using Ruby 2.5.
$ RBENV_VERSION=2.5.0 ruby grade_calculator_coverage.rb
Sam, you've got 'C' grade.
Anna, you've got 'A' grade.
Congratulations!
Bob, you've got 'F' grade.
Try hard next time!
{ "grade_calculator.rb" => {
:methods => {
[Object, :warn, 23, 0, 25, 3] => 1,
[Object, :greet, 19, 0, 21, 3] => 1,
[Object, :grade, 9, 0, 17, 3] => 3,
[Object, :average, 5, 0, 7, 3] => 3
}
}
}
The format of method coverage result is defined as shown below.
[ CLASS_NAME,
METHOD_NAME,
START_LINE_NUMBER,
START_COLUMN_NUMBER,
END_LINE_NUMBER,
END_COLUMN_NUMBER ]
Therefore,
[Object, :grade, 9, 0, 17, 3] => 3
reads that the Object#grade
method
which starts from line 9 & column 0
to
line 17 & column 3
was invoked 3 times.
We can measure all coverages at once also.
Coverage.start(lines: true, branches: true, methods: true)
What's the use of these different types of coverages anyway?
Well, one use case is to integrate this in a test suite and to determine which lines, branches and methods are executed and which ones are not executed by the test. Further, we can sum up these and evaluate total coverage of a test suite.
Author of this feature, Yusuke Endoh (mame)
has released coverage-helpers gem
which allows further advanced manipulation and processing of
coverage results obtained using Coverage#result
.