Executing shell commands in ruby

Ruby allows many different ways to execute a command or a sub-process. In this article we are going to see some of them.

backtick

1. Returns standard output

backtick returns the standard output(stdout) of the operation.

output = `pwd`
puts "output is #{output}"

$ ruby main.rb
output is /Users/neerajsingh/code/misc

backtick does not capture STDERR . If you want to learn about STDERR then checkout this excellent article .

You can redirect STDERR to STDOUT if you want to capture STDERR using backtick.

output = `grep hosts /private/etc/* 2>&1`

2. Exception is passed on to the main program

Backtick operation forks the master process and the operation is executed in a new process. If there is an exception in the sub-process then that exception is given to the main process and the main process might terminate if exception is not handled.

In the following case I am executing xxxxx which is not a valid executable name.

output = `xxxxxxx`
puts "output is #{output}"

Result of above code is given below. Notice that puts was never executed because the backtick operation raised exception.

$ ruby main.rb
main.rb:1:in ``': No such file or directory - xxxxxxx (Errno::ENOENT)
	from main.rb:1:in `<main>'

3. Blocking operation

Backtick is a blocking operation. The main application waits until the result of backtick operation completes.

4. Checking the status of the operation

To check the status of the backtick operation you can execute $?.success?

output = `ls`
puts "output is #{output}"
puts $?.success?

Notice that the last line of the result contains true because the backtick operation was a success.

$ ruby main.rb
output is lab.rb
main.rb
true

5. String interpolation is allowed within the ticks

cmd = 'ls'
`#{cmd}`

6. Different delimiter and string interpolation

%x does the same thing as backtick. It allows you to have different delimiter.

output = %x[ ls ]
output = %x{ ls }

backtick runs the command in subshell. So shell features like string interpolation and wild card can be used. Here is an example.

$ irb
> dir = '/etc'
> %x<ls -al #{dir}>
=> "lrwxr-xr-x@ 1 root  wheel  11 Jan  5 21:10 /etc -> private/etc"

If you are building a script which you mean to run on your laptops and not on server then most likely if there is an exception then you want the script to abort. For such cases backtick is the best choice.

For example let's say that I want to write a script to make my repo up-to-date automatically. The command would be something like this.

cd directory_name && git checkout main && git pull origin main

If there is any error while executing this command then you want to have the full access to the exception so that you could debug. In such cases the best way to execute this command is as shown below.

cmd = "cd #{directory_name} && git checkout main && git pull origin main"
%x[ cmd ]

system

The system command runs in a subshell.

Just like backtick, system is a blocking operation.

Since system command runs in a subshell it eats up all the exceptions. So the main operation never needs to worry about capturing an exception raised from the child process.

output = system('xxxxxxx')
puts "output is #{output}"

Result of the above operation is given below. Notice that even when exception is raised the main program completes and the output is printed. The value of output is nil because the child process raised an exception.

$ ruby main.rb
output is

system returns true if the command was successfully performed ( exit status zero ) . It returns false for non zero exit status. It returns nil if command execution fails.

system("command that does not exist")  #=> nil
system("ls")                           #=> true
system("ls | grep foo")                #=> false

system sets the global variable $? to the exit status of the process. Remember that a value of zero means the operation was a success.

The biggest issue with system command is that it's not possible to capture the output of the operation.

exec

Kernel#exec replaces the current process by running the external command.

Let's see an example. Here I am in irb and I am going to execute exec('ls').

$ irb
e1.9.3-p194 :001 > exec('ls')
lab.rb  main.rb

nsingh ~/neerajsingh
$

I see the result but since the irb process was replaced by the exec process I am no longer in irb .

Behind the scene both system and backtick operations use fork to fork the current process and then they execute the given operation using exec .

Since exec replaces the current process it does not return anything. It prints the output on the screen. There is no way to know if the operation was a "success" or a "failure" and hence it's not recommended to use exec.

sh

sh actually calls system under the hood. However it is worth a mention here. This method is added by FileUtils in rake. It allows an easy way to check the exit status of the command.

require 'rake'
sh %w(xxxxx) do |ok, res|
   if !ok
     abort 'the operation failed'
   end
end

popen3

If you are going to capture stdout and stderr then you should use popen3 since this method allows you to interact with stdin, stdout and stderr .

I want to execute git push heroku master programmatically and I want to capture the output. Here is my code.

require 'open3'
cmd = 'git push heroku master'
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
  puts "stdout is:" + stdout.read
  puts "stderr is:" + stderr.read
end

And here is the output. It has been truncated since rest of output is not relevant to this discussion.

stdout is:
stderr is:
-----> Heroku receiving push
-----> Ruby/Rails app detected
-----> Installing dependencies using Bundler version 1.2.1

The important thing to note here is that when I execute the program ruby lab.rb I do not see any output on my terminal for first 10 seconds. Then I see the whole output as one single dump.

The other thing to note is that heroku is writing all this output to stderr and not to stdout .

Above solution works but it has one major drawback. The push to heroku might take 10 to 20 seconds and for this period we do not get any feedback on the terminal. In reality when we execute git push heroku master we start seeing result on our terminal one by one as heroku is processing things.

So we should capture the output from heroku as it is being streamed rather than dumping the whole output as one single chunk of string at the end of processing.

Here is the modified code.

require 'open3'
cmd = 'git push heroku master'
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
  while line = stderr.gets
    puts line
  end
end

Now when I execute above command using ruby lab.rb I get the output on my terminal incrementally as if I had typed git push heroku master .

Here is another example of capturing streaming output.

require 'open3'
cmd = 'ping www.google.com'
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
  while line = stdout.gets
    puts line
  end
end

In the above case you will get the output of ping on your terminal as if you had typed ping www.google.com on your terminal .

Now let's see how to check if command succeeded or not.

require 'open3'
cmd = 'ping www.google.com'
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
  exit_status = wait_thr.value
  unless exit_status.success?
    abort "FAILED !!! #{cmd}"
  end
end

popen2e

popen2e is similar to popen3 but merges the standard output and standard error .

require 'open3'
cmd = 'ping www.google.com'
Open3.popen2e(cmd) do |stdin, stdout_err, wait_thr|
  while line = stdout_err.gets
    puts line
  end

  exit_status = wait_thr.value
  unless exit_status.success?
    abort "FAILED !!! #{cmd}"
  end
end

In all other areas this method works similar to popen3 .

Process.spawn

Kernel.spawn executes the given command in a subshell. It returns immediately with the process id.

irb(main)> pid = Process.spawn("ls -al")
=> 81001