#309 A Shell-Scripting Story pro

As Rails developers we generally turn to Ruby for our programming problems, including small administrative tasks. Sometimes, though, it’s more efficient to write shell scripts in the shell’s native language. In this episode we’ll show you the story of one such time.

Several years ago Ryan Bates wrote a Ruby script that adds autocompletion to Rake tasks. This script can be found on Github and is quite long for what it does, over fifty lines. The main part of it is a generate_tasks method. This method calls rake --tasks, which lists all of the rake tasks available, and then parses them to get the tasks’ names. It then caches the output to use by the autocompletion.

def generate_tasks
  tasks = `rake --tasks`.split("\n")[1..-1].collect {|line| line.split[1]}
  File.open(cache_file, 'w') { |f| f.write tasks.join("\n") }
  tasks
end

When Ryan decided to rewrite this as a shell script he was amazed at how concise the resulting script was. In this episode we’ll show you the steps he took to do this so that you get a feel for shell scripting.

Z Shell comes with Rake autocompletion but it can be terribly slow. If we type rake then press TAB there’s a noticeable delay before the results appear. This is because the results aren’t cached at all and so zsh has to ask rake for a list of the tasks every time. In this episode we’ll build on what we wrote last time where we set up oh-my-zsh and built a custom plugin. We’ll add our rake autocompletion plugin in the same folder in a new file called _rake, following the convention of starting autocompletion-related plugins with an underscore followed by the name of the command we’re adding autocompletion to.

We need to tell zsh that this is an completion command defined for the rake command and we do this by adding a comment as the first line. We can add some completion options manually by using compadd and listing out the options.

#compdef rake
compadd spec secret foo bar

If we open a new terminal window now and type rake followed by a space then hit TAB we’ll see the autocompletion options listed.

%rake
bar     foo     secret  spec 

We need this to be a list of real tasks instead of some hard-coded examples so we have to call rake --tasks at some point to fetch them all. This means that our command will be slow when we first run it but once we’ve cached the results it should run far more quickly. The output from rake --tasks looks like this; we’ll need to parse it to get each task’s name.

% rake --tasks
rake about              # List versions of all Rails frameworks and the environment
rake assets:clean       # Remove compiled assets
rake assets:precompile  # Compile all the assets named in config.assets.precompile
rake db:create          # Create the database from config/database.yml for the current Rails.env (use db:create:all to create all dbs in the config)
rake db:drop            # Drops the database for the current Rails.env (use db:drop:all to drop all databases)
rake db:fixtures:load   # Load fixtures into the current environment's database.
# rest of output omitted.

There’s a command called cut that we can use to parse this output. It does exactly what we want: cutting out a section of each line of a file. We can pipe the output of rake --tasks to cut and we need to pass it a couple of parameters. The default delimiter is a tab and we can use -d change this to a space. We also need to specify which portion of each line should be cut out (we want the second part). If we run this command now we get the results we want:

% rake --tasks | cut -d ' ' -f 2
about
assets:clean
assets:precompile
db:create
db:drop
db:fixtures:load
db:migrate

We can now replace the placeholders on compadd with the output from this command. For the output to be captured we need wrap the command in $().

#compdef rake
compadd $(rake --tasks | cut -d ' ' -f 2)

Each time we change this script we’ll need to open a new terminal window for the changes to take effect. Alternatively we can run this command:

% unfunction _rake; . ~/.zshrc

If we type rake now and hit TAB we’ll see a list of all of the Rake tasks we can run. We can filter the results by entering part of a task name before pressing TAB.

% rake db:
db:abort_if_pending_migrations  db:rollback                   
db:charset                      db:schema:dump                
db:collation                    db:schema:load                
db:create                       db:seed                       
db:create:all                   db:sessions:clear             
db:drop                         db:sessions:create            
db:drop:all                     db:setup                      
db:fixtures:identify            db:structure:dump             
db:fixtures:load                db:test:clone                 
db:migrate                      db:test:clone_structure       
db:migrate:down                 db:test:load                  
db:migrate:redo                 db:test:prepare               
db:migrate:reset                db:test:purge                 
db:migrate:up                   db:version  

Caching the Output From Rake

To make this command respond more quickly we can cache the results into a file.

#compdef rake
if [[ ! -f .rake_tasks~ ]]; then
  rake --tasks | cut -d ' ' -f 2 > .rake_tasks~
fi
compadd $(cat .rake_tasks~)

We cache the results to a file called .rake_tasks~ but we only want to do this if the file doesn’t exists so we check for this first. We then tell compadd to get its results from this file instead of from the rake command. We can try this out but we’ll need to reload the command first.

% unfunction _rake; . ~/.zshrc

Now when we type rake and hit TAB the command will take a while to show the autocompletion options the first time but on subsequent times it will run a lot more quickly.

Expiring The Cache

It would be nice if the cache automatically expired when update a related Rake file. We can find out what the most recently modified file in a directory is by running ls -t. If we pass various Rake-related files to this command we can then check to see if our cache file is the most recently modified of these. If it isn’t then we have to rebuild the cache. To get the first result from ls we can use head.

ls -t .rake_tasks\~ Rakefile **/*.rake | head -n 1

We can use this in our shell script to get the most recently-modified file.

#compdef rake
local recent=$(ls -t .rake_tasks~ Rakefile **/*.rake | head -n 1)
if [[ $recent != .rake_tasks~ ]]; then
  rake --tasks | cut -d ' ' -f 2 > .rake_tasks~
fi
compadd $(cat .rake_tasks~)

In our script we now get the most recently modified file and assign the result to a variable. Inside a completion script like this is executed at a global scope so it’s a good idea to set it as a local variable. We then check to see that the most recently-modified file is the .rake_tasks~ cache file and if it isn’t then we run rake --tasks again to update the cache.

Once we’ve reloaded this function again we can try this out. The command should be fast the first time we run it (as we already have a .rake_tasks~ file) but when we touch the Rakefile and run the command again there will be a delay as the cache is rebuilt.

Final Modifications

Our command works well now but if we run it in a directory with no Rakefile we’ll get an error. We’ll make a final change to our script so that it checks for a Rakefile in the current directory.

#compdef rake
if [[ -f Rakefile]]; then
  local recent=$(ls -t .rake_tasks~ Rakefile **/*.rake | head -n 1)
  if [[ $recent != .rake_tasks~ ]]; then
    rake --tasks | cut -d ' ' -f 2 > .rake_tasks~
  fi
  compadd $(cat .rake_tasks~)
fi

That’s pretty much it for our script. Its behaviour is similar to the Ruby script we started with but we’ve done it in only eight lines of code which gives you an idea of the power of shell scripting. That said the original Ruby script adds a little more functionality to make the completion work in Bash so it’s not truly a fair comparison. Both approaches have their pros and cons but you should know your tools and be able to pick the most appropriate one to solve a problem.

Some Useful UNIX Commands

Speaking of knowing your tools we’ll finish off by showing you a few useful command-line utilities that work well in shell scripts. The first one is find. This isn’t as useful in zsh as it is in Bash because of its advanced wildcard features but it’s still worth knowing. The first argument it takes is a path so if we just pass the current path we’ll get a list of the files in the current directory and its subdirectories.

% find .
.
./blog
./blog/.rake_tasks~
./blog/app
./blog/app/controllers
./blog/app/controllers/application_controller.rb
./blog/app/controllers/articles_controller.rb
./blog/app/helpers
./blog/app/helpers/application_helper.rb
./blog/app/helpers/articles_helper.rb
./blog/app/models
# Rest of files omitted.

We can filter these results by passing in some options such as name which will let us specify a filename filter.

% find . -name '*.rb'
./blog/app/controllers/application_controller.rb
./blog/app/controllers/articles_controller.rb
./blog/app/helpers/application_helper.rb
./blog/app/helpers/articles_helper.rb
./blog/app/models/article.rb
# Rest of files omitted.

Note that it’s important to quote the pattern so that the shell doesn’t try to handle it itself. If we want to find files that have been modified in a certain time period we can use mtime. This command will find all of the Ruby files that have been modified in the last three days.

% find . -name '*.rb' -mtime -3
````

<p>We can pipe the output from this command into another useful command, <code>grep</code>.</p>

``` terminal
% find . -name '*.rb' | grep production
./blog/config/environments/production.rb

The grep command will look through whatever is passed to it, in this case a list of file names and paths, and return any lines that match the pattern that is passed to it. If we want to search through the contents of the files we can use xargs.

% find . -name '*.rb' | xargs grep production
./blog/config/environments/production.rb:# The production environment is meant for finished, "live" apps.
./blog/config/environments/production.rb:# Use a different cache store in production

The xargs command works by taking its input, in this case the list of files and appending them as arguments to the command that we pass to it. In this case grep will search each of the files that passed to it for the word production.

There’s an alternative to grep called ack. If you’re running OS X this won’t be installed by default but if you have Homebrew on your system you can install it by running brew install ack. We use ack in a similar way to grep so we can run our previous command again and replace grep with ack.

%find . -name '*.rb' | xargs ack production
blog/config/environments/production.rb
3:# The production environment is meant for finished, "live" apps.
18:# Use a different cache store in production

Another useful utility is sed. This is a stream editor and it will modify any input passed to it. If for some reason we wanted to pretend that our Rails project was written in Python we could run this:

%find . -name '*.rb' | sed 's/rb/py/'
./blog/app/controllers/application_controller.py
./blog/app/controllers/articles_controller.py
./blog/app/helpers/application_helper.py
./blog/app/helpers/articles_helper.py
./blog/app/models/article.py
./blog/app/models/comment.py
./blog/app/models/feed_entry.py

The arguments we pass to sed are both regular expressions so we can so some fairly complicated matching if we want to. As before if we want to change the contents of the files rather than the name we can use xargs.

The files themselves aren’t changed when we do this, only the output from the command that shows the files’ contents. We can do this, though, by using sed’s -i option.

%find . -name '*.rb' | xargs sed -i 's/rb/py/'

Other commands that are worth looking at include sort, tr, comm, cut, paste, diff and patch. To find out how these commands work we can use the most important command of all, man, which gives us documentation about each of these commands.

For more information on learning the shell there’s a really useful book called “From Bash to Z Shell”. This contains information on both Bash and Z Shell and is a great resource if you’re moving from one to the other.