Website deployment with a shell script and rsync

by Michael Birch

There are many ways to deploy a website. Because I work on my own, I usually use a shell script and rsync.

I build development websites locally on my Mac and I use version control for every project — usually Subversion. I use Cornerstone which is a great Subversion app for Mac.

Workflow

  1. Develop locally on Mac.
  2. Use fabric to create tarball of the complete site and upload it to a staging directory on the web server.
  3. Create a “live” working copy locally.
  4. Deploy “live” working copy to the web server with a bash script and rsync.
  5. Copy to production: ssh into web server and copy from staging to production directory.
  6. Remove staging site. Update the deploy script for the production site.

Uploading to staging site

A couple of years ao I experimented with using Fabric to automate deployment. I now use fabric to automate step 2 (above). This is the fabric script I use:

from __future__ import with_statement
from fabric.api import *
env.hosts = ['xxxxx@xxx.xxx.xxx.xxx']

def stage():
local('gnutar czf  /tmp/stagedomain.co.nz.tar.gz ./ --exclude-from=tar_exclude.txt')

with cd('/home/www/stagedomain.co.nz'):
    put('/tmp/stagedomain.co.nz.tar.gz','./')
        run('tar xzfp stagedomain.co.nz.tar.gz')
        run('rm stagedomain.co.nz.tar.gz')

Replace xxxxx@xxx.xxx.xxx.xxx with your web server login and IP address.
To run the script in terminal, cd to the directory and type fab stage
I use a separate file tar_exclude.txt, which is in the same directory:

.sass-cache
sass
config.rb
.svn
.git
.gitignore
.DS_Store
*.sublime-project
*.sublime-workspace
fabfile.py
fabfile.pyc
tar_exclude.txt
deploy
processjs
rsync_exclude.txt
deploy_log.txt 

Deploy to the web server

I store my code repositories on my Mac and use a shell script and rsync to update the production web server. I generally only need a staging site during initial development. I use a local site for development and use a separate local working copy to update the web server via rsync.

This script is named “deploy” and is in the LOCAL_DIR directory as configured below. View a Gist.

#!/bin/bash
# create a file deploy_log.txt in LOCAL_DIR
# create a file rsync_exclude.txt in LOCAL_DIR
# configure the following
LOCAL_DIR="/Users/mike/Sites/domain/"
LIVE_DIR="/Users/mike/Sites/live.domain"
LIVE_CSS="/Users/mike/Sites/live.domain/public/css"
LIVE_JS="/Users/mike/Sites/live.domain/public/js/"
LIVE_JAVASCRIPT="/Users/mike/Sites/live.domain/javascript/*.js"
REMOTE_DIR="/path/new.domain"
REMOTE_USER="username"
REMOTE_IP="xxx.xxx.xxx.xxx"

((
echo -e "\n" 
echo -e "-----------------------------------\n"
echo "$(date)"
echo -e "\n"
echo 'SVN update of live working copy'
/opt/local/bin/svn update $LIVE_DIR
echo -e "\n"
echo 'compile CSS'
cd ${LOCAL_DIR}
bundle exec compass compile -s compressed --css-dir ${LIVE_CSS}
echo -e "\n"
echo 'uglify JavaScript and output to js directory in live working copy'
for f in $LIVE_JAVASCRIPT
do 
  filename=$(basename "$f")
  filename="${filename%.js}"   
  /usr/local/bin/uglifyjs $f -c -o ${LIVE_JS}${filename}.min.js 
  echo -e "created ${LIVE_JS}${filename}.min.js at" $(date)
done
echo -e "\n"
echo 'rsync with production dir on web server'
rsync -acvzh  --exclude-from ${LOCAL_DIR}'rsync_exclude.txt' ${LIVE_DIR}/ ${REMOTE_USER}@${REMOTE_IP}:${REMOTE_DIR}
) 2>&1) | tee -a ${LOCAL_DIR}'deploy_log.txt'

This process means that files and directories which are not version controlled are not overwritten, except for the public/css and public/js directories. I also use a separate file rsync_exclude.txt, which is in the same directory as the deploy script to exclude files which are version controlled, but shouldn’t be deployed. Here is an example, which I use when developing with CakePHP.

app/Config/core.php
app/Config/database.php
app/Config/bootstrap.php
public/.htaccess
public/index.php
public/test.php
public/img
public/uploads
.svn*
.git
.gitignore
.DS_Store
*.sublime-project
*.sublime-workspace
fabfile.py
fabfile.pyc
tar_exclude.txt
deploy
rsync_exclude.txt
deploy_log.txt
config.rb
.sass*
sass*
javascript*
processjs
Gemfile
Gemfile.lock

When committing in Cornerstone, I sometimes configure the deploy script to run as a post-commit script and set it to run once for the working copy after committing. This means that I can make changes to the dev site, and when I’m happy with the results, I can commit the changes and automatically them deploy to the web server.

Tips and tricks

  • Make shell scripts executable with chmod +x
  • You may need to debug your bash script.
  • Learn more about rsync
  • If Cornerstone doesn’t resolve the paths to to svn and uglifyjs, there are three alternative solutions (I chose number 1).
    1. Use the full paths for svn and uglifyjs. Type which svn and which uglifyjs in terminal to get these paths. If you get the error env: node: No such file or directory add a symlink for node: sudo ln -s /usr/local/bin/node /usr/bin/node.
    2. Open Cornerstone by typing open /Applications/Cornerstone.app/ in terminal.
    3. redefine the search path for applications.
blog comments powered by Disqus