Website deployment with a shell script and rsync
There are many ways to deploy a website. Because I work on my own, I usually use a shell script and rsync.
I build development websites locally on my Mac and I use version control for every project — usually Subversion. I use Cornerstone which is a great Subversion app for Mac.
Workflow
- Develop locally on Mac.
- Use fabric to create tarball of the complete site and upload it to a staging directory on the web server.
- Create a “live” working copy locally.
- Deploy “live” working copy to the web server with a bash script and rsync.
- Copy to production: ssh into web server and copy from staging to production directory.
- Remove staging site. Update the deploy script for the production site.
Uploading to staging site
A couple of years ao I experimented with using Fabric to automate deployment. I now use fabric to automate step 2 (above). This is the fabric script I use:
from __future__ import with_statement
from fabric.api import *
env.hosts = ['xxxxx@xxx.xxx.xxx.xxx']
def stage():
local('gnutar czf /tmp/stagedomain.co.nz.tar.gz ./ --exclude-from=tar_exclude.txt')
with cd('/home/www/stagedomain.co.nz'):
put('/tmp/stagedomain.co.nz.tar.gz','./')
run('tar xzfp stagedomain.co.nz.tar.gz')
run('rm stagedomain.co.nz.tar.gz')
Replace xxxxx@xxx.xxx.xxx.xxx with your web server login and IP address.
To run the script in terminal, cd to the directory and type fab stage
I use a separate file tar_exclude.txt, which is in the same directory:
.sass-cache
sass
config.rb
.svn
.git
.gitignore
.DS_Store
*.sublime-project
*.sublime-workspace
fabfile.py
fabfile.pyc
tar_exclude.txt
deploy
processjs
rsync_exclude.txt
deploy_log.txt
Deploy to the web server
I store my code repositories on my Mac and use a shell script and rsync to update the production web server. I generally only need a staging site during initial development. I use a local site for development and use a separate local working copy to update the web server via rsync.
This script is named “deploy” and is in the LOCAL_DIR directory as configured below. View a Gist.
#!/bin/bash
# create a file deploy_log.txt in LOCAL_DIR
# create a file rsync_exclude.txt in LOCAL_DIR
# configure the following
LOCAL_DIR="/Users/mike/Sites/domain/"
LIVE_DIR="/Users/mike/Sites/live.domain"
LIVE_CSS="/Users/mike/Sites/live.domain/public/css"
LIVE_JS="/Users/mike/Sites/live.domain/public/js/"
LIVE_JAVASCRIPT="/Users/mike/Sites/live.domain/javascript/*.js"
REMOTE_DIR="/path/new.domain"
REMOTE_USER="username"
REMOTE_IP="xxx.xxx.xxx.xxx"
((
echo -e "\n"
echo -e "-----------------------------------\n"
echo "$(date)"
echo -e "\n"
echo 'SVN update of live working copy'
/opt/local/bin/svn update $LIVE_DIR
echo -e "\n"
echo 'compile CSS'
cd ${LOCAL_DIR}
bundle exec compass compile -s compressed --css-dir ${LIVE_CSS}
echo -e "\n"
echo 'uglify JavaScript and output to js directory in live working copy'
for f in $LIVE_JAVASCRIPT
do
filename=$(basename "$f")
filename="${filename%.js}"
/usr/local/bin/uglifyjs $f -c -o ${LIVE_JS}${filename}.min.js
echo -e "created ${LIVE_JS}${filename}.min.js at" $(date)
done
echo -e "\n"
echo 'rsync with production dir on web server'
rsync -acvzh --exclude-from ${LOCAL_DIR}'rsync_exclude.txt' ${LIVE_DIR}/ ${REMOTE_USER}@${REMOTE_IP}:${REMOTE_DIR}
) 2>&1) | tee -a ${LOCAL_DIR}'deploy_log.txt'
This process means that files and directories which are not version controlled are not overwritten, except for the public/css and public/js directories. I also use a separate file rsync_exclude.txt, which is in the same directory as the deploy script to exclude files which are version controlled, but shouldn’t be deployed. Here is an example, which I use when developing with CakePHP.
app/Config/core.php
app/Config/database.php
app/Config/bootstrap.php
public/.htaccess
public/index.php
public/test.php
public/img
public/uploads
.svn*
.git
.gitignore
.DS_Store
*.sublime-project
*.sublime-workspace
fabfile.py
fabfile.pyc
tar_exclude.txt
deploy
rsync_exclude.txt
deploy_log.txt
config.rb
.sass*
sass*
javascript*
processjs
Gemfile
Gemfile.lock
When committing in Cornerstone, I sometimes configure the deploy script to run as a post-commit script and set it to run once for the working copy after committing. This means that I can make changes to the dev site, and when I’m happy with the results, I can commit the changes and automatically them deploy to the web server.
Tips and tricks
- Make shell scripts executable with
chmod +x
- You may need to debug your bash script.
- Learn more about rsync
- If Cornerstone doesn’t resolve the paths to to svn and uglifyjs, there are three alternative solutions (I chose number 1).
- Use the full paths for svn and uglifyjs. Type
which svn
andwhich uglifyjs
in terminal to get these paths. If you get the errorenv: node: No such file or directory
add a symlink for node:sudo ln -s /usr/local/bin/node /usr/bin/node
. - Open Cornerstone by typing
open /Applications/Cornerstone.app/
in terminal. - redefine the search path for applications.
- Use the full paths for svn and uglifyjs. Type