Intro

This article is about writing tools in Python to easily deploy your web projects.
A well know deployment library in Python is Fabric. While it’s not really mature (version 0.9.1), it is easy to understand and work with it. The best place to start is the Fabric tutorial.

The Challenge

You develop and maintain many projects. You backup and restore databases several times a day during the development phase or you need to upload or download files from the hosting server often. How would you improve these processes instead using Ftp, mysql/mysqldump or phpmyadmin ?

Quick Fabric Introduction

Fabric gives you instruments to run commands on different hosts and copy/download files to/from remote hosts. In order to do that, you have to prepare a file of available commands named fabfile.py which will be placed in the project’s directory. Having some methods in that file allows you to run the commands like that:

fab hello

where hello is one function defined in fabfile.py
Fabric installation process.
sudo easy_install fabric
OR
sudo pip install fabric

The Solution

Having a lot of projects to maintain and deploy makes difficult to write one fabfile.py for each of them.
Also using remote hosts or writing database queries require you to put sensitive information in the fabfile.py (or pass them as parameters but that’s a different talk)
My thought is to have one location available where I place a fabfile.py and configuration files of all the sensitive information required for deployments, i.e. remote host addresses and maybe passwords, database connection information, etc. If you care more about security than speed of development, don’t store sensitive information there.
I have created a directory /work/fabric with the following structure:
├── fabfile.py
├── projects
│   ├── __init__.py
│   ├── ajaxdaddy.py
│ ├── default.py

fabfile.py contains the deployment code based on Fabric
local.py store configuration details of the localhost database
projects/* store configuration details of specific projects
The project name is the same with the file name used for the configuration.

Instead of placing a fabfile.py inside the project’s directory, we will instruct Fabric where to find the fabfile.
Create a file ~/.fabricrc and write the following line:
fabfile = /path/to/your/fabfile.py
I have mine placed in /work/fabric/fabfile.py

There are more details on fabfile discovery here.

Now we can run “fab” in any location. Here are some examples:
fab c:localhost c:localdb e:db_name=test q:clean
fab e:db_name=test_elgg e:db_file=elgg.dump.sql q:export

Commands Supported

The fabfile.py available for download in the end of this article gives you the power to run several commands useful in deploying projects or databases. Many commands require some parameters to exist, for example a database “clean” command which deletes all the tables from one database would require some connection details i.e. host,user,pass,db_name

Several commands may use the same parameters. To make these parameters available they are loaded from configuration files or passed to the “fab” command. All parameters are stored inside fabfile.py in a dictionary called “env” (environment). The environment is a big dictionary like variable (associative array for the Php users) which may store connection parameters for remote hosts, database connection properties or other information.

Most of the time you will need several params available for one command, e.g. a database command cannot function with only host name and no user provided. To make things easy, configuration files can be written where you define groups of parameters. Such a configuration file is /work/fabric/projects/default.py which is loaded by default. Configurations are simple Python dictionaries defined in Python files under the “projects” directory. It’s up to you to choose the names and the purpose of those settings, i.e., some might refer to database settings and other to remote hosting details. If you want to enable a set of parameters from a config file, first you need to load the file then enable the params.

Let’s take for example:
fab c:localhost c:localdb e:db_name=test q:clean

q:clean is the command and it means: delete all tables from the database define before
e:db_name defines the database
c:localdb loads the localhost database configuration, i.e. host, user, password, etc
c:localhost loads the localhost host details, i.e. host name, port, etc

Example of /work/fabric/projects/default.py file:
config = {
‘localdb': { # configuration name
‘db_user': ‘root’,
‘db_pass': ‘betterprogramming rules’,
‘db_host': ‘localhost’,
‘db_name': ‘test’, # default db_name if none is givven
‘db_file': ‘dump.sql’, # default dump file
},
}

“l” loads a configuration file located under “projects” directory (see the tree structure above)

fab l:ajaxdaddy

# not needed because default is loaded
fab l:default

“c” enables a configuration

# “localhost” is hardcoded into the fabfile.py because it’s mandatory
fab c:localhost

# enables a database configuration
fab c:localdb
fab l:default c:localdb

It’s possible to enable one configuration for a command then enable a different config for the next command.

# the second configuration “secondlocaldb” would overwrite the first one, “localdb”
fab l:default c:localdb c:secondlocalddb

After you enable custom configuration, it is possible to change specific parameters by using the “e” command.

# after i load and enable ajaxdaddy’s parameters, i.e. db_user, db_pass, db_host, db_name i can change the previously stored db_name value by using “e”
fab l:ajaxdaddy c:ajaxdaddydb e:db_name=db2_ajaxdaddy e:db_user=ajaxuser

In conclusion, there are 3 ways to set the parameters which will be used by your commands:
– by writing configuration files and place them under “projects” directory, load the config file (e.g. “l:ajaxdaddy”) and enable a specific group (e.g. “c:ajaxdaddydb”)
– by passing single parameters to the “fab” command (e.g. “e:db_user=gigi”)
– by passing parameters to the command itself (you will see how it works later)

Datase Commands

Database commands starts with “q” and are followed by an action.
The available actions are:
clean: delete all the tables inside the “db_name” database
export: export the database “db_name” to the file “db_file” (db_file may be provided or created programmatically by the fabfile.py from db_name)
import: import “db_file” into “db_name”
show: list all the databases where the “db_user” has access
create: create a new database called “db_name”
grant: grant all on “db_name_new” to “db_user_new” identified by “db_pass_new”

You may define your own actions in the configuration files. You just have to create a class named “DbAction”+actionname , e.g. class DbActionCreate(DbAction) or class DbActionClean(DbAction)
When loading a configuration file, your action class will also load.

# enables localhost (it’s enabled by default anyway), enables localdb so that “q” knows where to operate, sets the environment variable “db_name” to “test” then execute command
fab c:localhost c:localdb e:db_name=test q:clean
same with
fab c:localdb e:db_name=test q:clean

# selects the “test” db and export that db into “dump.sql” file
fab c:localhost c:localdb e:db_name=test e:db_file=dump.sql q:export

# create a new database “test_2″ and grant access to the user defined in “localdbnew”
fab c:localhost c:localdb c:localdbnew e:db_name=test_2 q:create q:grant

# delete all tables from “test” db and import the “test_elgg.sql” file
fab c:localdb e:db_name=test e:db_file=test_elgg.sql q:clean q:import

The list of all parameters used by the database commands:
db_user
db_pass
db_host
db_name
db_file
db_user_new
db_pass_new
db_host_new
db_name_new

File commands

# zip the temp directory, temp must be located under the current directory, it will create “temp.zip”
fab zp:temp

# zip the “temp” directory and create the archive named “mytemp.zip”
fab e:file=temp,dest=mytemp.zip zp

# zip “temp” directory’s content, i.e. do not include the “temp” parent directory in the archive
fab zpi:temp

# unzip “filename.zip” into the “dest” directory
fab uzp:filename.zip,dest
# same with
fab e:file=filename uzp:,dest
# same with
fab e:file=filename,dest=dest uzp

# load the “temp” config file and enable dreamhost parameters (“c:dreamhost”) then upload fabric.localhost.zip into the “stuff” directory located on the remote host inside the base dir. The base directory is defined in “base_dir” parameter, if base_dir=/var/www then the uploaded file will be located in /var/www/stuff/fabric.localhost.zip
fab l:temp c:dreamhost up:fabric.localhost.zip,stuff

# download /var/www/remotefile.zip to the current directory if base_dir=/var/www
fab l:temp c:dreamhost dl:remotefile.zip

List of file commands:
zp: zip a “file” (file or directory) to “dest”
zpi: zip a directory’s contents, exclude the directory itself from the archive
uzp: unzip a “file” to “dest”
up: upload a “source” (local host) to “dest” (remote host)
dl: download a “source” (remote host) to “dest” (local host)
cl: delete a “file”

Complex Commands

# exports “test_elgg” database into “dump.sql” and upload it to “dreamhost”. “up:e.db_file” means that the value of “db_file” located in the environment should be uploaded.
fab c:localhost c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export l:dreamhost c:dreamhost up:e.db_file
# same with
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export l:dreamhost c:dreamhost up:dump.sql
# similar but different dump name “test_elgg.sql” created by default from “db_name” + “.sql”
fab c:localdb e:db_name=test_elgg q:export l:dreamhost c:dreamhost up:test_elgg.sql

# same as above, but upload a zip file which will be called by default “dump.sql.localhost.zip”
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export zp:dump.sql l:dreamhost c:dreamhost up

# same but unzip on destination
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export zp:dump.sql l:dreamhost c:dreamhost up uzp

# same but delete “cl” the dump.sql from localhost and delete “cl” dump.sql from the remote host
fab c:localdb e:db_name=test_elgg e:db_file=dump.sql q:export zp:dump.sql cl:dump.sql l:dreamhost c:dreamhost up uzp cl:dump.sql

Download fabfile.py and a few examples