Posted on by admin

Deploying a Pylons App to Production, Step-by-Step (Part 2 of 2)

In Part 1 of this tutorial, I described how to prepare Nginx (along with Apache) to serve a production Pylons app. In this article, I walk you through packaging and installing your application.

Packaging Your Pylons App

The official Pylons book offers a lengthy section on how to package your application. However, I feel that it there is a bit too much focus on how to prepare it for distribution. I think the bulk of beginning Pylons developers aren’t all that interested in putting their application on PyPI for the world at large to download (not yet, anyways). Therefore I’m going to distill the information into what you need to know to package you app for deployment into your own production environment.

Just as if we were going to post our project on PyPI, we’ll need to package it up into an .egg (which isn’t much more than a compressed tarball with installation metadata) so it can be installed using easy_install. The file that we’re most interested in is setup.py in your project root. Here is an example from one of my projects:

setup.py
try:
    from setuptools import setup, find_packages
except ImportError:
    from ez_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages

setup(
    name='MyAwesomeApp',
    version='0.1.0',
    description='My app which is most awesome',
    author='Your Name',
    author_email='[email protected]',
    url='http://blog.rightbrainnetworks.com',
    install_requires=[
        "Pylons>=0.9.7,=0.5.2,=0.2.4,=2.0.13,=1.2.7,=1.6.3"],
    packages=find_packages(exclude=['ez_setup']),
    include_package_data=True,
    test_suite='nose.collector',
    package_data={'myawesomeapp': ['i18n/*/LC_MESSAGES/*.mo']},
    #message_extractors={'myawesomeapp': [
    #        ('**.py', 'python', None),
    #        ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),
    #        ('public/**', 'ignore', None)]},
    zip_safe=False,
    paster_plugins=['PasteScript', 'Pylons'],
    entry_points="""
    [paste.app_factory]
    main = myawesomeapp.config.middleware:make_app

    [paste.app_install]
    main = pylons.util:PylonsInstaller
    """,
)

You’ll want to modify the obvious fields regarding your application information, plus figure out your version number. Since we’ll be pulling your source from Subversion, the version number will automatically be appended with the build number, so don’t worry too much about what to set as the version number. The Pylons book has a good overview.

The “install_requires” section contains a list of your project dependencies. These will automatically be pulled down from PyPI by setup tools when your application is installed. In this example, I have five dependencies other than the standard Pylons stuff. Your project will vary, but the dependency that you must list here is “flup”. flup is the WSGI module that we will be using with FastCGI and Nginx.

Once you’ve modified your setup.py, go ahead and check your project into your SVN repo one last time before we deploy it.

Creating production.ini

My production.ini file does not exist in the SVN repo. It’s contained separately on the production server and is copied over each time I deploy an updated version of an app. It’s very similar to the development.ini file, but has several important differences:

production.ini
#
# myapp - Pylons development environment configuration
#
# The %(here)s variable will be replaced with the parent directory of this file
#
[DEFAULT]
# Uncomment and replace with the address which should receive any error reports
#email_to = [email protected]
smtp_server = localhost
error_email_from = paste@localhost

[server:main]
#use = egg:Paste#http
use = egg:Flup#fcgi_thread
host = localhost
port = 9000

[app:main]
use = egg:myapp
full_stack = true
static_files = true

DSN = dbname='ap' user='ap_dbo' host='my_db_server'

cache_dir = %(here)s/data
beaker.session.key = myapp
beaker.session.secret = somesecret

# If you'd like to fine-tune the individual locations of the cache data dirs
# for the Cache data, or the Session saves, un-comment the desired settings
# here:
beaker.cache.data_dir = %(here)s/data/cache
beaker.session.data_dir = %(here)s/data/sessions

# SQLAlchemy database URL
sqlalchemy.url = sqlite:///%(here)s/development.db

# WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*
# Debug mode will enable the interactive debugging tool, allowing ANYONE to
# execute malicious code after an exception is raised.
set debug = false

# Logging configuration
[loggers]
keys = root, routes, myapp, sqlalchemy

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[logger_routes]
level = INFO
handlers =
qualname = routes.middleware
# "level = DEBUG" logs the route matched and routing variables.

[logger_myapp]
level = DEBUG
handlers =
qualname = myapp

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither.  (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

Line 13 is commented out and replaced with line 14. This instructs Paster to use WSGI/FastCGI via flup rather than answer the HTTP requests directly. Line 16 must match the TCP port specified on line 41 of nginx.conf (as discussed in Part 1 of this tutorial). Finally, line 41 disables the interactive debugger.

Deploying to Production

Before we can install our application, we need to install a virtual environment using the same process that was used when you first began developing your application. (I’m assuming that you used python-virtualenv here; if you’re using Buildout, set up your environment similarly).

cd /path/to/my/project/
wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.4.5.tar.gz
tar zxfv virtualenv-1.4.5.tar.gz
cp  virtualenv-1.4.5/virtualenv.py ./
rm -r virtualenv-1.4.5
rm virtualenv-1.4.5.tar.gz
python virtualenv.py --python=/usr/local/bin/python2.6 /path/to/my/project/env

In above example I’m using the “–python” option to specify that I want to use Python 2.6 in the virtual environment since that is the version I used to develop my app. If I hadn’t made the distinction, I would have ended up with the default interpreter on this server, Python 2.4

Now that we have the HTTP server installed and configured, our application package set up, and our virtual environment ready to go, it’s time to deploy! Here’s what we need to do:

  1. Stop our application if it’s running (e.g., if we’re upgrading an existing app)
  2. Checkout the HEAD revision from our SVN repo into a temporary directory on the production server.
  3. Use setup tools to package the app into an .egg
  4. Use easy_install to install/upgrade our app and install the dependencies we listed in setup.py
  5. Clean up our temp files
  6. Copy over our production.ini
  7. Start/restart the application

Luckly, I’ve created a shell script to automate this process for me. I would not recommend that you try to use this script verbatim. It’s not very smart, does zero error checking, and is tailored for my environment. I will update this post when I have the time to make it a bit more flexible and bulletproof. But it should be a good starting point:

deploy.sh
#!/bin/sh

PROJECT_NAME='myapp'
PROJECT_DIR='/home/myapp'
SVN_REPO='svn+ssh://[email protected]/svn/repos/myapp'
SVN_DIR='/tmp/svn-'${PROJECT_NAME}'/'
SVN_USERNAME=''
SVN_PWD=''

# Save current directory
pushd . 

# Kill our current server if it's running
OLD_PID=`pgrep -f ${PROJECT_NAME}`
if [ "$?" -ne "1" ]
then
   kill ${OLD_PID}
fi

# Remove any previous versions of our app
rm -R ${PROJECT_DIR}/env/lib/python2.6/site-packages/${PROJECT_NAME}*.egg

# Create our temp dir for the SVN checkout
mkdir $SVN_DIR

# Checkout the HEAD version from the repo
svn co ${SVN_REPO} $SVN_DIR --username $SVN_USERNAME --password $SVN_PWD
cd $SVN_DIR

# Create our .egg from the SVN repo
${PROJECT_DIR}/env/bin/python2.6 setup.py bdist_egg
EGG_FULL=`ls ${SVN_DIR}/dist/*.egg`
EGG=`basename ${EGG_FULL}`

# Install/Upgrade application using our new .egg
${PROJECT_DIR}/env/bin/easy_install -U $EGG_FULL

# Copy over the production.ini
cp -f ${PROJECT_DIR}/deploy/production.ini ${PROJECT_DIR}/env/lib/python2.6/site-packages/$EGG/

# Recreate our sym link
rm ${PROJECT_DIR}/app
ln -s ${PROJECT_DIR}/env/lib/python2.6/site-packages/$EGG ${PROJECT_DIR}/app

# Nuke temp SVN dir
rm -R ${SVN_DIR}

# Jump into our app directory and restart the production daemon
cd ${PROJECT_DIR}/app
${PROJECT_DIR}/env/bin/python2.6 ${PROJECT_DIR}/env/bin/paster serve production.ini --daemon

# Restore original directory
popd

And if you haven’t already done so, now start Nginx:

/etc/init.d/nginx start

Congrats! Your Pylons app is now deployed in production! The most challenging part about customizing this shell script for your use will probably be the SVN portion. I am using SVN over SSH which works very well, but does require a bit of configuration (which is beyond the scope of this article).

Conclusion / Caveats

There are a few things that I still need to address yet about this process:

  • I am having issues using OpenID and Authkit under this configuration with one of my Pylons apps. I believe it’s related to the Nginx FastCGI configuration. I need to spend some time developing a deeper understanding of FastCGI and Nginx and then will update my config files appropriately.
  • Paster will not start at system boot. This can be fixed rather easily by creating an init.d script for it. I have not yet done so, but there’s already at least one floating around on the web that will do the trick.
  • I have not set up logging in the production.ini file. This is important if you care about how well your app is running once in production. I will likely be making these changes ASAP.
  • deploy.sh needs quite a bit of work. I will likely rework it the next time I have a new Pylons app that needs to be deployed.

Having said that though, I’m finding this setup to be a joy to work with. I simply check-in my latest revision of my app, SSH into the production server, and run deploy.sh. Bam! New version online in about 20 seconds.