One of my many functions as the KARL Champion at Six Feet Up is to release updates for our KARL customers such as OSI and Oxfam Great Britain. We release copies of KARL software for each customer in both a testing and production environment. During the KARL migration, from Xens to a combined KARL Hosting Infrastructure, we spent some time addressing some concerns with how we deploy new updates.

This represents an oversimplification of the process and reviews some of the steps involved in the old process. In the old process, we often performed some of the following steps:

• svn up (retrieve the updates from the repository) on the buildout directory for the customer's copy of the KARL buildout
• Running the buildout
• Restarting Supervisor

One of the biggest concerns when we ran buildouts is "Will it successfully complete?" Much of this concern is managed through adequate levels of testing. Even with the thorough testing and running projects through QA teams, there always is a chance that something will be missed. For KARL, as with other projects at Six Feet Up, the developers will have one set of configurations for running the product locally so that they can perform development tasks. When deploying a buildout to a staging or a production environment, we often used a different set of configurations established for the environment on which that buildout is being deployed. These configuration changes, environment differences or even something that slipped into the code base after QA completed testing can lead to random and unexpected results.

When a buildout is being updated, it deletes the old commands in the bin directory and recreates them. If an unexpected error is encountered during buildout, a buildout may not complete. On a production environment this could be devastating.

This single, critical issue began a conversation and led to the creation of a process to identify how we could we improve our release processes for KARL sites. As the exclusive host for KARL sites, we realized that our customers deserve greater certainty, which made this issue a priority. For some of the projects at Six Feet Up, we archive a copy of the buildout before updating. For our KARL deployments, the KARL Development team identified and created a list of proposed changes that might be considered in order to handle this type of issue.

I spent quite a bit of time trying to implement this plan while making sure that we automated as much as possible. Chris Rossi of the KARL Development team and I took some of my automation scripts and converted them into a package of tools to assist in the KARL release management process.

High level overview of the change.

For each customer, we have a separate directory location where we used to checkout the buildouts. We now make sure we check them out to a unique directory, which we use the tag number as part of the filename to make the sure the directory is unique. For tag 3.0 for customerA we might produce a directory structure like this:

• /all_KARL_Sites/customerA/tag_3.0/buildout_files

Next, we run the buildout. If, and only if, the buildout was successful, we then make a sym link from the folder tag_3.0 to a directory for "current". This leads to a directory structure such as:

• /all_KARL_Sites/customerA/tag_3.0/buildout_files
• /all_KARL_Sites/customerA/current (-> points to /all_KARL_Sites/customerA/tag_3.0/)

For our supervisor, webserver or any other application that needs to use files for customer_A, we point them to

• /all_KARL_Sites/customerA/current/buildout_files

When we do the next release, we start with a clean svn checkout in a new directory. We run the buildout and update the "current" sym link to the new tagged directory, leaving us a directory structure of:

• /all_KARL_Sites/customerA/tag_3.0/buildout_files
• /all_KARL_Sites/customerA/tag_3.1/buildout_files
• /all_KARL_Sites/customerA/current (-> now points to /all_KARL_Sites/customerA/tag_3.1/)

What's our gain?

When doing this, we can retain the last buildout. If something is discovered immediately after the release, we could switch the "current" sym link back to the previous install. If the buildout fails to finish, the "current" link is already pointing to the previous install and therefore the site is not offline and there is no "fire" to put out. We can stop the release and take our time to find the problem.

What are the side effects?

When we began to implement this process, we identified some issues and tackled them one by one.

First, each time we release, we install a fresh copy of the buildout. This means any "data", "files" or "logs" that are created inside the buildout directory need to be managed or they will be lost when the next release occurs. For our KARL sites, we had mailout, zodb, blobs and some other items that were stored inside the buildout directory.

We managed some items like mailout by moving those folders to the home directory of the user that the KARL application was running as. For other items like zodb and blobs, we created the same type of directory structure in a different location on the server. We only touch the zodb and blobs for a release to KARL if we perform an evolve, on the data. We typically don't have an evolve for each release. By moving the zodb and blobs outside of buildout, we only touch them when we need to do an evolve.

Since we were taking so much care in keeping backup copies of code by using fresh copies of buildout, it made sense to extend this to our data management as well. When we have to perform an evolve statement, instead of doing it live against the current data, we make a copy of the data into a separate directory. If the evolve is successful, we then update the data's "current" link to point to the correct new data directory in the same fashion we did above for the code.

Evaluating Pro's & Con's:

Cons:

• Complexity can add more steps to do an update
• Each update is a full, fresh install
• Each update takes longer with this method
• Backup copies of code, data, etc. can use more disk space

Pros:

• Safer Installs
• Rollback capability
• Backups of code and data
• Failures of install don't kill the live site
• Standardization of folders and setups can allow for easier capability to automate

Conclusion

By performing this process, we turn every release into the equivalent of installing for the first time. Everything is run fresh:

• fresh svn checkout,
• fresh buildout,
• fresh copy of the zodb and blob data, etc.

This complicates the update process which makes the requirement for writing scripts to automate the deployments a must. Now with scripts, it is easier than before to do an update. Typically, one command and I can deploy an updated tag for a customer. The best benefit of all: when running a buildout, I know we aren't going to take down a site if a buildout fails.

If you would like to read a more technical perspective of the exact concerns identified and the proposal of changes from the KARL Development Team, you can read more on the KARL Project site

We recently migrated all of our KARL Hosted Clients to a new infrastructure. Six Feet Up is the exclusive provider of hosting services for KARL, a knowledge management solution. Previously, we provided each of our KARL clients with a separately-managed Xen (a virtual server) to host their KARL Application.

The new infrastructure we deployed no longer requires a separate Xen for each client. This substantially has increased the efficiency of our hosting support efforts. This reduces the number of maintenance requirements of items such as operating system patches, dependency patches and other issues outside of KARL for each client. These improvements allow us to provide much more attention to the maintenance of the KARL application itself for each client. Additionally, it allows us to better utilize the system and disk usage requirements needed to run our clients' KARL sites.

Once discussions began about moving to a new infrastructure, Six Feet Up and the KARL Development Team worked together to create a pilot project to evaluate what the scope of a migration and how to evaluate the performance of the pilot. Once the pilot project was completed and its results were evaluated, the decision was made to move forward with the new infrastructure. To do this we migrated all of our existing clients to a new single hosted platform free from Xens.

After the migration, I became the release manager for our clients' KARL Hosted Sites. I will manage the updates and the deployment of new clients' KARL sites. By managing the releases for KARL updates, Six Feet Up now is providing more support for our KARL clients and allowing the KARL Development Team to spend more time on KARL application improvements. With the KARL Development Team, we implemented new methods to deploy multiple production KARL Sites on one machine and worked to automate as much as possible.

With the new infrastructure, I have set guidelines for standardization. Because of the decision to put more standardization in place, we now use a common directory layout, setup and configuration style on each KARL client. Since we have that more uniformity, we found it much easier to automate more of our processes. Before we actually performed the migration, Chris Rossi of the KARL Development Team and I wrote scripts and developed processes to automate the release of an update. This has been a great launching point for me to begin to automate as much as I can to manage our KARL client sites.

During the week of the migration, we accomplished almost no down time for each client. We achieved this by spending the necessary time on each client's configurations and standardizing them as much as possible. Next, we deployed to the new server using our scripts. By updating our /etc/hosts files, we could verify that the site was working correctly while the world was still going to the production site on the old Xen.

When we were ready to go live, we did the following steps in rapid succession per client:

• Sync data to the new server for this client
• Put up a maintenance page on the new server for this client (The world was still going to the site on the old Xen at this point)
• Turn off mail processing on the new server for this client
• Redirect mail from the old Xen to the new server and watch the logs to verify the e-mail was successfully making the redirect to the new server.
• Restart Apache on old Xen with a proxy redirect to the new server. (The world sees the maintenance page on the new server by being redirected there from the old Xen)
• Final Data Sync from old Xen to new server (This took from <1 sec to < 1 minute typically)
• Put client site on new server in production mode (Users could use KARL site again on new server)
• Restart mail processing on the new server for this client
• Update DNS to new server/Notify client if they manage their own DNS that they were ready to update their DNS

It was a resounding success. We migrated all of our clients without any major surprises and with very little down time.

We are excited about the new infrastructure. We expect to be able to roll out more new features for our clients in a faster, more standardized fashion to benefit all of our KARL clients. In addition, this migration allows us to deploy KARL to new customers more easily.

If you would like to learn more about KARL, collaborative intranets or knowledge management systems, please contact us.

Six Feet Up System Administrators Lars Noldan and Kurt Maier will participate in the Hope Ride to benefit the Hope Food Bank. The 25-mile bike ride will be September 18, beginning at Hauser High School in Hope, IN.

Six Feet Up wishes them a safe ride and sunny skies as they pedal for a great cause. Look for more information on how Sixies are making a different in their communities in the coming months.

One of the common concerns with Plone has been whether its complexity causes it to perform more slowly than other Content Management Systems. This complexity gives Plone more security, better enterprise controls and ease-of-use for end users, but in the past has lead some to call Plone "heavy".

Jon Stahl, Senior Strategist at Groundwire, has been instrumental in testing Plone 4. His tests show that not only has Plone improved over previous versions, it is now moving past its competitors. His post from January led to some worthwhile discussion in comparing the benefits and drawbacks of the most common CMSs. Matt Hamilton's findings also support this.

As we move toward the official Plone 4 release, we expect that some who had concerns about Plone's speed in the past will be pleased with the progress that has been made. Plone has been able to "lighten up" without losing the functions and features that have helped make it an industry leader.

Search Engine Optimization (SEO) is a hot topic these days and you want to ensure that your site is discoverable by the crawlers that index the World Wide Web. One of the most frequent questions we get from new clients is "How will the public find our site?" You are going to want your on-site SEO be as easy and automated as possible.

Deploying Plone as your Content Management System has many advantages such as making it easy to keep your content fresh and add new content easily, but it also handles many of these SEO tasks for you. Quite often we have launched a Plone site and had Google index most of the site in a matter of days without any additional work. Searches for key phrases that are relevant to our content quickly surface in the Search Engine Results Page (SERP).

So without further ado, here is the list:

1. Automatic <title> tag generation with page and site title

   The <title> tag is important for a couple of reason. First, on the SERP, it is used typically as the heading for that result. The second benefit is that the Search Engines weight these keywords a bit more heavily and having relevant terms here is especially important.

2. Easy control of the description <meta> tag

   Just like the <title> tag, the description <meta> tag is used to display text on the SERP and it is also considered as indexable text by the crawlers.

3. Keyword rich, human-readable URL structure

   Plone will automatically take the title of your page in the site and create an ID that is used in the URL by separating the words with hyphens. In the SEO world, the search engines will use these as additional keywords when indexing the content. The folders that are above your page in the site also are used as segments in the URL for additional keywords and context. This performs much better than a URL like this:

   http://example.org/node/43
   http://example.org/Page.aspx?id=123
   

   Plone will create URLs that look more like this:

   http://example.org/consulting/data-center-buildout-and-design
   

4. Automatic XML Sitemap file generation

   Many of the search engines will allow you to help its crawlers index your site more efficiently by submitting a XML file that describes the structure of your content. This can help ensure that you have the maximum coverage of your content indexed. Google goes so far as to give you tools to show you how much of that sitemap has been indexed and lets you know how often it checks with your site.

5. Automatic "breadcrumb" navigation generation

   "Breadcrumb" navigation allows your pages to contain more keywords based on the folders that are above the current page in the site. Since the folders above should have titles with relevant keywords to your content, this can add a small boost to the relevance of your page.

6. Useful 404 page

   Plone will respond to requests for pages that do not exist with the correct status code, 404, and will also suggest content that you may have been looking for by querying the site using keywords from the URL entered. This helps humans, and it also provides links to pages that may have moved to crawlers.

7. Built-in editor encourages appropriate use of header tags

   Plone will use an <h1> tag around the title of the page in your site and any headings or subheadings in your content are converted into <h2> or <h3> tags and so on. This will help highlight what should be the important concepts in your content and give more weight to the headings. The search engines value well-structured content and can use this structure to determine the relevance of keywords in your content.

8. Built-in support for analytics such as Google Analytics

   From the Site Setup panel inside of your Plone site, you can copy and paste the analytics javascript code into a textbox for inclusion into every page in your site. Now you can easily measure the effects of your SEO efforts.

9. Clean/validatable HTML

   This is related to the appropriate usage of headers in your pages. The search engines will place some favor on pages that are valid. You get some extra credit here for knowing what you are doing.

10. Easily exposes Dublin Core metadata

    The Dublin Core Metadata Initiative (DCMI) has taken on the task of developing a standard set of interoperable metadata standards for a broad ranges of purposes. Plone allows the site to expose these as additional <meta> tags in the <head> of your page. These additional metadata tags can be used by any system that also understands the Dublin Core and search engines can choose to weight this additional information when indexing your site.

Not many systems can say they cover as many areas for SEO out of the box as Plone does. In addition,there are even some community contributed add-ons that can increase the on site SEO of your site even further.

The whole reason to have a website is to have people find it. Using Plone can increase the chances of that happening with less effort from the members of your organization.

The next major version of Plone is now in the Release Candidate stage and will be officially available in the coming months. I have been working with Plone 4 through the beta stages by fixing tickets during Plone Tune-Ups, but it wasn't until recently that I got to work with the new version as if I were implementing a design for a client.

When I style Plone sites for clients, I use the initial theme that comes with Plone as a base, which is easier than starting from scratch (without preset styles). Plone 4 has a brand new theme (Sunburst) that is installed by default.

There are a lot of great new features in Plone 4 that I am excited about like the ability to hide portlets and using TinyMCE as the default editor. However, while creating a new theme based on Sunburst, I ran into minor difficulties because of some differences seen in Sunburst. I quickly realized that using this theme as a base for customizations may be a little more complicated than I had hoped.

This led me to the decision to create a new theme, that I would use as a base theme for Six Feet Up's future projects. The result is Plone Basic. It is similar to the Sunburst Theme, and in fact I started by copying the CSS from Sunburst, and pasting it into Basic. I stripped out a bit of the code and did some cleaning and reorganizing. My whole purpose for this new theme was to create something that can be easily customized. This allowed me to strip out many of the "prettier" styles, since I don't intend to use this theme on its own.

If you are a Plone themer, I encourage you to test this new theme and give me feedback and/or suggestions. One of the major components is the re-addition of base_properties. Some new properties have been added to help with quick overall color changes. It also reverts to the original main_template, which uses a table. There has been discussion of using a grid-based layout for Plone Basic, which I may do as long as it is still easily customizable.

The 0.1 version of Plone Basic has been released and is available for use. It is on pypi, or can be checked out from the collective.

At Six Feet Up, we strive to make Plone as helpful as we can. Please let me know what you think.

As I was coming to the office the other day, and I heard that one of Six Feet Up's clients, American Red Ball, was sponsoring an exhibit for another one of our clients, The Indiana Historical Society.

These two companies have come together to present Destination Indiana, which means they had to discuss underwriting costs, how to position the advertising, signage and all that goes into a large sponsorship like this one. In all of these meetings, they probably never had the opportunity to recognize that their websites had something in common: Six Feet Up.

With social networking in our personal lives, we find that the degrees of separation have shrunk rapidly. We are finding this in our Plone client networking too. I am sure I will hear more and more about our clients collaborating on community projects. I wonder if any of them will recognize their mutual connection to Six Feet Up and Plone.

We wish the Indiana Historical Society and American Red Ball great success with the exhibit and continued success with their built-on-Plone websites.

The following post talks a bit about the hosting environment I built for my personal site. Please feel free to ask me questions on the following configuration and I will do my best to answer them. Now, let's light this candle.

Lnoldan.com in the past has been a testbed I've used for getting more familiar with technologies used during my day job at Six Feet Up. I recently re-deployed the site on Plone4 Beta 3 with RelStorage using some of the technologies we deploy daily to create the best hosting environment possible for this site. The attached diagram shows the pieces I am using, and how they are connected to one another to host this site.

At the head of the hosting chain we have the nginx web server. This is configured primarily to manage virtual hosting between lnoldan.com and my soon-to-be deprecated w9zeb.org WordPress site. It also manages SSL for encrypted site access and authentication.

If the web request is for w9zeb.org, it is passed to a FastCGI process, which in turn connects to the MySQL database on the back end. Nginx isn't able to parse PHP content by itself, which is the reason for the FastCGI process. I hope to move the content from that WordPress blog into Plone in the near future.

If the web request is for lnoldan.com, it is passed to my varnish caching server. If varnish has the request in cache, it will serve the request directly, which is very fast. If varnish did not have the request cached, it goes to its back end, which in this case is another running nginx process. This instance of nginx is where I have the VirtualHostMonster configured for Plone. The advantage here is that the URLs being processed by varnish are identical to the URLs the end user sees in his or her browser address bar.

The VirtualHostMonster proxypass sends traffic to HAProxy load balancer, which sets a cookie to help determine which back end any given session should be re-directed to on future requests. This balances traffic to four Zope 2.12 instances. These instances are configured to use RelStorage with PostgreSQL as the back end datastore for the ZODB. I have also configured RelStorage to use memcached.

The method currently used by RelStorage to interact with memcached isn't quite ideal, but it's functional. When a request comes in, the Zope instances first query memcached. If the data is there, it serves it. If memcached does not have the data, RelStorage sends the query to PostgreSQL. Once the response is received from PostgreSQL, RelStorage sends the response to memcached so it's available for future requests. Simultaneously the request is processed by Zope and sent upstream towards the end user.

Hopefully this makes sense and provides ideas to help solve your hosting needs as well. The specifications on this Virtual Private Server are as follows: I am running FreeBSD 8.0-Release (32bit) with 1.5gb of memory, 2 virtual CPUs, and 45Gb of disk space. I am using supervisord with memmon to manage the four running Zope instances. Memmon is configured to allow not more than 150mb of memory to be used per instance before restarting them. All other services are configured to run via the FreeBSD rc system.

If you would like to see any of the configuration files in this hosting stack, including the buildout for the site you can email me at: lars@sixfeetup.com.

Cheers!

Six Feet Up team members Calvin Hendryx-Parker, Clayton Parker and David Blewett traveled to Ohio State University in Columbus for the 3rd PyOhio conference last weekend. This year's event continued the momentum from earlier conferences. Python users from the Midwest gathered to learn new strategies, maximize Python capabilities and wrap up with two days of development sprints. The location of the event was a great choice, making use of OSU's excellent Ohio Union.

The quality of the talks was outstanding, while the open-space area saw increased usage from last year. The atmosphere generated practical and enthusiastic discussions among conference goers who readily shared their experiences using Python.

Some of the talks that we found especially interesting included:

• Micro-optimization Strategies for Python by Nick Bastin
• Genetic Programming in Python by Eric Floehr
• Log Analysis with Python by Scott McCarty
• Wrangling the bits, standardizing how apps get built by Rick Harding
• PyPy and Unladen-Swallow: Making your Python Fast by Alex Gaynor

Six Feet Up CTO Calvin Hendryx-Parker presented a well-attended talk: Controlling UNIX Processes using Supervisor. Our lead developer Clayton Parker presented Code With Style: Your Code Should Be As Fashionable As It Is Functional.

Thanks again to all of the PyOhio Volunteers. Their hard work and passion for Python made the conference successful, enjoyable and informative. I look forward to seeing everyone next year!

A strong Content Management System must be accessible to as many people as possible. One of the areas that doesn't get much attention is that the CMS you choose must be easy to use. If it isn't, why bother to have one? A good CMS will not stand in your way of producing great content for the site; it will not complicate the process, only improve it. After all, the content is who you are, and you won't want to fight the tool to get things done.

In my opinion, a great CMS will allow people at all levels of your organization to participate in the creation of content on the site. That being said, you need to make sure your CMS also has a robust workflow engine incoroprated so that it streamlines and matches any processes you already have in place.

These two specific areas in which Plone excels particularly well as compared to other systems. Not only is it easy to use, but it is also easy to download and install from the download section of plone.org. Once the program is installed, anyone in your organization can contribute to creating your web presence and help generate content that will increase traffic.