This article describes how we deployed our issue tracker system (RT). If you've browsed around our web site, you may have noticed that we encourage our community to submit bug reports via launchpad. We discovered however that we also needed to track issues that don't fall into the category of 'bugs' (e.g., customer's issues, internal IT issues, etc.); or if they do classify as bugs, they are not Eucalyptus related. To meet these needs, we decided to investigate a few alternative issue tracker systems.

Searching for an Issue Tracker

When we began our search for an issue tracker, all our efforts were centered on Eucalyptus releases and development. We had not much spare time, nor did we have a surplus of machines on which to test the various alternatives (these were in use by our QA system). Consequently, as often occurs with IT projects, we started with two desktop machines and hatched a plan over coffee breaks. The issue tracker search was more of a fun activity pursued while taking breaks from coding.

In summary, we had limited time for a project with relatively extensive requirements—the need to install multiple applications along with their full environments; the need to install diverse OS; the capability to switch between systems for comparative analysis; the need for multiple individuals to access and experiment with each system on their own time—and all of this with just two desktop machines! Fortunately, we knew of this very cool project that turns commodity machines into a private cloud. Yes, we turned to the Eucalyptus cloud for help.

Setting up the test cloud

The first thing we did was to convert the two desktops into a test cloud. To do so, we installed a second network card (NIC) into one of the machines and christened it the 'front end' (the host machine for the Cloud Controller (CLC), Walrus, Storage Controller (SC) and Cluster Controller (CC)). The other machine we used for the Node Controller (NC)). On the front-end machine, we connected one NIC directly to our internet uplink since we wanted to make the issue tracker accessible from everywhere and our firewall would have interfered. The other NIC we connected directly to the second machine to ensure a totally private network connection between CC and NC (no interference from other dhcp servers and/or other network traffic). Fig. 1 represents a schematic of the current production network, not the original test cloud, but in Fig. 2 you can see virtually the same setup as in the describe here, with the NCs having connection with the front-end only.

Figure 1. The current production cloud resides on the same hardware as the ECC. (Note that subsequent figures in this article focus on the front end (CLC, Walrus, SC, CC) and node (NC) machines only.)

With the physical infrastructure in place, we installed the latest version of Eucalyptus (1.6.1 if I recall correctly) on the machines, configured Eucalyptus in MANAGED mode (see the networking guide for more information), assigned a few public IPs that we had scavenged, and uploaded a few of our test images. Not much to report here: All worked very well. We were ready for the issues tracker show down!.

Moving the service into the cloud

With limited knowledge of the various issue tracker applications available, we were a bit uncertain how to proceed: Should we create an image for each issue tracker application? Should we install the issue tracker each time we start an instance? Should we create images containing the various database back-ends required by certain issue tracker apps? Where do we store the persistent data when we don't yet know which data is persistent?

Figure 2. An instance is started using euca-run-instances. The instance will see a local disk (sda) where the root file system (sda1), swap (sda2), and ephemeral0 (sda3) reside. The virtual disk sda physically resides on the NC disk allocated for the instances, and the disk is volatile, that is, it will disappear once the instance terminates.

We knew that we wanted the database on Elastic Block Storage (EBS) for persistence. But knowing little about each of the issue trackers, we decided it would be wise to ensure persistence of all data , and thus proceeded to install everything on EBS. First we created appropriately-sized volumes to host each issue tracker along with a full OS (see Fig. 3). Then we created an instance (see Fig. 2), attached the volume to it (see Fig. 4), and installed the OS, issue tracker and database in a chroot environment (for more detailed instructions see our community wiki). The chroot environment was used since Eucalyptus 1.6.2 and 2.0.2 do not support yet the booting from EBS primitive. We also made sure that all important services (e.g., syslog, logrotate, cron, etc.) would run in the chroot image, since the OS on the instance was only to be used to access the real environment on the EBS volume. We repeated the process for each issue tracker thus priming all volumes.

The final task was to allocate one Elastic IP (EIP) for each issue tracker we wanted to test. We then started all of the various instances, but quickly encountered the limitations of a desktop machine—not enough RAM, only 2 cores, and a small and slow disk. But with Eucalyptus no fear: We configured the small instance to use only 256MB of RAM and we overcommitted the cpus on the NC (the option MAX_CORES in eucalyptus.conf) and presto! our test environment was ready.

Figure 3. A volume is created using the euca-create-volume command. The Storage Controller (SC) handles the creation, maintenance, and exports of the volumes to the instances. Note that each availability zone has its own SC, and the volumes are local to that zone (i.e., only the instances in that zone can use the volume).

Moving into production

The real test began soon after we announced internally the availability of the different services. Less than a month into it, we began to receive complaints from our engineers that one of the services had become exceedingly slow. A quick check of the service revealed that it was already being used in production, with a load clearly beyond the puny 256 RAM allocated to the instance! We had no choice but to conclude that the test was over and that our engineers had declared—through their prolific use of the service—that request tracker (RT) was the winner.

Figure 4. The volume created in Fig. 3 is attached to the instance created in Fig. 2. The attached volume will appear to the instance as another disk on the SCSI bus (in the figure as sdc) and it can be used as such (i.e., partitioned, formatted etc.). While the disk appears to the instance as local, it resides on the SC and is persistent storage, that is, the volume will remain when the instance terminates (unlike the instance-store sad, which is volatile and will terminate when the instance terminates). Volumes can be attached and detached as needed.

Finally, we terminated all other instances, removed the extra volumes to create space for snapshots (since we went live without really planning for a backup solution, we used snapshot as backups), restarted the instance with all available RAM, gave it the 2 available cores, and christened it production. When our infrastructure at CoreSite (see Case Study: Installing the Eucalyptus Community Cloud (ECC) for a picture of one of our racks) came online, we quickly reserved a few nodes and a few public IPs and moved our original production cloud to the new location, where we upgraded it to the latest Eucalyptus (2.0.1 at the time) and restored the service with minimal downtime.

Next steps

With RT running in a real production cloud on a superior infrastructure, we have seen no performance issues, and we have added a public queue for our contributors to submit patches to Eucalyptus (see Viewing submitted patches in our Developer's corner section). We are however still using snapshots for backup, and are thus planning our next steps.

Figure 5. Our support site (https://support.eucalyptus.com) lives in a chroot environment in the volume exemplified in Figure 3. This allows RT to be moved easily across instances, and to take snapshots (which works as a backup) at will.

Taking snapshot of a live file system is a bad idea: Thus far ext3 has been forgiving, but we need to check each snapshot for possible corruption. So a proper, more application friendly backup will be employed, and we are planning on using Walrus to store it. Also, RT is still running on the original EBS volume we created for testing it. We are considering the idea of splitting the database from the RT front end, and keeping only the data on EBS, while RT will be reinstalled and reconfigured each time the instance starts.

The winner is ...

Eucalyptus! This tale is another very simple example of how decoupling real infrastructure from cloud infrastructure allowed us to ramp up a service quickly and easily, with our users (the very engineers you have to thank for Eucalyptus!) barely noticing the changes behind the scenes.

Attachment	Size
ecc-volume-attach-1.png	28.89 KB
ecc-volume-attach-2.png	30.84 KB
ecc-volume-attach-3.png	32.94 KB
RT.png	43.27 KB