wildchild

Yesterday, a friend called me with the most craziest deadline ever. Six hours after the phonecall, a website on his server would be aired in the Oprah Winfrey show. The server was already taking a big performance hit when the pre-casting aired in Chicago delivering a load of over 10 letting the serverbeast crawl to it’s servings. The precasting was expected to deliver 250,000 to 1,000,000 visitors while the national airtime would deliver inbetween 4 million and 50 million visitors; which required a solution to be made to distribute that load evenly; even before the main show gets aired to protect the server and sites from being brought to their knees…

This Cloud technique could be used to save hosting costs where bandwidth costs an arm and a leg by adding the script to a crontab to update its synchronizing process; it’s having as advantage to be fast, locally served with the nearest datacenter and rendundant.

A few solutions were in my mind, although it had to be one which was easy to plug into the existing infrastructure without loosing the site or the production server . My mind fell directly on Coral CDN , Amazon S3 and Google clouds; together with a third party provider to act as proxy/web server for images, movies and other space-eating critical files to a website.

After checking out the options and alternatives, I’ve landed on Cloudfront , a new beta program by Amazon, offering that kind of transparant proxy/web server services for files; so another 3rd party provider was not needed anymore for webserving cloudspace.

First, I started writing a little shell script which would go recursively through all directories, uploading all .jpg, .jpgeg, .gif and .png images towards the Amazon cloud. After checking the s3cmd tool a little bit more, I’ve found a much more easier way, which was the “sync” command through s3cmd.

Basically, I’ve done 6 times the same task, but every time linking to another root directory; because the images were too scattered around the website.

The files created were:

migrate.sh :

Very important, do not forget:

1. the –acl-public parameter setting your files public accessible (website files are public anyways!);
2. the -r parameter to go recursively through directories (since images were scattered all over the place);
3. ending backslash at both directory locations; else the command will fail!
4. use the –dry-run parameter to see what your script stores, before you will upload dozens of files to the cloud for nothing.

migrate.exclude :

# skip all _directories and directories
_*.
*.
*.mno
*.bak
*.db

migrate.exclude :

*.jpeg
*.gif
*.jpg
*.png
*.mov

After this succesful migration of images, I started looking towards the needed rewrite rules to get images served by the new cloudspace.

The rewrite rules :

RewriteEngine on
RewriteCond %{REQUEST_URI} \.(gif|jpg|png)$
RewriteRule ^/(.*) http://d4k6hf1klh8oo.cloudfront.net/tgw/$1

Final implementation:

After implementing in .htaccess they appeared not to work, because of hsphere and it’s configuration. To skip the entire web-control-panel thingy hell, I hard-coded these rules directly into the /hsphere/local/config/httpd/sites/#####.conf file; where the original site and cloud seemed to work perfectly in harmony. Almost all images were instantly distributed towards cloud computing and the serverload was instantly dropped from 8 to 3.

Before starting the work, I could almost strangulate my friend because of that deadline on short notice; but; after the work I got to thank him graciously, of having such great experience to get a site protected against such overloads. The task was done in 2 hours, while optimizing and implementing took another 2 hours. I’m for sure glad my friend has choosen the Linux way, because it wouldn’t be done inside 4 hours, if I had to find out a dozen more ways around Windows….

Thanks to cloud technology, such kind of disasters can be easy intercepted by distributing load evenly towards other service providers.

I hope to help others with this article, c(l)o(u)ding out; Freaking Wildchild !

No related posts.