Dreamhost… you win… or lose really.

You have already activated rack 1.1.0, but your Gemfile requires rack 1.2.1. Consider using bundle exec.

So with the impending release of Rails 3 there is a little dependency that was added. Rack 1.2.1. Dreamhost is using passenger which also requires a version of Rack. The problem is that it’s 1.1.0. I’m pretty sure it’s not going to be possible to get Rails 3 running on Dreamhost without the servers being upgraded…

If you’re a Dreamhost customer please be sure to vote this suggestion up.
Get Ready for Rails 3

Please be sure to file a support ticket as well.

For me, if this isn’t fixed within the next few weeks I plan on moving to linode.com

adding rails log rotation to dreamhost

While Dreamhost may rotate the apache logs for you there is nothing automatic to rotate the rails production logs. This may not be an issue since you have “unlimited” disk space but it’s a good idea anyway.

You need to install logrotate since it doesn’t exist by default on the server and then place it in a location where it can be run. You also need to create the configuration file and status files. Once that is set up, you can install a cron job via the Dreamhost panel.

Now it should be rotating your application logs nightly. You can add more sites to the conf file as needed.

I beat Dreamhost. How to really get rails 3, bundler and dreamhost working.

Please check out my earlier post so you can get the whole git/capistrano setup working as well.

So I’ve been trying to get rails 3 running with bundler on Dreamhost for a while. I’ve had a few posts on here about how to do it. In the end, they didn’t work completely. I’ve sent requests into Dreamhost to upgrade rubygems to 1.3.6 so the newest versions of bundler would work. I was told ‘No’ but go to this wiki site.

I wasn’t too happy about that since the top of the page has a big warning ‘DON’T DO THIS’. If you’re here looking at this you probably have the tech knowhow to do this and realize the world won’t end when you do.

So here’s the step by step directions for how I got it working.
You can check out the original wiki instructions at http://wiki.dreamhost.com/RubyGems.

Add this to your .bashrc
This will setup your shell to use local gems installed in your .gems directory, setup the path to check there first and opt/bin as well. Next we need to install a newer version of rubygems.

This will get rubygems installed and make sure you run your version before dreamhost’s. You then install bundler and rake. These are the only two gems I install in the system as I prefer to have all of my apps use their own gem versions. Putting everything into system is a big mess and a dependency nightmare on updates.

Next we need to make sure your app is setup to use the bundler gem. You need to modify your environment.rb and boot.rb

Make sure you change the application name to your name. Really you’re only adding the single line for the GEM_PATH

I was using the suggestions in the wiki but I eventually figured out that Dreamhost wasn’t properly picking up my gems even with the changed GEM_PATH. Adding the ‘Gem.clear_paths’ to boot.rb allowed the gems to be seen. This is what finally cracked the problem.

Hopefully this works for you.

dreamhost, please upgrade rubygems

I’ve posted before about trying to use rails 3 with bundler and dreamhost. This is now not possible. The shared servers for dreamhost use rubygems 1.3.5. In order to use the current version of bundler you need rubygems 1.3.6.

If you are a dreamhost user please make sure to vote up the install rubygems 1.3.6 suggestion from the panel (panel.dreamhost.com) For a company that has been good about supporting ruby and rails this is something that needs to get fixed ASAP.

facebook, privacy and failure

Hmmm. They contradict themselves…

As Facebook expands, I’m getting more concerned about the “privacy” they provide. There is more than one example of how they said they would not share information and changed the ToS to begin sharing that same information… without asking me. I don’t like that. I don’t have anything particularly personal up there but the idea that they just do what they want when they want doesn’t sit well with me.

mongodb, mongoid, will_paginate and sorting

Had a bit of an issue with will_paginate and mongoid. I couldn’t find an example of how to sort the pagination query and paginating without a defined sort order defeats the purpose.

paginate(:page => page, :per_page => size, :sort => [['ontology_term_id', :desc], ['_id', :asc]])

Instead of using “order” or “order_by” you can just use “sort” with an array of arrays.

Apple’s Numbers is part of iWork

The problem is that it just doesn’t work.

Take a file with more than 50 lines. Paste it into Numbers. Notice that it gets cut off (for me it was 45).

How can you not expand the number of rows to fit the content you’ve pasted? Just wow.

If there’s a Kindle app on the iPad, I want my Google Voice app for the iPhone

So it seems Amazon has created a Kindle app to run on the iPad…

If this makes it into the store, Apple will have a lot to answer for. There is no way they can approve this. It directly competes with the base functionality of the device. If it does get through, I want the Google Voice app on my iPhone immediately.

http://www.wired.com/gadgetlab/2010/03/amazon-shows-off-kindle-for-ipad/

distributed job processing options for GMiner

I’ve been working on a project that requires the processing of a series of jobs. I had originally written my own system for doing this because I wanted to know more about how they work. After a time, I decided to modify it, and found that I had broken it. Instead of trying to fix it, I decided to see if there was anything out there that someone else had done that would work better for me.

Resque

My first attempt was to use resque. It worked, but as I started to scale things up, I ran into some issues that I didn’t like.
It polled the DB a lot. While it was in memory, it was a lot of “do I have a message?” checks which seemed messy.
It wasn’t fast. There was a lot of overhead. Things “felt” slow as they were running.
It was memory based. Redis will store data to the disk, but it’s meant as an in memory system which gives it the speed.

What I did like is that it worked. The jobs finished and there was a nice web interface to see what was going on.

Cloud Crowd

I had looked at Cloud Crowd before and it seemed interesting. I like the web dashboard but it was also one of the biggest problems with Cloud Crowd. According to the authors, it was created to handle a small number of very expensive jobs. I have no doubt based on my experience that it would excel in that environment. My problems consists of a very large number of small fast jobs. Cloud Crowd ground to a halt pretty quickly. The dashboard was taking too much time to render which began to multiple the render time and eventually it needed to be turned off.
The other big issue I ran into was with how the workers processed their jobs. It wouldn’t start another job until all of the other jobs it had created finished. If you have a scheduling job that launches 10 processing jobs, the system gets stuck waiting for the 10 processing jobs to finish before it can start another scheduling job. Again, it works very well. Everything gets done and you get a result string but it was slow.

RabbitMQ

I decided to figure out what was wrong with my system since it was working at one point. I’m using RabbitMQ as a message broker to pass the jobs back and forth between daemons running on linux machines. I believe my issue was caused by using a topic exchange with a key per worker. I was running into issues where some processors were picking up messages from the topic that were not assigned to their key. Once I realized this was happening I decided to go back to a queue per worker. I wanted to get away from that since originally I had been creating multiple queues in rabbit that never disappeared. I changed the queues to be exclusive. Exclusive means that only one client (processor) can read from that queue. It also makes the queue self-delete when the consumer disappears.

I’m attaching the code for my system below. I’ll post more about how each of the parts works later. I hope to add a bit more control into the system, but as of now it’s pretty self healing and very fast.

http://github.com/mcwbbc/gminer_scheduler
http://github.com/mcwbbc/gminer_node
http://github.com/mcwbbc/gminer_processor
http://github.com/mcwbbc/gminer_databaser

properly rendering 404 errors from inside a rails application

I just migrated a site that had a bunch of links that have been in in the search engines for a while. Oddly it seems that the only thing hitting those links seem to be the crawlers themselves. I needed a way to invalidate those links, since I couldn’t create a proper redirect because of changing IDs.

/records/show/12345 used to be valid, but has been replaced with the RESTful version /records/00123. The ID is now also meaningful instead of a MySQL generated id.

My first attempt was to just redirect to the 404 page.
record = Record.find(params[:id]
rescue ActiveRecord::RecordNotFound
redirect_to("/404.html")

But as I watched the logs, I noticed that this really wasn’t right since it was still returning a 302 (redirect) and the a 200 (OK) code for those links. The crawlers were getting the instruction that you should just display the 404 page for those links. That might seem OK, but really I wanted them to get the 404 immediately and remove the page from their databases.

record = Record.find(params[:id]
rescue ActiveRecord::RecordNotFound
render(:file => "#{RAILS_ROOT}/public/404.html", :layout => false, :status => 404)

By rendering the 404.html directly and including the 404 status code, it should help to fix the situation.