Ubuntu error: key_from_blob

If you have seen this error before on Ubuntu (or any flavor of linux for that matter), after some annoying digging i finally resolved my issue.  When adding a new DSA or RSA key to the authorized_keys file of your remote ssh server, you may see the following error

Nov 30 11:58:56 li321-228 sshd[7292]: error: key_from_blob: can't read dsa key
Nov 30 11:58:56 li321-228 sshd[7292]: error: key_read: key_from_blob AAAAB3NzaC1kc3MAAACBALZa7U63gJeJm5zHAaP9x1fQKfRdvkbHukV6T8S+392Vs74gQTLn\n failed

Not noticing the "\n" at first glance, i realized that when i pasted the multi-line DSA key into the authorized_keys file, it had newlines in it.  Take out the newlines and you are set !

An MD5 Hashing function for varnish

I was toying with varnish, a very cool open source web accelerator that seems to be getting a lot of attention recently. Since we have done some fairly complex caching setups at my current employer using a well known CDN, I figured I would dig in and seen how capable Varnish really was.

For starters, the documentation is unfortunately a bit slim. There is virtually nothing around on google and no real advanced or complex examples anywhere. So when you need to do some serious tinkering and you hit an error, you just need to go through some trial and error.

The first thing i noticed was that there were no builtin "hash functions" you can call, in particular MD5. I had a need to take 3 parameters from the query string (something you need to do with a regex) and concatenate them into an MD5 hash. Since Varnish didn't supply this functionality I figured I was pretty much out of luck. However, a little more digging and I discovered a couple of VERY interesting things. Firstly, the DSL that Varnish uses is basically compiled into C to be super fast. What this means is that there is the ability to put inline C directly into the config file. Thats right, you can basically wrap your C code with C{}C right in the config file and do almost anything in there. Now we are getting somewhere. But was I really going to find an implementation of md5.c and stick that in the middle of a config file? That seemed like serious overkill...

Then i discovered my second secret feature, the load_module. There is an example buried in the wiki on how to compile the MaxMind GeoIP library into VCL as a module and execute its function to query for a Country to IP Lookup. Ah, now we are talking. So what did I need to do? What anyone would do in this situation, grab an implementation of md5 in C and write your own library of course! So that is what I did. I downloaded the md5 implementation written by L. Peter Deutch (http://sourceforge.net/projects/libmd5-rfc/files/) and then wrote my own library. This involved a couple of steps....

First I had to write my own md5 library that i can expose to Varnish, which I conveniently named md5_hash. I had to create basically a C source file with the following contents:

char * md5_hash(char * md5_string)
{
int status = 0;

md5_state_t state;
md5_byte_t digest[16];
static char hex_output[16*2 + 1];
int di;

md5_init(&state);
md5_append(&state, (const md5_byte_t *)md5_string, strlen(md5_string));
md5_finish(&state, digest);

for (di = 0; di < 16; ++di)
sprintf(hex_output + di * 2, "%02x", digest[di]);

return hex_output;
}

Then i built a Makefile that turned this into my very own libmd5varnish.so to be used inside of Varnish. Now we need to load this into Varnish and make the md5_hash() function available. To do this you need to use inline C and place the following in your VCL file :

C{
#include
#include
#include

static const char* (*md5_hash)(char* str) = NULL;

__attribute__((constructor)) void
load_module()
{
const char* symbol_name = "md5_hash";
const char* plugin_name = "/etc/varnish/modules/md5/libmd5varnish.so";
void* handle = NULL;

handle = dlopen( plugin_name, RTLD_NOW );
if (handle != NULL) {
md5_hash = dlsym( handle, symbol_name );
if (md5_hash == NULL)
fprintf( stderr, "\nError: Could not load MD5 plugin:\n%s\n\n", dlerror() );
else
printf( "MD5 plugin loaded successfully.\n");
}
else
fprintf( stderr, "\nError: Could not load MD5 plugin:\n%s\n\n", dlerror() );
}
}C

Excellent. Now i have a function called "md5_hash" available to me via inline C. So how do I call it? Say you want to set a header that is the MD5 Sum of a string called "random blog post" (could just as easily be a header you extract via VRT_Get_Hdr). You stick this anywhere you need in your config:

C{
VRT_SetHdr(sp, HDR_REQ, "\006X-MD5:", (*md5_hash)("random blog post"), vrt_magic_string_end);
}C

Thats pretty much it, i did some basic testing and it works like a charm. To save someone else the hassle I opensourced the whole library I wrote and stuck it on github here https://github.com/denen99/libmd5varnish. Feel free to fork or improve it.

Hope this helps someone, I know I searched for hours with nothing in sight on how to solve this. There are a couple of posts on the mailing list that claim this will be natively party of version 3.0 but I didnt want to wait :-).

Good luck.

Using hazelcast with Jruby

So, i have been doing some reading on Hazelcast as I have been researching a bit on In-Memory Data grids. i spent some time with Gemfire (which is a very cool product) and then found this seemingly very close open source alternative. Its a linear scalable memory cluster that allows you to do a lot of very cool things in memory such as Hashmaps, Queues, Topics, Callbacks, etc. Since i have been toying with building some Java clients in Jruby i thought i would share some really simple examples of how i got up and running calling and interacting with the Hazelcast cluster from a Jruby client API.

To run hazelcast, simply download the latest version, cd into the bin/ directory and run "./run.sh". This should get your cluster up and running. My example uses the ip of 192.168.1.3:5701, you should substitute your own ip here.

Example 1: Basic Read/Write cache

This example simple connects to a hazelcast cluster and then writes 2 entries and then read 2 entries. Certainly not very valuable but shows how easily you could replace a memcache library, for example.

require 'java'
require 'hazelcast-client-1.8.4.jar'
import com.hazelcast.core.Hazelcast;
import com.hazelcast.client.HazelcastClient;
import java.util.Map;
import java.util.Collection;

class MyClass
def initialize
@client = HazelcastClient.newHazelcastClient("dev", "dev-pass", "192.168.1.3:5701");
@map = @client.getMap("default");
end

def write(k,v)
@map.put(k,v)
end

def read(k)
puts "Key: " + k + " Value: " + @map.get(k)
end

end

c = MyClass.new
c.write('key1','value1')
c.write('key2','value2')

c.read('key1')
c.read('key2')

When you run it here is the basic output as exected:

$ jruby test2.rb
Key: key1 Value: value1
Key: key2 Value: value2

Example 2: Enable event callbacks with hazelcast

This one i found a bit more interesting. Here, we register a callback for the "addEvent" method, which basically allows us to "subscribe" to the cluster interface that would give us a callback whenever an entry is added. There are also methods for eviction and updates as well, but kept it simple here for demo purposes. here i demo how when i add a new key on the console, i immediately get a callback in my jruby api script that a new item was added, very powerful stuff here.

require 'java'
require 'hazelcast-client-1.8.4.jar'
import com.hazelcast.core.Hazelcast;
import com.hazelcast.client.HazelcastClient;
import java.util.Map;
import java.util.Collection;

class MyListener
include com.hazelcast.core.EntryListener
include com.hazelcast.core.ItemListener

def initialize
end

def entryAdded(e)
puts "Event found : " + e.getKey + " value = " + e.getValue
end

end

class MyClass

def initialize
@client = HazelcastClient.newHazelcastClient("dev", "dev-pass", "192.168.1.3:5701");
end

def listen
sample = MyListener.new
map = @client.getMap("default")
map.addEntryListener(sample, true);
end

end


c = MyClass.new
c.listen


#######################
Hazelcast Server

hazelcast[default] > m.put 'key2' 'value2'
null
hazelcast[default] >


########################
Output

# jruby test.rb

Event found : 'key2' value = 'value2'

Use mod_rails with Freebsd 7.0

Man, been a while but i forgot how much i hate FreeBSD. Reminds me of the old days where solaris refused to ship their OS with a compiler or a decent shell, or require you to update /etc/nsswitch.conf to use DNS. Oh wait they still they do that with Solaris 10.. Ok, sorry tangent.

So if you need to compile mod_rails (aka passenger) on FreeBSD 7 with apache 2 and you used the default port install then you need to recompile with

make USE_THREADS=yes

and you will avoid that very nasty "Bus Error (core dumped)"

The other option is to use the Worker MPM and you should be good as well. I still cant mod_proxy to work but thats another issues.

Fun times.

Run db:migrate in jruby from warfile

If you have production servers running jruby/rails applications packaged as war files and dont actually have the full jruby installed to run your migrations, you can follow these steps. This will allow you to run db:migrate without a jruby installation on your java application servers

Assuming you are deploying your app called 'myapp' into tomcat5, you would have a warfile located in $CATALINA_HOME/webapps/myapp.war. if your war is not unwar'd you can run "jar xvf myapp.war" to generate the unwar'd directory of your app (which you will need)

# cd $CATALINA_HOME/webapps/YOURAPP/WEB-INF
# export GEM_HOME=$CATALINA_HOME/webapps/YOURAPP/WEB-INF/gems
# cp $YOUR_REPISTORY/Rakefile .
# mkdir -p db/migrate
# cp $YOUR_REPISITORY/db/migrate/* db/migrate
# java -jar lib/jars/jruby-complete-1.3.1.jar -S rake db:migrate


Thats it.

GemFire RubyGem (activesupport-gemfirecache)

So i finally got around to building my first Ruby gem, activesupport-gemfirecache. Its with a very cool distributed memory fabric called Gemfire. Its a java based distributed cache application, which has a fairly easy to use Java API. Since we are using Jruby/Rails I thought it would be an interesting project to try to build a custom Rails Cache store that can be used natively from within Rails. This is one of the real nice things about JRuby, is being able to call native Java libraries.

So I finally got version 0.0.5 out and so far it seems to work !

>> Rails.cache.region.getName
=> "RubyGemFireCacheStore"
>> Rails.cache.write('test1','adam1')
=> true
>> Rails.cache.read('test1')
=> "adam1"

Check it out here http://github.com/denen99/activesupport-gemfirecache

Phone Number Forms in Rails

So an interesting problem with rails accessors, I had to create a form that accepted a phone number as 3 different form elements to ensure that the phone number didnt have any special characters (instead of using a regex).  So the form had phone1, phone2, phone3 mapped to a number that looked like 212-555-1212.  The model needed to accept these 3 form params and then stuff it into the phone_number attribute, as well as vice versa to render the parts of the phone number into the appropriate text boxes.  So here is how I did it.

  Create the form view with the 3 form elements

    (<%= f.text_field :phone1, :size => 3, :maxlength => 3, :class => "text" %>)

      <%= f.text_field :phone2, :size => 3, :maxlength => 3, :class => "text" %> -

      <%= f.text_field :phone3, :size => 4, :maxlength => 4, :class => "text" %>

  Update model to accept these params

      Add an accessor

        attr_accessor :phone1,:phone2,:phone3

      Add a before_save to create the phone number

        before_save :make_phone

      Create the private method to generate a phone_number

       def make_phone

           self.phone_number = @phone1 + @phone2 + @phone3

       end

       Create the reader attributes to generate the appropriate phone elements

         def phone1

            self.phone_number.slice(0,3) if self.phone_number

         end

       def phone2

         self.phone_number.slice(3,3) if self.phone_number

       end

      def phone3

         self.phone_number.slice(6,4) if self.phone_number

      end

   Thats it, you should be good to go.  I think there is some other ways to do this with composed_of but it seemed like a lot of extra class writing that wasnt necessary.