Feb 3, 2015 - Updating & Rebooting Production Server Securities Efficiently


The GHOST vulnerability last week required me, as well as many other developers to update and reboot our servers. For services that have like 50 application servers behind a load balancer, this can be pretty tiresome.

As an AWS user, I haven’t cracked the answer to fully automating this task, but I have made it somewhat easy using some AWS CLIs.

I put together a rather crude script:


It helps you easily perform the following tasks:

  1. Detach an instance from the ELB
  2. Remotely xecute a command on the instance
  3. Reboot the instance
  4. Attach the instance to the ELB

It takes the instance_id and a step number as an argument. So a typical use-case would be like this:

$ ./instance-maintainer.sh i-1234abcd 1 #detach from ELB
$ ./instance-maintainer.sh i-1234abcd 2 "yum --security -y update" #perform yum update
$ ./instance-maintainer.sh i-1234abcd 3 #reboot
$ ./instance-maintainer.sh i-1234abcd 4 #attach to ELB

Since this is still a very crude and immature script, the developer must manually execute each step of the script. I wish I can automate all of this, but there are a few bumps to clear.

  1. When you run the deregister-instance-from-load-balancer, it takes a little while until there are no more requests coming in. I need to be able to be sure with that timing, otherwise I may be performing commands on step 2 that will affect incoming requests.
    Maybe I can just have the script sleep for a certain time between step 1 and 2, but the sleep time would be completely arbitrary and wouldn’t provide a 100% confidence.

  2. After rebooting the instance, there isn’t a way to know when an instance successfully rebooted. I will need to insert some script inside the instance to notify me when the reboot procedure is complete; Yeah that’s okay if you don’t mind embedding that kind of scripts into all of your instances, but for me, I want this script to be able to work without fiddling inside the target instances.
    Alternatively, I can execute

    $ ./instance-maintainer.sh i-1234abcd 2 "ps ax | grep httpd" 

    or something of that sort in a certain interval, in a sense “polling” for the correct process to be up and running. It’s kind of dirty, but can be a workaround.

So this script will be a very elementary building block for automating instance maintainance - Most likely it’d be better to write a script that performs the 1 ~ 4 of the script in whatever combination you want it to, while having various code to let the script know when to proceed to the next step.

Jan 30, 2015 - Creating my own Redis command - ZGTSCORE



As a part of my effort of trying to gain more knowledge in C-written middlewares, I have been looking into Redis for the past few days. I’ve heard that it’s relatively neatly written and easy to read, so I was giving it a try.
The core modules like networking and event-looping are still too complex for me to wrap my head around, but the implementation of commands are pretty straightforward and extremely easy to comprehend. I was surprised with how easy I was able to read and write through it, even though I suck at C.

So I’ve went into the source code and added a command, after reading through a nice tutorial here.

What I’ve Made - ZGTSCORE

I made a simple command named ZGTSCORE. It’s a command for sorted sets.
By with the command ZGTSCORE <key> <score>, you can get values of the <key> sorted set which is specifically greater than or equal to the <score>. This command will be useful when you are using a sorted set as a chronological list, with the score being the timestamp, in a use-case where you want to get data newer than a certain timestamp (checking for updates).

So here’s my work, with my branch.



1. Add command to redisCommandTable

This is the first step. Redis has all its commands neatly organized in the redisCommandTable inside src/redis.c, with flags and parameters. I added my command:


2. Function Declaration

Declare your function with the name you speficied above, in redis.h. All redis command functions only take one argument, redisClient *c. As you will see later, the command arguments and everything are all stored in there.

void zgtscoreCommand(redisClient *c);

3. Write zgtscoreCommand()

I start writing zgtscoreCommand() in src/t_zset.c, since it’s a sorted set command.
Here, rather than directly implementing this function, I’m going to call another function inside: zcompscoreGenericCommand(redisClient *c, int reverse).

This is a design pattern I was seeing in src/t_zset.c with some of the other methods - They wrap a generic function which gets data in a certain manner, and just flips the direction with the reverse argument. I might want to implement a ZLTSCORE (get data with values less than score), so I decide to copy how they do it.

void zcompscoreGenericCommand(redisClient *c, int reverse) {
/* will implement code ... */

void zgtscoreCommand(redisClient *c) {
    zcompscoreGenericCommand(c, 0);

4. Implement zcompscoreGenericCommand()

Now I start writing the actual data retreival. There’s quite a few interesting things going on, but I won’t talk about all of them. One part I had to think a little bit was how to return the reply. If you know how many lines your return data is going to be, you use addReplyMultiBulkLen(c, length) . If you don’t know beforehand, you folow this pattern:

	int rangelen, replylen;

    /* We don't know in advance how many matching elements there are in the
     * list, so we push this object that will represent the multi-bulk
     * length in the output buffer, and will "fix" it later */
    replylen = addDeferredMultiBulkLength(c);


	// add a line of reply, and increment the count


	// set the length
    setDeferredMultiBulkLength(c, replylen, rangelen);

The other interesting point was how I deal with two different data structures to store sorted sets - ziplist and skiplist. I haven’t looked into the internals of each of these implementations, but apparently ziplist is more memory efficient but cannot deal with large data, and skiplist is the opposite of that.
To deal with these two implementations, I had to write two implementations of the same algorithm, one for zip list and one for skip list.
They both have slightly different ways of retreiving data and manipulating the pointer, so it took me a while before I got comfortable with both ways. You can go to my actual code, but here is how the condition looks.

    if (zobj->encoding == REDIS_ENCODING_ZIPLIST) {
	/* write ziplist data code */
    } else if (zobj->encoding == REDIS_ENCODING_SKIPLIST) {
	/* write skiplist data code */
    } else {
        redisPanic("Unknown sorted set encoding");

Even when I read the code for ZSCORE or ZRANK or other commands they seem to be redundantly writing the same thing for both ziplist and skiplist. I wonder if there wasn’t a cleaner way to write this, or if it just wasn’t worth the effort.

4. Write tests

After I’m done with the implementation, I should add tests. Tests in Redis are written in a language called Tcl (which I haven’t heard til this day); it’s a fairly simple language and tests are written in a very easy to read way. Here’s my test code, in tests/unit/type/zset.tcl:

        test "ZGTSCORE basics" {
            r del ztmp
            r zadd ztmp 1 a
            r zadd ztmp 2 b
            r zadd ztmp 3 c
            r zadd ztmp 4 d

            assert_equal {c d} [r zgtscore ztmp 3]
            assert_equal {d} [r zgtscore ztmp 4]
            assert_equal {} [r zgtscore ztmp 5]

The r seems to represent a redis-client. Again I have not looked deeply into the inner workings of how this Tcl test suite is structured, but it sure is clean and easy to add tests.

5. make and test

After everything is done, build and test Redis with make && make test to make sure everything is working.


I was able to easily add a command - the whole thing took me less than 2 hours from seeing the code for the first time to having a successful make test. Given that I’m not at all an experienced C programmer, I was able to feel first-hand how organized the Redis source code is.

I’d love to be able to contribute to the Redis code base someday.

Jan 28, 2015 - About the GHOST Vulnerability


So the GHOST vulnerability has been a hot topic. Articles like this and this discuss what it’s about, and it’s efffects, so I won’t go into that.

Reading further, I find that articles like this highlight the point that the danger of this vulnerability is limited, since initially there can only be 4 to 8 bytes of code execution, and additional coding is necessary to further take advantage of this.

Exploit Example

There is a Proof-of-Concept demonstrating an exploit with the Exim mail server in this python script, with futher explanation here. In this example the attacker can retreive the segmentation fault error of the server remotely.

user@...ian-7-7-64b:~$ telnet 25
Connected to
Escape character is '^]'.
220 debian-7-7-64b ESMTP Exim 4.80 ...
HELO 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Connection closed by foreign host.

user@...ian-7-7-64b:~$ dmesg
[ 1715.842547] exim4[2562]: segfault at 7fabf1f0ecb8 ip 00007fabef31bd04 sp 00007fffb427d5b0 error 6 in libc-2.13.so[7fabef2a2000+182000]

This is only a Exmin specific exploit, but opens up possibilities of other softwares having similar(or worse) exploits.

Protecting yourself against it

Amazon came out with a patch already.
So any of us using Amazon Linux AMI should go ahead and patch this up by

yum clean all
yum update glibc

Check that the glibc update version is glibc-2.17-55.93.amzn1.
After that, reboot your instance for everything to take effect.

It takes some work to remove each production server from the load balancer, patch & restart it, then attach it back on to the load balancer… this took me some time and effort. I wish there is a way to do this more easily. Maybe it calls for some automation.

Oh well. Since we’ve seen Heartbleed, Shellshock, and POODLE already in this past year or two, I’m pretty sure we’ll run ino more.
And I think that is a good thing, because we all are benefiting from vulnerabilities being patched up and in result refining the software the community uses.