Writing comments in your code is fine and dandy but sometimes its just a fucking waste of time. Here’s an example:

<?php
class UserDataAccessClass
{
    /**
     * Get the number of followers a user has
     *
     * @param string $userID The user id
     *
     * @return int
     */
    public function getNumFollowers($userID)
    {
        return valueFromAServiceCallOrQueryEtc($userID)
    }
}
?>

So, yay – that chunk of code passes PHPCS (using the PEAR standard). All the parameters are documented, there’s a line of text explaining what the function does and the docblock even states the return type… how cute!

But why? Why do I have to type all that crap? The function’s name is self documenting & its sole parameter is obvious. The return type makes sense to me but the rest is bullshit. God forbid your function takes multiple parameters; then you’d have to line up the @param‘s types and descriptions ’cause PHP people have a strange hardon for lining shit up.

The truth is the only reason I do all that stuff is cuz we run PHPCS on our code at work and I don’t wanna be “that guy”. If it wasn’t a “standrad” at work there’s no way in hell id ever bother.

I’d much have the documentation go like this instead:

<?php
class UserDataAccessClass
{
    /**
     * @return int
     */
    public function getNumFollowers(string $userID)
    {
        return valueFromAServiceCallOrQueryEtc($userID)
    }
}
?>

…And to be 100% honest the only reason I’d include the @return is ’cause I’m an Eclipse & it helps PDT’s static analysis (aka autocomplete gets more better).

I’ve been playing with #neo4j quite a bit lately. It’s a great & fun project. It’s a graph database that mitigates all the bullshit you have to deal with when trying to, ya know, do graph stuff. Example: find all User Nodes who’s gender property is set to female, have an outgoing likes relationship to the Node punk music and are less than 3 degrees of separation from Node #4. Stuff like that. Its super good at doing this.

But here’s the deal… each Node is gettable via ID which is nice – but the ID’s are Neo4J’s internal ID; you don’t get to set ‘em when you create a Node. So, what if I want to get a Node who’s username property is phatduckk & start the traversal from there? The problem lies in the fact that you don’t know that phatduckk is Node #4 so you need a simple & efficient way to do that lookup & grab that Node.

If your dataset is small, I guess, you can just use a Map and store the mapping yourself but that solution will fall over pretty quickly. You could also toss info into MySQL but why would you do that? It just doesn’t feel right to use 2 different stores. So, checking out some of the docs you’ll see that Neo4J’s got some indexing capabilities.

Initially I tried out the SingleValueIndex which fell over in a multi-threaded scenario. So, I hit up the list and was advised to check out the LuceneIndexService. This worked like a charm. Even with multiple threads constantly indexing the same Node.

Here’s a little test app. It’s a brute force, little hack that creates a single Node and indexes it by its username property 100,000 times using 10 threads. This is a pretty unrealistic situation but I really wanted to make sure it behaved well in a multi-threaded scenario and didn’t frustrate me like the SingleValueIndex did.

package com.digg.tmp;

import org.neo4j.api.core.*;
import org.neo4j.util.index.IndexService;
import org.neo4j.util.index.LuceneIndexService;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class LuceneIndex {
    private static final String USERNAME_INDEX = "usernameIndex";
    private static final int NUM_THREADS = 10;
    private static final int NUM_LINES = 1000000;
    private static final String USERNAME = "phatduckk";

    public static void main(String[] args) {
        // always use a new store
        NeoService neo = new EmbeddedNeo("test-" + System.currentTimeMillis());

        // now create the node we want indexed:
        Transaction txUser = neo.beginTx();
        Node userNode = neo.createNode();
        userNode.setProperty(USERNAME_INDEX, USERNAME);
        txUser.success();
        txUser.finish();

        // now create the index & setup a pool
        IndexService idxServ = new LuceneIndexService(neo);
        final ExecutorService executorService = Executors.newFixedThreadPool(NUM_THREADS);

        // now let's index that same node NUM_LINES times
        // the reason we're indexing the same node is cuz i'm checking for thread safety during indexing issues
        // otherwise you'd normally be indexing new nodes who's data you got from some external source
        for (int i = 0; i < NUM_LINES; i++) {
            System.out.println("line: " + i);
            IndexRunner command = new IndexRunner(userNode, neo, idxServ);
            executorService.execute(command);
        }

        // should do a clean neo.shutdown() at some point ;-)
    }

    static class IndexRunner implements Runnable {
        NeoService neo;
        IndexService idxServ;
        Node userNode;

        IndexRunner(Node userNode, NeoService neo, IndexService idxServ) {
            this.userNode = userNode;
            this.neo = neo;
            this.idxServ = idxServ;
        }

        public void run() {
            Transaction nodetx = neo.beginTx();
            Node nodeFromIndex = idxServ.getSingleNode(USERNAME_INDEX, USERNAME);
            
            if (nodeFromIndex != null) {
                System.out.println("found " + USERNAME + " in the " + USERNAME_INDEX
                        + " index. Node ID is: " + nodeFromIndex.getId());
            } else {
                idxServ.index(userNode, USERNAME_INDEX, USERNAME);
            }
            
            nodetx.success();
            nodetx.finish();
        }
    }
}

Although this is an off the wall example it can also serve as a simple example of how to index a Node. Anywho – hope this helps out a few folks that ran into the same needs/problems/scenarios I did. In hindsight it’s all pretty simple & straightforward – I just went down the wrong path with the SingleValueIndex… when browsing the docs it sounded like the right tool for the job but, from what I can tell, you should avoid it and use the LuceneService instead.

For the last month or two the Digg engineering team has spent quite a bit of time looking into, playing with and finally deploying Cassandra in production. It’s been a super fun project to take on – but even before the fun began we had to spend quite a bit of time figuring out Cassandra’s data model… the phrase “WTF is a ‘super column’” was uttered quite a few times.

If you’re coming from an RDBMS background (which is almost everyone) you’ll probably trip over some of the naming conventions while learning about Cassandra’s data model. It took me and my team members at Digg a couple days of talking things out before we “got it”. In recent weeks a bikeshed went down in the dev mailing list proposing a completely new naming scheme to alleviate some of the confusion. Throughout this discussion I kept thinking: “maybe if there were some decent examples out there people wouldn’t get so confused by the naming.” So, this is my stab at explaining Cassandra’s data model; It’s intended to help you get your feet wet & doesn’t go into every single detail but, hopefully, it helps clarify a few things.

BTW: this is long. If you’d rather have a PDF version of this you can download it here.

The Pieces

Let’s first go thru the building blocks before we see how they can all be stuck together:

Column

The column is the lowest/smallest increment of data. It’s a tuple (triplet) that contains a name, a value and a timestamp.

Here’s a column represented in JSON-ish notation:

{  // this is a column
    name: "emailAddress",
    value: "arin@example.com",
    timestamp: 123456789
}

That’s all it is. For simplicity sake let’s ignore the timestamp. Just think of it as a name/value pair.

Also, it’s worth noting is that the name and value are both binary (technically byte[]) and can be of any length.

SuperColumn

A SuperColumn is a tuple w/ a binary name & a value which is a map containing an unbounded number of Columns – keyed by the Column‘s name. Keeping with the JSON-ish notation we get:

{   // this is a SuperColumn
    name: "homeAddress",
    // with an infinite list of Columns
    value: {
        // note the keys is the name of the Column
        street: {name: "street", value: "1234 x street", timestamp: 123456789},
        city: {name: "city", value: "san francisco", timestamp: 123456789},
        zip: {name: "zip", value: "94107", timestamp: 123456789},
    }
}

Column vs SuperColumn

Columns and SuperColumns are both a tuples w/ a name & value. The key difference is that a standard Column‘s value is a “string” and in a SuperColumn the value is a Map of Columns. That’s the main difference… their values contain different types of data. Another minor difference is that SuperColumn‘s don’t have a timestamp component to them.

Before We Get Rolling

Before I move on I wanna simplify our notation a couple ways: 1) ditch the timestamps from Columns & 2) pull the Columns’ & SuperColumns’ names component out so that it looks like a key/value pair. So we’re gonna go from:

{ // this is a super column
    name: "homeAddress",
    // with an infinite list of columns
    value: {
        street: {name: "street", value: "1234 x street", timestamp: 123456789},
        city: {name: "city", value: "san francisco", timestamp: 123456789},
        zip: {name: "zip", value: "94107", timestamp: 123456789},
    }
}

to

homeAddress: {
    street: "1234 x street",
    city: "san francisco",
    zip: "94107",
}

Grouping ‘Em

There’s a single structure used to group both the Columns and SuperColumns…this structure is called a ColumnFamily and comes in 2 varieties Standard & Super.

ColumnFamily

A ColumnFamily is a structure that contains an infinite number of Rows. Huh, did you say Rows? Ya – rows To make it sit easier in your head just think of it as a table in an RDBMS.

OK – each Row has a client supplied (that means you) key & contains a map of Columns. Again, the keys in the map are the names of the Columns and the values are the Columns themselves:

UserProfile = { // this is a ColumnFamily
    phatduckk: {   // this is the key to this Row inside the CF
        // now we have an infinite # of columns in this row
        username: "phatduckk",
        email: "phatduckk@example.com",
        phone: "(900) 976-6666"
    }, // end row
    ieure: {   // this is the key to another row in the CF
        // now we have another infinite # of columns in this row
        username: "ieure",
        email: "ieure@example.com",
        phone: "(888) 555-1212"
        age: "66",
        gender: "undecided"
    },
}

Remember: for simplicity we're only showing the value of the Column but in reality the values in the
map are the entire Column.

You can think of it as a HashMap/dictionary or associative array. If you start thinking that way then you’re are the right track.

One thing I want to point out is that there’s no schema enforced at this level. The Rows do not have a predefined list of Columns that they contain. In our example above you see that the row with the key “ieure” has Columns with names “age” and “gender” whereas the row identified by the key “phatduckk” doesn’t. It’s 100% flexible: one Row may have 1,989 Columns whereas the other has 2. One Row may have a Column called “foo” whereas none of the rest do. This is the schemaless aspect of Cassandra.

A ColumnFamily Can Be Super Too

Now, a ColumnFamily can be of type Standard or Super.

What we just went over was an example of the Standard type. What makes it Standard is the fact that all the Rows contains a map of normal (aka not-Super) Columns… there’s no SuperColumns scattered about.

When a ColumnFamily is of type Super we have the opposite: each Row contains a map of SuperColumns. The map is keyed with the name of each SuperColumn and the value is the SuperColumn itself. And, just to be clear, since this ColumnFamily is of type Super, there are no Standard ColumnFamily‘s in there. Here’s an example:

AddressBook = { // this is a ColumnFamily of type Super
    phatduckk: {    // this is the key to this row inside the Super CF
        // the key here is the name of the owner of the address book

        // now we have an infinite # of super columns in this row
        // the keys inside the row are the names for the SuperColumns
        // each of these SuperColumns is an address book entry
        friend1: {street: "8th street", zip: "90210", city: "Beverley Hills", state: "CA"},

        // this is the address book entry for John in phatduckk's address book
        John: {street: "Howard street", zip: "94404", city: "FC", state: "CA"},
        Kim: {street: "X street", zip: "87876", city: "Balls", state: "VA"},
        Tod: {street: "Jerry street", zip: "54556", city: "Cartoon", state: "CO"},
        Bob: {street: "Q Blvd", zip: "24252", city: "Nowhere", state: "MN"},
        ...
        // we can have an infinite # of ScuperColumns (aka address book entries)
    }, // end row
    ieure: {     // this is the key to another row in the Super CF
        // all the address book entries for ieure
        joey: {street: "A ave", zip: "55485", city: "Hell", state: "NV"},
        William: {street: "Armpit Dr", zip: "93301", city: "Bakersfield", state: "CA"},
    },
}

Keyspace

A Keyspace is the outer most grouping of your data. All your ColumnFamily‘s go inside a Keyspace. Your Keyspace will probably named after your application.

Now, a Keyspace can have multiple ColumnFamily‘s but that doesn’t mean there’s an imposed relationship between them. For example: they’re not like tables in MySQL… you can’t join them. Also, just because ColumnFamily_1 has a Row with key “phatduckk” that doesn’t mean ColumnFamily_2 has one too.

Sorting

OK – we’ve gone through what the various data containers are about but another key component of the data model is how the data is sorted. Cassandra is not queryable like SQL – you do not specify how you want the data sorted when you’re fetching it (among other differences). The data is sorted as soon as you put it into the cluster and it always remains sorted! This is a tremendous performance boost for reads but in exchange for that benefit you’re going to have to make sure to plan your data model in a such a way that you’re able satisfy your access patterns.

Columns are always sorted within their Row by the Column‘s name. This is important so i’ll say it again: Columns are always sorted by their name! How the names are compared depends on the ColumnFamilys CompareWith option. Out of the box you have the following options: BytesType, UTF8Type, LexicalUUIDType, TimeUUIDType, AsciiType, and LongType. Each of these options treats the Columns’ name as a different data type giving you quite a bit of felxibility. For example: Using LongType will treat your Columns’ names as a 64bit Longs. Let’s try and clear this up by taking a look at some data before and after it’s sorted:

    // Here's a view of all the Columns from a particular Row in random order
    // Cassandra would "never" store data in random order. This is just an example
    // Also, ignore the values - they don't matter for sorting at all
    {name: 123, value: "hello there"},
    {name: 832416, value: "kjjkbcjkcbbd"},
    {name: 3, value: "101010101010"},
    {name: 976, value: "kjjkbcjkcbbd"}

So, given the fact that we’re using the LongType option, these Columns will look like this when they’re sorted:

    <!--
    ColumnFamily definition from storage-conf.xml
    -->
    <ColumnFamily CompareWith="LongType" Name="CF_NAME_HERE"/>

    // See, each Column's name is treated as a 64bit long
    // in effect, numerically ordering our Columns' by name
    {name: 3, value: "101010101010"},
    {name: 123, value: "hello there"},
    {name: 976, value: "kjjkbcjkcbbd"},
    {name: 832416, value: "kjjkbcjkcbbd"}

As you can see the Columns’ names were compared as if they were 64bit Longs (aka: numbers that can get pretty big). Now, if we’d used another CompareWith option we’d end up with a different result. If we’d set CompareWith to UTF8Type our sorted Columns’ names would be treated as a UTF8 encoded strings yielding a sort order like this:

    <!--
    ColumnFamily definition from storage-conf.xml
    -->
    <ColumnFamily CompareWith="UTF8Type" Name="CF_NAME_HERE"/>

    // Each Column name is treated as a UTF8 string
    {name: 123, value: "hello there"},
    {name: 3, value: "101010101010"},
    {name: 832416, value: "kjjkbcjkcbbd"},
    {name: 976, value: "kjjkbcjkcbbd"}

The result is completely different!

This sorting principle applies to SuperColumns as well but we get an extra dimension to deal with: not only do we determine how the SuperColumns are sorted in a Row but we also determine how the Columns within each SuperColumn are sorted. The sort of the Columns within each SuperColumn is determined by the value of CompareSubcolumnsWith. Here’s an example:

    // Here's a view of a Row that has 2 SuperColumns in it.
    // currently they're in some random order

    { // first SuperColumn from a Row
        name: "workAddress",
        // and the columns within it
        value: {
            street: {name: "street", value: "1234 x street"},
            city: {name: "city", value: "san francisco"},
            zip: {name: "zip", value: "94107"}
        }
    },
    { // another SuperColumn from same Row
        name: "homeAddress",
        // and the columns within it
        value: {
            street: {name: "street", value: "1234 x street"},
            city: {name: "city", value: "san francisco"},
            zip: {name: "zip", value: "94107"}
        }
    }

Now if we decided to set both CompareSubcolumnsWith & CompareWith to UTF8Type we’d have the following end result:

    // Now they're sorted

    {
        // this one's first b/c when treated as UTF8 strings
        { // another SuperColumn from same Row

            // This Row comes first b/c "homeAddress" is before "workAddress"
            name: "homeAddress",

            // the columns within this SC are also sorted by their names too
            value: {
                // see, these are sorted by Column name too
                city: {name: "city", value: "san francisco"},
                street: {name: "street", value: "1234 x street"},
                zip: {name: "zip", value: "94107"}
            }
        },
        name: "workAddress",
        value: {
            // the columns within this SC are also sorted by their names too
            city: {name: "city", value: "san francisco"},
            street: {name: "street", value: "1234 x street"},
            zip: {name: "zip", value: "94107"}
        }
    }

I want to note that in the last example CompareSubcolumnsWith & CompareWith were set to UTF8Type but this doesn’t have to be the case. You can mix and match the values of CompareSubcolumnsWith & CompareWith as necessary.

The last bit about sorting I want to mention is that you can write a custom class to perform the sorting. The sorting mechanism is pluggable… you can set CompareSubcolumnsWith and/or CompareWith to any fully-qualified class name as long as that class implements org.apache.cassandra.db.marshal.IType (aka you can write custom comparators).

Example Schema

Alrighty – Now we’ve got all the pieces of the puzzle so let’s finally put ‘em all together and model a simple blog application. We’re going to model a simple app with the following specs:

• support a single blog
• we can have multiple authors
• entries contain title, body, slug & publish date
• entries can be associated with any # of tags
• people can leave comments but cant register: they enter profile info each time (just keeping it simple)
• comments have text, time submitted, commenter’s name & commenter’s name
• must be able to show all posts in reverse chronological order (newest first)
• must be able to show all posts within a given tag in reverse chronological order

Each of the following sections will describe a ColumnFamily that we’re going to define in our app’s Keyspace, show the xml definition, talk about why we picked the particular sort option(s) as well as display the data in the ColumnFamily w/ our JSON-ish notation.

Authors ColumnFamily

Modeling the Authors ColumnFamily is going to be pretty basic; we’re not going to do anything fancy here. We’re going to give each Author their own Row & key it by the Author‘s full name. Inside the Rows each Column is going to represent a single “profile” attribute for the Author.

This is an example of using each Row to represent an object… in this case an Author object. With this approach each Column will serve as an attribute. Super simple. I want to point out that since there’s no “definition” of what Columns must be present within a Row we kinda sorta have a schemaless design.

We’ll be accessing the Rows in this ColumnFamily via key lookup & will grab every Column with each get (ex: we won’t ever be fetching the first 3 columns from the Row with key ‘foo’). This means that we don’t care how the Columns are sorted so we’ll use BytesType sort options because it doesn’t require any validation of the Columns’ names.

<!--
    ColumnFamily: Authors
    We'll store all the author data here.

    Row Key => Author's name (implies names must be unique)
    Column Name: an attribute for the entry (title, body, etc)
    Column Value: value of the associated attribute

    Access: get author by name (aka grab all columns from a specific Row)

    Authors : { // CF
        Arin Sarkissian : { // row key
            // and the columns as "profile" attributes
            numPosts: 11,
            twitter: phatduckk,
            email: arin@example.com,
            bio: "bla bla bla"
        },
        // and the other authors
        Author 2 {
            ...
        }
    }
-->
<ColumnFamily CompareWith="BytesType" Name="Authors"/>

BlogEntries ColumnFamily

Again, this ColumnFamily is going to act as a simple key/value lookup. We’ll be storing 1 entry per Row. Within that Row the Columns will just serve as attributes of the entry: title, body, etc (just like the previous example). As a small optimization we’ll denormalize the tags into a Column as a comma separated string. Upon display we’ll just split that Column‘s value to get a list of tags.

The key to each Row will be the entries slug. So whenever we want to grab a single entry we can simply look it up by its key (slug).

<!--
    ColumnFamily: BlogEntries
    This is where all the blog entries will go:

    Row Key +> post's slug (the seo friendly portion of the uri)
    Column Name: an attribute for the entry (title, body, etc)
    Column Value: value of the associated attribute

    Access: grab an entry by slug (always fetch all Columns for Row)

    fyi: tags is a denormalization... its a comma separated list of tags.
    im not using json in order to not interfere with our
    notation but obviously you could use anything as long as your app
    knows how to deal w/ it

    BlogEntries : { // CF
        i-got-a-new-guitar : { // row key - the unique "slug" of the entry.
            title: This is a blog entry about my new, awesome guitar,
            body: this is a cool entry. etc etc yada yada
            author: Arin Sarkissian  // a row key into the Authors CF
            tags: life,guitar,music  // comma sep list of tags (basic denormalization)
            pubDate: 1250558004      // unixtime for publish date
            slug: i-got-a-new-guitar
        },
        // all other entries
        another-cool-guitar : {
            ...
            tags: guitar,
            slug: another-cool-guitar
        },
        scream-is-the-best-movie-ever : {
            ...
            tags: movie,horror,
            slug: scream-is-the-best-movie-ever
        }
    }
-->
<ColumnFamily CompareWith="BytesType" Name="BlogEntries"/>

TaggedPosts ColumnFamily

Alright – here’s where things get a bit interesting. This ColumnFamily is going to do some heavy lifting for us. It’s going to be responsible for keeping our tag/entry associations. Not only is it going to store the associations but it’s going to allow us to fetch all BlogEntrys for a certain tag in pre-sorted order (remember all that sorting jazz we went thru?).

A design point I want to point out is that we’re going have our app logic tag every BlogEntry with the tag “__notag__” (a tag I just made up). Tagging every BlogEntry with “__notag__” will allow us to use this ColumnFamily to also store a list of all BlogEntrys in pre-sorted order. We’re kinda cheating but it allows us to use a single ColumnFamily to serve “show me all recent posts” and “show me all recent posts tagged ‘foo’”.

Given this data model if an entry has 3 tags it will have a corresponding Column in 4 Rows… 1 for each tag and one for the “__notag__” tag.

Since we’re going to want to display lists of entries in chronological order we’ll make sure each Columns name is a time UUID and set the ColumnFamilys CompareWith to TimeUUIDType. This will sort the Columns by time satisfying our “chronological order” requirement So doing stuff like “get the latest 10 entries tagged ‘foo’” is going to be a super efficient operation.

Now when we want display the 10 most recent entries (on the front page, for example) we would:

1. grab the last 10 Columns in the Row w/ key “__notag__” (our “all posts” tag)
2. loop thru that set of Columns
3. while looping, we know the value of each Column is the key to a Row in the BlogEntries ColumnFamily
4. so we go ahead and use that to grab the Row for this entry from the BlogEntries ColumNFamily. this gives us all the data for this entry
5. one of the Columns from the BlogEntries Row we just grabbed is named “author” and the value is the key into the Authors ColumnFamily we need to use to grab that author’s profile data.
6. at this point we’ve got the entry data and the author data on hand
7. next we’ll split the “tags” Columns value to get a list tags
8. now we have everything we need to display this post (no comments yet – this aint the permalink page)

We can go through the same procedure above using any tag… so it works for “all entries” and “entries tagged ‘foo’”. Kinda nice.

<!--
    ColumnFamily: TaggedPosts
    A secondary index to determine which BlogEntries are associated with a tag

    Row Key => tag
    Column Names: a TimeUUIDType
    Column Value: row key into BlogEntries CF

    Access: get a slice of entries tagged 'foo'

    We're gonna use this CF to determine which blog entries to show for a tag page.
    We'll be a bit ghetto and use the string __notag__ to mean
    "don't restrict by tag". Each entry will get a column in here...
     this means we'll have to have #tags + 1 columns for each post.

    TaggedPosts : { // CF
        // blog entries tagged "guitar"
        guitar : {  // Row key is the tag name
            // column names are TimeUUIDType, value is the row key into BlogEntries
            timeuuid_1 : i-got-a-new-guitar,
            timeuuid_2 : another-cool-guitar,
        },
        // here's all blog entries
        __notag__ : {
            timeuuid_1b : i-got-a-new-guitar,

            // notice this is in the guitar Row as well
            timeuuid_2b : another-cool-guitar,

            // and this is in the movie Row as well
            timeuuid_2b : scream-is-the-best-movie-ever,
        },
        // blog entries tagged "movie"
        movie: {
            timeuuid_1c: scream-is-the-best-movie-ever
        }
    }
-->
<ColumnFamily CompareWith="TimeUUIDType" Name="TaggedPosts"/>

Comments ColumnFamily

The last thing we need to do is figure out how to model the comments. Here we’ll get to bust out some SuperColumns.

We’ll have 1 Row per entry. The key to the Row will be the entries slug. Within each Row we’ll have a SuperColumn for each comment. The name of the SuperColumns will be a UUID that we’ll be applying the TimeUUIDType to. This will ensure that all our comments for an entry are sorted in chronological order. The Columns within each SuperColumn will be the various attributes of the comment (commenter’s name, comment time etc).

So, this is pretty simple as well… nothing fancy.

<!--
    ColumnFamily: Comments
    We store all comments here

    Row key => row key of the BlogEntry
    SuperColumn name: TimeUUIDType

    Access: get all comments for an entry

    Comments : {
        // comments for scream-is-the-best-movie-ever
        scream-is-the-best-movie-ever : { // row key = row key of BlogEntry
            // oldest comment first
            timeuuid_1 : { // SC Name
                // all Columns in the SC are attribute of the comment
                commenter: Joe Blow,
                email: joeb@example.com,
                comment: you're a dumb douche, the godfather is the best movie ever
                commentTime: 1250438004
            },

            ... more comments for scream-is-the-best-movie-ever

            // newest comment last
            timeuuid_2 : {
                commenter: Some Dude,
                email: sd@example.com,
                comment: be nice Joe Blow this isnt youtube
                commentTime: 1250557004
            },
        },

        // comments for i-got-a-new-guitar
        i-got-a-new-guitar : {
            timeuuid_1 : { // SC Name
                // all Columns in the SC are attribute of the comment
                commenter: Johnny Guitar,
                email: guitardude@example.com,
                comment: nice axe dawg...
                commentTime: 1250438004
            },
        }

        ..
        // then more Super CF's for the other entries
    }
-->
<ColumnFamily CompareWith="TimeUUIDType" ColumnType="Super"
    CompareSubcolumnsWith="BytesType" Name="Comments"/>

Woot!

That’s it. Out little blog app is all modeled and ready to go. It’s quite a bit to digest but in the end you end up with a pretty small chunk of XML you’ve gotta store in the storage-conf.xml:

    <Keyspace Name="BloggyAppy">
        <!-- other keyspace config stuff -->

        <!-- CF definitions -->
        <ColumnFamily CompareWith="BytesType" Name="Authors"/>
        <ColumnFamily CompareWith="BytesType" Name="BlogEntries"/>
        <ColumnFamily CompareWith="TimeUUIDType" Name="TaggedPosts"/>
        <ColumnFamily CompareWith="TimeUUIDType" Name="Comments"
            CompareSubcolumnsWith="BytesType" ColumnType="Super"/>
    </Keyspace>

Now all you need to do is figure out how to get the data in and out of Cassandra . That’s all accomplished via the Thrift Interface. The API wiki page does a decent job at explaining what the various endpoints do so I won’t go into all those details. But, in general, you just compile the cassandra.thrift file and use the generated code to access the various endpoints. Alternatively you can take advantage of this Ruby client or this Python client.

Alrighty… hopefully all that made sense & you finally understand WTF a SuperColumn is and can start building some awesome apps.

OK – so last time I wrote about serialization and how json_encode()
and other “external” serializers do not include any private members in the serialized
string they return. Pretty simple but useful. This time I want to talk about transient data; data that shouldn’t be there.

The “problem” is that in PHP you can arbitrarily add new members/fields to an object. There’s nothing stopping you from doing something like
$user->someRandomAssField = $someOtherVariableEvenAnArrayOrObject. These new, on-the-fly members get assigned public scope… so, for example, json_encode() will include that data in the serialized string (so will XML_Serializer, serialize etc). Here’s an example:

The output looks like this:

Whoops – we’ve got shit that we don’t want/need. There’s a ton of reason why this could happen; Sometimes you want this data for internal use (tho I dunno why you wouldn’t make it a proper member of the class) but for the most part its stray shit that got tacked onto an object. Really it just comes down to control. As the author of the API it’s your job to determine what info is exposed to the public & unfortunately (in this case) PHP makes it way too easy for this to happen: no warnings, nothing. So how do we get rid of it?

Getting rid of this stuff’s pretty easy using PHP’s magic methods. This is one of the only cases where I actually like using the magic methods as a solution… usually I consider them evil magic. Read up on magic methods if you dunno what they are then check out the remainder of this post…

So let’s modify the User class above and add some magic so that these transient fields aren’t serialized/encoded via json_encode() (and XML_Serializer etc).

First thing is to add a private dictionary to the class. “Dictionary”, “hash”, “associative array”… whatever you wanna call it… its a key/value lookup.

So that’s simple… we’ve got a new member which is private (protected would work fine too). Now it’s time to sprinkle in some magic and see where that gets us:

You can see from the var_dump that the transient data (aka “random shit”) is all stored in
the private $dictionary member. And, right after that you see a json_encode()‘ed
version of the same instance of a User… and, woot, the transient data ain’t there. We got rid of it with some PHP magic. Kinda cool huh?

One random thing I like about this solution is that most uses of magic methods (that I’ve seen) are to get data or to proxy requests to a child element; It’s kinda cool to use __get and __set in a pseudo-backwards way: as a means to getting rid of data.

I should note that I’ve only used json_encode() in the example above but the same would apply to XML_Serializer.

Anywho… that’s it. Again, pretty easy stuff. Next time I’ll either write about sets of data (which I lied and said I’d do this time) or something else… um, maybe the data access layer.

I recently started on a new side project. It’s not web 2.0 sexy or anything like that but it could be pretty cool. Anyways… I started off using the framework I built for Blip.fm and refactored that and done a few tweaks. I’m kinda happy with the data access layer so I thought I’d talk about that for a bit:

The core of this data layer is what I call an SRD. SRD stands for “service request definition”. The goal here is remove tight coupling of my data layer from MySQL (or any backend) so I coded up this SRD concept.

An SRD itself is an array. You can write ‘em up as array’s in PHP, XML or even ini files but in the end they need to be unserialized into an array that looks kinda like this:

So that’s 2 different SRD‘s: One for getting a User via login credentials and another for POSTing a new User object.

Coneceptually it goes something like this:

1. instanciate a new Duckk_Service_Request and give it the SRD key
2. call execute on the Duckk_Service_Request you just instanciated with an associative array ($args)
3. execute will do the rest and return stuff to you

Here’s an example of using the SRDs listed above:

So when you instanciate a new Duckk_Service_Request I check for the SRD key in APC… if not I make sure that the key is valid and that the SRD looks right and throw Exceptions on various error conditions etc & finally cache it. The main stuff happens when execute is called.

Inside execute a few things happen before we even try to hit the data-store. The first thing I do is sanitize the $args. The main purpose here is to ditch the “extra” args… a user can pass in way more $args than the SRD‘s query needs – so I ditch the extras (I’ll explain why later).

The next step is to examine the SRD‘s value for Duckk_SRD::KEY_TRANSPORT and figure out what type of data transport we’re going to use & instanciate an instance of it. The transports implement an interface called Duckk_Data_Transport. The transport used in the examples above is MySQL – so for this example we instanciate a Duckk_Data_Transport_Mysql (via a factory), pass it the value from the SRD‘s Duckk_SRD::KEY_QUERY key and finally call execute on that transport w/ the slimmed down $args passed in. All in all you end up with something like:

You see that it’s the transport’s job to know what to do with the value in SRD‘s
Duckk_SRD::KEY_QUERY key. So, your queries can be 100% arbitray… as long as your
Duckk_Data_Transport class knows how to parse it and make some sense of it. Also the
Duckk_Data_Transports constructor takes the SRD
array as an argument – so you can add more keys to help you transport figure stuff out.

In this example my Duckk_Data_Transport_Mysql class know what’s going on because, well, the query is almost SQL. So it parses this “almost” SQL to figure out what it’s going to do: select, update, insert, replace etc. But, before it does anything like that it takes the value from the SRD‘s
Duckk_SRD::KEY_QUERY and the passed in $args to generate the “real” SQL that it’s gonna have to
run. My MySQL transport layer takes all the $args, escapes their values and all that good stuff. It’s also in charge or getting a DB handle (it does this first actually). Currently I let it use the default DBPool but I can optionally override the pool in the SRD and it will pull from there instead.

So when all is said and done the Duckk_Data_Transport_Mysql ends up with some result: For
SELECTs we get an array of rows (an array of arrays), # affected rows for
UPDATE & DELETE etc etc but it doesn’t return the result quite yet. Instead, it instanciates a Duckk_Service_Response with the result and returns that.

The Duckk_Service_Response is a simple, Countable object.
It only has a few properties: $data, $messages, $errors & $paginationInfo. So having all the Duckk_Data_Transport classes’ execute method return a Duckk_Service_Response makes sure that our data is consistent no matter what
kind of backend they use.

Finally the response from the transport is available in Duckk_Service_Request::execute
as an Duckk_Service_Response object. So we futz with this data one last time before we return it for consumption:

We check for any errors in the $response then examine the value of $response->getData(): if it’s a primitive (int, bool etc) we just return it. If it’s an array we then have to check out what to do with this data… we don’t wanna play with array‘s of array‘s… so we take a look at what data type the SRD says it wants via the Duckk_SRD::KEY_DATA_TYPE key. We mow thru $response->getData() and create Model objects of type Duckk_SRD::KEY_DATA_TYPE and inject the row values in as class variables.

So, we now have an array of Model objects. The next thing we do is check out what the value in the SRD is for Duckk_SRD::KEY_IS_SINGLE. If the value is true we pop off the first Model and return it.

If Duckk_SRD::KEY_IS_SINGLE is false or isn’t set we assume you want more than one record… but we don’t just return it as an array. I wrap it in a Duckk_Collection instead. This is essentially a wrapper over the whole thing to add some convenience. Duckk_Collection implements IteratorAggregate and Countable. It also has “pagination know-how”. So we take $response->getData() and $response->getPaginationInfo() construct a Duckk_Collection and finally return that.

The Duckk_Collection has some extra convenience methods like last(), isEven(), isNth() etc to make iteration operations easier (like setting the background color to blue for every 4th DIV while rendering templates etc).

The final bit is caching… The Duck_Service_Request can be cache aware… that’s why I sanitize (trim) the $args. The key based SRD lookup along with the necessary $args can be pretty easily combined for a simple cache key. I haven’t done this yet cuz I’m lazy – but based this predictable cache key I can do cache lookups at the top of Duck_Service_Request::execute and cache inserts before Duck_Service_Request returns. Since the SRD is just an array I can add more stuff in there for caching: ttl, other cache keys to expire etc.

These can be dynamic too:

So, with some additions to the SRD and some pretty trivial code additions to Duckk_Service_Request I can make it pretty simple to invalidate a bunch of cache keys in multiple caches upon POSTing a new record. Notice the Users_$countryCode entry in expireKeys. Since we have a bunch of vairables in $args this would be totally possible and easy!

The opposite is true for getting a record as well… just do the opposite:

We can check the cache first, then go on with all real work if there’s a miss – and cache the data when we get a response from the transport.

Obviously another thing that’s doable here is to create a bunch of new Duckk_Transport classes. Example: Duckk_Transport_Twitter. As long as this new class implements Duckk_Transport the rest of the stuff will work as well. The “apps” engineers use will never have to change their code. Just write the Duckk_Transport_Twitter class and then change/update/create SRDs and you’re done.

An example of changing login to use Twitter’s API instead of MySQL would be:

So just code up Duckk_Transport_Twitter to use deal with the $args, escape them with urlencode, run curl & parse some JSON and you’d essentially turn over your site’s authentication to Twitter.

So, that’s it… my new data access layer. I’m gonna go code up the cache stuff I mentioned above since it’s so simple.

Woot