Handling failover with memcached as a session store?
I also just posted this together with questions to the spymemcached group:
I’m just thinking about using memcached as a session store. For handling node failures and node adds/removals the normal way is to store sessions in the database - so that memcached is used as no more than the caching layer - data is persistent and available even when no memcached node is there.
This is quite reasonable, however I want to think about an alternative: Would it be possible to store the session in two memcached nodes, so that the 2nd node is chosen if the primary node is not there (or the session is not found in this node)? One could store the session not only in the primary node but additionally in the “next” node. Together with consistent hashing both node failures and adding new nodes should be handled well.
Making an example
Let’s look at an example to see if this seems to work. Asume you have 3 nodes, with hashes 1, 2 and 3 (just for simplicity in this example).
Then we put a value that consistently hashes to “3″. In this case the data is stored to the nodes 3 and 1.
Another case would be a value that consistently hashes to 4: this data is stored to nodes 1 and 2.
Just trying to put this in a matrix, with node hashes on the x-axis and the key hash on the y-axis:
| Hash/Nodes | Node 1 | Node 2 | Node 3 |
| 3 | X | X | |
| 4 | X | X |
Then lets look at two cases: a node failure/removal, and adding a node.
Node failure
Let’s look at what happens when e.g. node 3 or node 1 fails.
1. Node 3 fails
If a key hashes to 3, it is read from node 3 (not available) and node 1 (success).
If a key hashes to 4, it is read from node 1 (success).
2. Node 1 fails
If a key hashes to 3, it is read from node 3 (success).
If a key hashes to 4, it is read from node 1 (failure) and node 2 (success).
Adding a node
So what happens when a new node 4 (hashing to 4) is added?
If a key hashes to 3, it is still read from node 3 (success).
If a key hashes to 4, it is read from node 4 (failure) and then from node 1 (success).
These example show, that simple node failures and adds are fine. Of course only 1 node at a time can fail or can be added. If more than one node would be added or removed, previously stored data would be unavailable. Perhaps it would really be better to store data in the database…?
