[flud-devel] DHT justifications (was: DHT Performance? Design?)

[zooko]

>   flud recently switched
> to zfec for erasure coding (created by zooko for Tahoe).  Its
> performance is quite satisfactory compared to rizzo, but of course has
> to pay the python wrapper penalty.  You can find benchmarks in the
> expected places

For the record, I started zfec by copying Rizzo's code.  I tried to  
simplify and optimize his code, but I never did solid benchmarks of  
the result, or if I did I don't remember and I no longer have my notes.

Also, you might be interested in this ticket:

http://allmydata.org/trac/tahoe/ticket/167 -- try out Jerasure

"""
Professor James Plank has published a new library that has  
implementations of various erasure coding algorithms, Jerasure:

http://www.cs.utk.edu/~plank/plank/papers/CS-07-603.html

It would be fun to benchmark these against the current zfec

http://pypi.python.org/pypi/zfec

and if some algorithm in there is faster, switch the internals of  
zfec to use it instead.
"""

I benchmarked an earlier library of Professor Plank's (with the help  
of someone named Alen, as it happens) for Mnet back in the day, and  
it was slower than Rizzo for the parameters that I was interested in  
at the time.  Maybe the new one is faster.  Also the parameters that  
I am interested in has changed...

By the way, I've enjoyed following this discussion.  Tahoe currently  
doesn't have a DHT implementation at all.  There is a DHT design note  
-- http://allmydata.org/trac/tahoe/browser/docs/denver.txt -- but we  
currently aren't attempting to scale up the number of active nodes.   
We're currently focussed on the "friendnet" and "proprietary grid"  
use cases -- http://allmydata.org/trac/tahoe/wiki/UseCases .

Regards,

Zooko