[flud-devel] private flud network and flud goals

[Stuart Langridge]

On 9/5/07, Alen Peacock <alenlpeacock at gmail.com> wrote:
> I think you'll find that flud, once finished, will do essentially what
> you describe.  I'm hoping to make another official release in the next
> week or so, but if you downloaded a snapshot from svn right now, you
> would be able to:
>
> 1. create a groupID that you could share with friends in order to
> create a private flud network. [*1]

This sounds OK, although there's some discussion below about this...

> Files are encoded using LDPC, a technique which I believe is superior
> to reed-solomon encoding used in parchive, especially for large
> amounts of data.  Current parameters require 2x remote space,
> splitting files up into 40 chunks, needing only any 20+ chunks to
> recover a file (these parameters might be adjusted in the future, or
> could even be tweaked per user).

No problem there. I wasn't wedded to PAR files specifically, just to the
idea that you can restore your data without all nodes present and
without storing a complete copy of everyone's data on every node (for
disc space reasons).

> There is *no* centralized component in flud, and thus, no sign-up
> procedure.  flud uses a shared secret, unforgeable identities, and
> challenge-response queries for nodes to a) prove their identities and
> b) prove their group membership.  If you want to set up a private flud
> network among friends, the network will remain private as long as no
> members expose the private groupID.  This corresponds fairly closely
> with the "group name" idea you mention -- as long as members know the
> group name, they can participate in the group.

...and they know how to tunnel ports through their firewall. This is the
big problem; the server isn't in my design because I want
centralisation, it's there because the moment you mention "open port
4242 on your firewall and forward it to your flud server" you have lost,
in my opinion. I was pretty serious about the "just start the program"
thing. Note that you can't just provide a "private group ID" to
potential members and have them type it in, either, without a server
that dictates which flud nodes that provate group ID pertains to. I'm
thinking here of a private group ID being something like
"langridge-family", not a block of code listing lots of flud nodes! My
use case is:

1. I say to my dad: install flud using Add/Remove Programs in Ubuntu
2. I say to my brother: download flud.exe from flud.org and run it
3. I say to both of them: run flud and type "langridge-family" and the
password "ourSekr1tPassword" in the box
4. That's it.

No port forwarding, I don't have to know which IP addresses their
machines are on, we're not on the same network (we live hundreds of
miles apart). If it can be that easy it's a huge win. I can't see how it
can be made this easy without a central server, not to store the backups
but to (a) coordinate access between different nodes, so you can just
refer to a private flud network by name and (b) to proxy connections
between two firewalled nodes.

> flud uses symmetric storage relationships among peers to enforce
> fairness.  This corresponds to your statement "if you want to back up
> N megabytes you have to offer 3N megabytes of space to the group."
> [*2]

There is a bit of a risk here, which I haven't managed to think of a
solution to: imagine a private flud network between Alice, Bob, and Dr
Evil. Alice wants to back up 1MB of data, Bob wants to back up 2MB of
data, and Dr Evil wants to back up his BitTorrent downloads folder with
800GB of downloaded episodes of 60 Minutes and Santa Barbara. Alice and
Bob will presumably have to allocate something like 150GB of disc space
for backing up the rest of the network, even though each of them only
want to back up a couple of Word documents and that's it. I can't think
of a way around this other than to have Alice and Bob nail Dr Evil's
head to a tree if he tries a trick like this.

Note also that Dr Evil will take about nine hundred years to ship all
his data over the network to be backed up, which might be a problem.

> Now, the caveats:
>
> - currently, flud is not cross-platform.  I've had to narrow focus to
> Linux in order to try to make better progress.

I, personally, don't mind about this right now; my trust in people to
run a good service and make their node available to me to restore
backups is pretty highly correlated with whether they're Linux users :)
Also, as you say, making it cross-platform shouldn't be hard; most of
the effort will be in a GUI.

> - currently, you select files from the gui and not the filemanager.  I
> initially thought that making filemanager plugins for flud on all the
> platforms it runs on was the way to go.  I've since become somewhat
> convinced (and surprised) that average users don't find that as
> intuitive as a specialized application.  But there's no reason someone
> couldn't add such interfaces if desired.

No problem with both. Some people like one window, some people like the
filer.

> - currently, there is no differential backup for changed files.  This
> is a must-have, but it is currently low priority (because it can be
> layered very nicely on top of the flud substrate).  It will become
> higher priority once we have proven flud "in-the-wild".  BTW, I think
> this will be quite easy to implement using rdiff (or somesuch),
> without the existing flud substrate even needing to know about it.  Do
> note, however, that flud won't re-backup files that are currently
> stored, and benefits from other efficiencies, such as convergent
> storage (also referred to in the literature as single-instance-store).

Yeah. This sort of thing is not that much of a problem if you're backing
up small stuff like config files. It's also not much of an issue for
large multimedia collections, since a photo or mp3 or movie don't
*change* much once they're created. Differential backup within files
isn't hugely important, IMO, as long as flud doesn't back up files it's
already got, which is already sorted.

sil


-- 
New Year's Day --
everything is in blossom!
I feel about average.
   -- Kobayashi Issa