[Sputnik-list] Forum implementation

[Jim Whitehead II]

On 3/6/08, Yuri Takhteyev <yuri at sims.berkeley.edu> wrote:
> > I'm in the process of implementing a very simple forums system on top
>  >  of Sputnik, but I'm running into a few problems.  A bit about the
>  >  structure first:
>
>
> First, I've been thinking of a more general concept of "comments",
>  something that could potentially be attached to any node.  I see your
>  forums as a particular way of organizing nodes with comments.  So, it
>  would be really great if your implementation could be split into
>  generic "comments" functionality that would work with any old node
>  (and could be eventually included with basic sputnik) and the more
>  complete "forum" solution (using "comments").
>
>  Here is how I imagine comments working: There would be two new
>  actions: .comment and .save_comment.  .comment will present a comment
>  form (like .edit but much simpler).  The result will get submitted to
>  .save_comment (via .post), which will _append_ something like this to
>  the "comments" field of the node:
>
>  COMMENTS[#COMMENTS+1] = {
>      user = "Yuri",
>      in_reply_to = 932,
>      content = [[Me to!]]
>  }

I don't have any problems with this, but I can't say its exactly how
I'd want the forum system to work.  The big win (for me) here is
making sure there is all of the functionality that is needed in core
to make this work.  I'd rather that it didn't need to be comment
specific, and adding a way to replace the save action would give me
exactly what I need to implement my specific forum implementation, and
a generic comments implementation.

In order for this to work, however, we need to have a way to lock a
node ESPECIALLY if we're going to be editing the parent node.  Two
edits overwriting each other in this case is a really bad idea.  This
is something that needs further thought.

>  There would be a config filed (in the prototype) that would specify
>  whether the node should allow adding comments and whether it would
>  display them.  It would probably also make sense to allow comments
>  without login.  (Though, at some point we will need to think about how
>  to make the whole authentication/registration thing more flexible.)

I'm doing user registration with email confirmation today.  It will be
done in a general way that should be easily plugged into any sputnik
installation (in theory).

>
>  >  /forums - Scans node names for anything beginning with forums: and
>  >  lists all the open "boards"
>  >  /forums:boardname - Scans node names for anything beginning with
>  >  /forums:boardname, and lists all open topics
>  >  /forums:boardname:topicid - Shows a given forum topic along with any
>
> Let's just allow "/" in node names.  It's a two-line fix to versium
>  (see below).  To start off, "forums/boardname" would be a separate
>  node, not related to "forums" (except that you can select it by
>  prefix).  We'll later add functionality (in the higher-level
>  super-versium/saci, formerly known as "versium.smart") to ask "forum"
>  to tell us where to look for it's child "boardname."  If "forums"
>  doesn't tell us this (or doesn't exist) then we'll just default to
>  asking versium for a module called "forums/boardname".  This would
>  make it backwards compatible.
>
>  BTW, Carregal gave his blessing for calling the higher level part of
>  versium "Saci".  Which is also appropriate since half of my versium
>  ideas are Carregal's former plans for Saci.
>
>  So, we then have:
>
>  versium: a very basic API/facade to (optionally) versioned storage
>  versium implementations: simple.lua, svn.lua, luasql.lua, etc.
>  (distributed separately, except for simple.lua)

Node locking needs to be added to this in some way, as far as I can
tell.  The fact that it's a node based solution and all storage is
done in the nodes itself gives us quite a problem with concurrent
edits.

>  saci: a fancy hierarchical storage solution based on versium, with
>  inheritance and eventually indexing, etc.
>
>  For those who are new to the "Saci" discussion, see
>  http://en.wikipedia.org/wiki/Saci_%28Brazilian_folklore%29.  But it's
>  also an abbreviation for something.  I think "C" is for CRUD.
>
>  >  I would like to reiterate my request for a way to manage child/parent
>  >  relationship to support categorization.  Specifically I was thinking
>  >  the following:
>  >
>  >  local nodes = sputnik:get_node_children("forums")
>
>
> Agreed.  This would be a feature in "saci.node" (today's
>  "versium.smart.smartnode").  Here is how it could work:  We would
>  check node.children, and if this field is _not_ defined, we'll just
>  ask versium for a list of nodes that start with "forums/" and do not
>  have the second slash.  However, alternatively, node.children can
>  contain an new versium configuration.  In which case we'll ask _that_
>  versium instance of _all_ of its nodes and return them with a
>  "forums/" prefix.  I am thinking that it should return the list of
>  node IDs rather than loaded nodes.

Again, I'm wary of storing that relationship in the node itself,
simply because of the issues with concurrent edits.  If we can get
that resolved, then I agree.


>  In other words, we could have:
>
>  ~/sputnik/wiki-data <- stores all the nodes
>  ~/sputnik/forums <- stores forums data
>
>  and "forums" could say that it's children are stored in a simple
>  versium at ~/sputnik/forums.  Or, first level nodes could be stored on
>  the file system, while children of "forums" could be in the database.
>
>  >  "forums/general" and "forums/support" but not "forums/general/1".  I
>  >  know you've talked/thought about this a bit.. but I'm willing to put
>  >  any work necessary to make this sort of change (With guidance).
>
> Please do.
>
>  The first change (allowing "/" in versium nodes by default) is just a
>  matter of adding an extra gsub to escape_id and unescape_id in
>  rocks/versium/lua/versium.lua
>
>  The second change is a matter of adding a new method in
>  rocks/versium/lua/versium/smart/repository.lua and (optionally) in
>  rocks/versium/lua/versium/smart/smartnode.lua.  (In some cases we want
>  to get children without assuming that the nodes exist, but in other
>  cases node:get_children() might come in handy.)
>
>  >  2. Page locking/Name generation
>  >
>  >  It would be nice if I could expect each page to have a known title,
>  >  but when we're talking something like forum threads, it's actually
>  >  much easier to provide node numbers on the backend, and give them
>  >  titles when they are displayed.  I'm looking for page names like the
>  >  following:
>  >
>  >  /forums/general/8823 - A topic in the forum "general"
>  >  /forums/general/8823/1 - A reply to the topic 8823 in forum "general"
>
> In the short term, you can use a hash of the title/content instead of
>  a sequential number.  But I agree that in the long term it would be
>  good to be able to do what you are suggesting.
>
>  >   * How do I determine the highest node number so I can generate the new name?
>
> If you have get_children method, then it should be trivial, right?  In
>  fact, I do this with Tickets in a somewhat round-about way.  When you
>  go to http://sputnik.freewisdom.org/en/Tickets and click on "new
>  ticket" it will find the next number for you.

Trivial in a single threaded application.  This is the old file system
based mkstemp race condition.  Since simple storage doesn't have
access to the posix O_CREAT and O_EXCL flags for fopen, by the time
we've checked what version we need, it could have already been opened.
 Any "existence check" that could be done is useless by the time we
need to create the file, without those flags.

>  >   * How do I handle problems with concurrency
>
> That one will take some thought.  One solution would be leave the
>  choice of name until the node is actually saved.  Another is the kind
>  of locking that you are proposing.
>
>  >  I need to be able to generate a node id (ideally when saving) for the
>  >  new reply.  I'm really not sure how best to handle this or even if
>  >  there is a good solution.  Versium _SHOULD_ be able to handle basic
>  >  locking such as through through whatever kind of storage backend is
>  >  being used.  Has this been discussed previously?
>
> Not sure what you mean by "ideally when saving".  Minding the
>  "versium/saci" distinction, I would say that versium _shouldn't_ be
>  able to do this, but perhaps "saci" or "versium" should.  One solution
>  would be to create a node to keep track of the locks.  Also, whenever
>  saci saves a node to versium, it should probably ask it for the list
>  of revisions and check if revisions have been made.  Again, I want to
>  make things simpler for storage implementors, so I don't want them to
>  deal with this.  When versium is told to save the node, it should save
>  the node.  (The only thing that it should mind is handling race
>  condition in actual disk writes.)  Saci, on the other hand, should be
>  smarter, and keep track of the possibility that the user is saving
>  edits to revision 2 while revision 3 has been saved by someone else.

Indeed, and this is the crux of most of the problems I've run into.

>  >  Another part of this is an easy way to override the save action for a
>  >  page.  Right now, these are all sent through the post action, but
>  >  there is no way to override the post actions themselves.  Is there a
>  >  way the code could be changed so the handler for "post" checks to see
>  >  if the action attribute has been set (in this case save) and redirects
>  >  the save to the handler instead?
>
> Yes.  For requests to .post get dispatched to different actions based
>  on the presence of "action_x" parameter ("action_save",
>  "action_preview", etc.)  So, if you want to post to Node.save_comment,
>  you should post to Node.post with action_save_comment set to 1.  This
>  is done in order to be able to add multiple buttons ("save", "preview"
>  and "cancel") to the same form.

Oh great, I wasn't aware of this.  This should at least allow me to
get a hacked solution to the forums finished up.

- Jim