[Sputnik-list] versium (was blueprint)

[pierre pracht]

Le 16 févr. 08 à 03:04, Yuri Takhteyev a écrit :
> On Fri, Feb 15, 2008 at 5:35 PM, pierre pracht <pierre.pracht at gmail.com 
> > wrote:
>>
>> Le 14 févr. 08 à 08:59, Yuri Takhteyev a écrit :
>>
>>> I _think_ we are on the same page here.
>>
>> repo.versium = versium.open{
>>       storage = "versium.storage.simple_compose",
>>       params = {
>>          params_write_store = {
>>             storage = "versium.storage.simple",
>>             params = initial_config.VERSIUM_PARAMS,
>>          },
>>          params_default_store = {
>>             storage = "versium.storage.simple_node_defaults",
>>             params = {base_module="sputnik.node_defaults."},
>>          }
>>       }
>>   }
>>
>> More after sleep ;)

It's a drop-in replacement of existing code.
I based it on Versium rocks, svn version give me an error

> Any indication as to what this does? :)

Nothing...     Yes seriously !
I just write it to illustrate how I see Sputnik architecture, and how  
it could evolve.

What I have done :

Previously Repository:get_node delegated storage to a aggregated  
versium.storage (actually versium.storage.simple). If no node where  
returned, it fall back to requiring a module and turning it in a node.

I have written versium.storage.simple_node_defaults who make the same  
work but under the versium.storage interface. Used in place of  
versium.storage.simple , it give you a read only wiki who give just  
node_defaults content .

To get back previous functionality I make  
versium.storage.simple_compose who take two versium.storage as  
parameters, the write store and the default store :
  - if an node can't be lookup in the write store, it would take it  
from the default store.
  - if a note is write, it send it to the write store.

So :
    - versium.storage.simple_compose
          - versium.storage.simple
          - versium.storage.simple_node_defaults

Aside illustrating some trough, the unique change is that default no  
weren't write to wiki-data upon modification.

> Ok, after you sleep...

Le 14 févr. 08 à 08:59, Yuri Takhteyev a écrit :

> Perhaps there would have been
> less confusion if versium documentation was more complete.

More than documentation, Versium is full of garbage that belongs to  
its evolution ;)
Hum... This comment is very harsh. lol
Never-mind, I just had bad time with code of packaged Sputnik. :(

I see that you made many cleanup in svn. However It wouldn't run out- 
of box.

> There are two very different parts to versium, which I almost
> separated into two projects and two rocks, but then didn't.  In my
> (yet to be released) documentation I refer to them as "versium core"
> and "smart repository" (= "versium.smart").

What I try to illustrate with supplied code is :
  - Why not define Versium as a technology.
	(general interface of node provider)
  - Push down all functionality
	(get ride of versium.lua)
  - Clean-up, refine Sputnik / Versium interaction

> Versium Core is a simple API to versioned storage.  It defines 8
> functions and there is a 264 lines of a wrapper on top of them
> (versium.lua).

In my mind, versium.lua is simply a historical artifact of wikistorage.

>  I am sure it will grow a little over time despite my
> best intents, but the idea is to keep Versium Core simple and small.
> Versium core explicitly just stores "data".  Each Versium node is a
> string.  Versium does not look into the data itself, though it
> provides a hook for "inflators".

I strongly dislike this point of view. Why data must be a string ?
I understand that Sputink starting point was to all data in a lua  
representation.
But it has spread in all the code, even becoming an interface  
requirement.

> Basically, you can give it a module
> to use for turning node's payload into a Lua table and back.  I could
> be convinced to remove the hook and put it outside versium.lua
> (though, please look at the actual code first).

Just to make the point : What if a node store an image ?

> Apart from this (and
> a few small issues discussed below), I don't see Versium Core changing
> much.

Personally I see it as versium.storage.simple
What I try to define is Versium as an abstract interface, the  
requirement for composable node storage / transformer.

For example, if a page get an attachment file (ex :pdf). You wouldn't  
make a diff on it. Worst, it didn't even make sense to consider his  
history.

So what stay in mina requirement of a Versium interface ?
Not all have to be tear off. Some could be emulated example :
  - file history : juste one entry , id : file name , version :  
modification date
  - node_defaults : author : Sputnik , version : 1 ....
In fact many property could be optional. Clarifying this point could  
be important. Code could deal with nil value if needed.

> Then there is Versium's "Smart Repository".  SR implements a whole
> bunch of weird stuff on top of Versium, starting with prototype
> inheritance.

Prototype inheritance can be implemented by a storage decorator.

> SR makes assumptions about the payload that versium
> nodes carry: it assumes them to be key-value pairs represented as Lua
> code.

I would be tempted to define this requirement down to storage.
  - Some key could store some raw data
  - Other store lua data : string, number, table, ...

> SR will probably do more weird stuff later.  Anything fancy
> like the stuff you are proposing with "Metasium" would either go into
> SR or form a layer above it.

One point is to proscribe layering by letting a system configuration  
chain decorator over node storage.

Each node would be a table implementing a composite of each  
functionality injected by storage decorator. Some convention on key  
name would represent interface of each functionality.

Better a namespace functionality can be implemented to avoid name  
collision in resulting composite node. More later ......... later ......

> About indexing: I don't think Versium Core needs to know about
> indexing at all.  It's job is to provide simple read-write interface
> to "dumb" S3-like storage.  My current thinking is that indexing
> should be implemented as a separate module and then used by SR, which
> will be responsible for updating the index when the content is
> changed.

A decorator over a storage :
   - versium.smart.search_service
     - versium.storage.?

It would intercept write
   SearchService:save_version( id, data ...)
     self:index_data(data,id)
     self.decorated_storage:save_version( id, data ...)
   end

and provide additional function to perform search :
   SearchService:find_node(list_of_keyword)
  # @return node list

> Now, one thing that I've been thinking about (and perhaps this is one
> of the things you are suggesting) is making Versium Core a bit more
> generic, and allowing it to represent storage backends that may or may
> not allow saving and may or may not maintain history.

Yes, but it can be keep in standard requirement

>  Obviously, we
> would need to be careful here, but the idea is that each storage
> implementation would report a table of "capabilities" which would
> (optionally) have fields like capabilities.saving, capability.history,
> capability.author.  I am also open to getting rid of "extra" fields.

Such marker can be implemented :
  NodeDefaultsStorage.is_writeable()
It would open introspection mechanism, but I'm not sure it's always  
useful.
   if a_node_at_some_point.can_save() then ...
Bringing them to resulting node could be more general :
   - node is not writable due to read-only storage
   - write in disable by user permission
In some case we could simply define a sensible default :
   NodeDefaultsStorage.get_node(self, id, version)
     node.comment = nil -- no comment
     return node

> André proposed a while back to that I treat versium nodes as
> "composites."

I have previously used "composite" to refer to additional node  
functionality added by decorator.

> In other words, to allow them to have children, and
> assume that any node may represent a collection of nodes.

I would be tempted to define node as atomic resource. They can hold id  
of other resources, but not resources themselves.
The objective is to let storage decorator take on every node without  
being aware of node structure or relation ship.
In the same time, a decorator can be aware of resource nesting and  
handle it appropriately :
    NesstedNodeStorage.remove_node(id)
       children_id = self:get_children_id(id) -- May be provided by an  
other decorator
       for child_id in children do
          self:remove_node(child_id)
       end
      -- let underlaying storage technology do the dirty work
       self.decorated_storage:remove_node(id)
    end


> With your
> suggestions of stacked storage (if I understood it correctly), we get
> an new twist to this.  Suppose we say that each node (and we are
> talking about "Smart" nodes, here, not Versium Core) would not only be
> allowed to have children, but can also decide where and how to store
> them.

Node can effectively come from different source.

> Which is to say that each SR node can configure it's own
> versium core.

No a storage decorator can split responsibility of storage according  
to node id. The id can reflect this fact, but not necessary. It's the  
responsibility of decorator to take care of it

> An extra benefit is that in this scheme Versium Core
> could stay oblivious of node nesting.

In previous scheme you get same abstraction.

> We could have something like this:
>
> 1. Root Node ("/") - itself "stored" as a table defined in
> "sputnik.cgi", storing it's children in wiki-data directory with
> versium.storage.simple  (BTW, don't confuse "/" with @Root.)
> 2. A bunch of children of "/" stored in wiki-data, most of them  
> child-less.
> 3. Some child of "/", say, "Section_1" gets configured to store
> children in some other directory, say "wiki-data-section-1", also
> using versium.storage.simple.
> 4. Another child, say "Temp" gets configured to store it's children in
> memory, using versium.storage.in_memory.  Those will just disappear
> when the process dies, but let's assume we don't care for some reason.
> 5. Yet another node, perhaps "Section_1/Gigs_of_Data" gets configured
> to store it's children in S3.
>
> Is that sort of what you were suggesting?

Yes, but with caution of previous point.

> In this case indexing could be done hierarchically, in the sense that
> each node will decide whether to pass a query to children and how to
> filter the results.

No, id can be URI, a storage decorator can be aware of URI structure  
and dispatch storage accordingly. To illustrate the difference, all  
"child" can have attachment, all attachment go to S3. So storage isn't  
tied to nesting.

Rq: and version could be generalized to UUID

> Now, this is not a promise to implement any of this.

As I have show in the example code, the problem isn't code to be  
written.

> But I suppose it would be good to come to some consensus

It's the main point, ensure convention and naming so architecture keep  
evolving.

> as to where this could or should go when someone has time to write it.

If we can achieve previous point, each can implement is own  
functionality.
But specification can be sketched as implementation evolve.
If we start with some basic functionality who illustrate previous  
scheme, it would be a good start. I volunteer to implement them. No  
need to express them in previous design term. Just express some  
requirement.

To conclude :
   - What do you thing of the whole idea ?
   - What functionality is needed ?

- pierre

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.luaforge.net/pipermail/sputnik-list/attachments/20080216/58c1e46a/attachment.html