gtg-user team mailing list archive

[Lionel Dricot <ploum@xxxxxxxxx>]

I'm sorry for being really late and I wish you a very happy/funny/free
as in speech and as in beer 2010.


Le jeudi 17 décembre 2009 à 18:12 +1100, Jonathan Lange a écrit :
> On Thu, Dec 17, 2009 at 9:56 AM, Lionel Dricot <ploum@xxxxxxxxx> wrote:
> > The following describes my vision of backends and storage in GTG. This
> > is what I intended to do for 0.2. If you agree, it will be done for 0.3!
> >
> 
> Thanks for sharing the vision.
> 
> I don't want to stop you from doing this work, but if you want to have
> the best outcome, then it's worth being clear on exactly *why* you
> want to do this, and on what the result will look like to the user.
> Because I believe this, I'm going to ask quite a few questions.

That's fair enough and exactly why I was writing this and not blindly
implementing/comiting. Thanks for your reply.


> 
> > >From the very first lines of GTG, we knew that storing your tasks would
> > be different for everybody. That's why we designed our backend
> > architecture and we designed GTG to work with multiple backends at the
> > same time.
> >
> 
> Why would storing tasks be different for everyone? I don't use Tomboy
> any more, but I used it for years without caring how it stored my
> notes. Why would I care about how GTG stores my tasks?


Because, in fact, that's one of the biggest tomboy complain. The most
requested feature ever for GTG (and perhaps for tomboy) is the online
storage.

But some want Ubuntu One, other Weddav server or a custom webservice
like RTM.

So in fact, people care about it. It's important to them and they want
to back it up.

Tomboy neglected that fact and only recently added a hack to have sync
with Ubuntu One. It simply doesn't work : you have to manually press a
button to sync, you have conflict, no error management. 

I do want GTG to have a seamless online experience but I want to keep it
offline for those who prefer it that way.


> 
> > Did you know ? GTG already supports multiple backends since 0.1 ! If you
> > manually add a second backend in the backends.xml, it will work like
> > expected.
> >
> 
> I did know that. It makes me wonder, is there an explanation of the
> layers of the GTG architecture?

Yes. The backends layer was done to allow easy multi-backends.

The datastore/requester layer was done to isolate as much as possible
GTG core from the GUI to allow other GUI. It might be a Qt gui for KDE
guys but it could be a webinterface.

I admit : some loose bits are purely historical.

> 
> > That's fine to read task, but how will the user define in which backend
> > he wants to store a specific task ?
> >
> 
> Why would someone need more than one backend for storing a specific
> task? Why not have a document-based approach where users can open a
> task database and read & write from there?

I think I've replied to that with my example. Simpler example : Bryce
Harrington is already working on a launchpad backend. But, of course, he
will want another backend for his personal stuffs.

> 
> > Use the already existant Luke! Do you see? Indeed, I'm looking at
> > @tags !
> >
> > So, will it work ?
> >
> > I imagine 4 types of backend you can have : read-write, read-only,
> > import, export.
> >
> 
> "Backend" is a pretty generic term. I know it has a particular use
> within the GTG codebase, but I think that the discussion would be
> clearer if we could find something better to use.

Admittedly, this is becoming a GTGesque jargon ;-)

> 
> > * Read-write is the traditionnal backend you already know. It will also
> > be used for the couchdb backend and, in fact, most backends.
> 
> Is there anything already in the code or documentation that specifies
> what you must do to implement one of these? I think that writing such
> a thing would be an excellent first step.
> 
> Looking at GTG/backends/localfile.py, it seems that there's a bunch of
> top-level functions that must be provided by a module, and a Backend
> class.
> 
> The purpose of most of the top-level methods is clear, but
> get_parameters() seems pretty vague. It's not clear why some things
> are top-level functions and other things are methods.

Sorry for not explaining that. Top-level functions should be used by the
GUI to display a list of available backends and build the preferences
window for that backend.

They are top-level because you want to call them without instantiating
the backend object by itself.

Methods, on the opposite, will be used only in a particular instance of
that backend.


Do you see my point ?


> 
> > * Read-only displays a list of tasks that you have to do but you should
> > trigger something external to close them. It might be useful for
> > backends like bugzilla (at least as a first step) or for centralized
> > backend where your boss have to acknowledge that you finished a task.
> 
> AIUI, the _only_ thing this gives you that import backends don't is
> that it stops you from editing the tasks.
> 
> In that case, rather than having multiple types of backends, why not
> have a property on tasks that says whether or not that task is
> read-only?

Not at all. If you have a read-only backend, they can still be modified
elsewhere. When you use an "import" backend, you create new tasks in
your existing default backend.

Do you see the difference ?

> 
> > * Import backends will simply take tasks from the backend, just as the
> > user entered them manually. A typical example might me a Twitter
> > backend. It would retrieve all of your tweets with the tag #todo and
> > create a new task with them.
> 
> I don't think it's helpful to think of this as a "backend". It's
> actually two things:
>   - a process for putting things into a backend
>   - configuration to say how often this should take place

I don't see how it is simpler than considering it as a backend.
> 
> > * Export backend write the tasks somewhere but still keep them in GTG.
> > For example, it could export your tasks to a simple task viewer on your
> > cellphone without any complicated sync mechanism.
> >
> 
> As above. It's not a backend at all. It's a way of getting stuff out
> of a backend and into something else.

Same remark.


But you are right : we could make import/export a different kind of
plugin and use backends only for RW/RO.

That's, IMHO, not really important and still a bit far for me.
> 
> > As you can see, I think that RW and Import will already cover most
> > usecases.
> >
> > So, let's call a given instance of a backend a "TaskSource"
> >
> > By default, GTG will start with one tasksource using the current text
> > file backend. It will be possible for users to add other tasksources
> > like you add accounts in Empathy.
> >
> 
> Or couchdb by default.

I'm not sure it would be a good default as we cannot consider that our
users use couchdb. But it's a discussion we could have once I've
implemented support for multi-backends.

I personally use Ubuntu One so I'm pretty sure I would use a couchdb
backend ;-)


> 
> > When adding multiple tasksources, the user has to define a default
> > tasksource. By default, every task will go there.
> >
> 
> If you are going to allow for multiple tasksources, then this is a
> very sane approach.
> 
> > For other tasksources, user will define tags related to each
> > tasksource.
> >
> > For example, if I define a tasksource couchdb with my tasks and that I
> > have, at work, a local file tasksource (tasks about my work are
> > confidential and cannot leave the computer), I can simply say that
> > couchdb is my default and that the local file tasksource catch all tasks
> > with the tag "@work" (including subtags).
> >
> > It means that, at work, I will see my personal tasks and my work tasks
> > when, at home, I will only see my personal tasks.
> >
> > Tasksources will also have the "catchall" option so that every task goes
> > into that particular tasksource.
> >
> 
> I'm not sure how I feel about this. Although you're re-using an
> existing entity (the tag), it seems a bit complicated.
> 
> You are already going to need to have a user interface around task
> source, because people will need to be able to configure it. Why not
> make that user interface explicit when you are editing tasks, and just
> allow people to choose a task source there?

Because it will makes the task editor really complex. Configuring
tasksources should happen only once. Then, you want to forget it.

My guess is that most users will have only one tasksource. Therefore,
this should be hidden. Users that will use more tasksources will have to
learn. But I really think the tag approach is good (my feeling about
that is way better since we have subtags).

As always : any suggestion is welcome. Someone might suggest something
better, even after we have implemented the tag approach. Nothing is
written in stone.
> 
> > But wait, what if a task as multiple tags and go into multiple backend ?
> >
> > I see two solutions :
> > - we give unique ID to each task, regardless of the backend so GTG
> > understand that this is the same task.
> 
> Do that. It's a well-understood solution to a well-understood problem.
> 
> > - we adopt the "one title = one task" mantra, like Tomboy. Our unique ID
> > is the title! If you create a task with an already existing title, a (1)
> > is added "New Task (1)". If quickadd/create children with an existing
> > name, that existing task is reopened. (that might be useful to find
> > informations about the last time you did that task).
> >
> > If there's a conflict, we might put both version or does something
> > similar : it's only text after all. Also, it means that a given task
> > could have multiple ID. That's not a problem, it only has to be done.
> >
> 
> Don't do that.

I've put more thought in your answer and I will follow your advice.
Unique ID is easier for everybody.

> 
> >
> > Each backend will define itself with a set of options (a bit like
> > plugins). In order to not copy/paste too much, we might have some
> > predefined backends that are instantiated by others. For example, we
> > will have a local-text-file backend for which the user can choose the
> > location of his datas. The default backend will only be an instantiation
> > of that backend but with a fixed data path (value=XDG_HOME_DATA).
> >
> 
> That's a great idea. Another way of thinking about it is that you want
> an object to represent a format (what we call a backend), and an
> object to represent a document of that format (what we call a
> tasksource). It should be possible to create a document of a
> particular format from the format object.
> 
> The critical thing is that there has to be a clearly defined interface
> that any valid TaskSource must implement. Ideally, we'd have unit
> tests for this interface, and run these tests against all known types
> of tasksource.

Don't you mean backend ? Tasksources are instance of Backends.

The interface that should be implemented will be more or less what you
can see in localfile.py. I already know some change that have to be made
but the spirit is there.

> 
> You probably will also want an object that looks like a tasksource but
> in fact represents multiple tasksources.
> 
> > Backends will also declare if they support specific GTG features like
> > start_date, subtasks and others in the future. For example, the RTM
> > backend will not support subtasks. As a consequence, tasks from the RTM
> > backend will not accept subtasks and this feature will be disabled in
> > them. (of course, RTM plugin will be ported as a backend, at least I
> > hope so !)
> >
> 
> That's also a good idea.
> 
> Another way to do this is to have the tasksource raise a well-known
> error if you try to access a feature that it doesn't provide. You
> might find that you'll end up doing both.

Indeed.

> Although we're a long while away from this, disabling the features is
> correct, hiding them is bad. Also, the user should always have some
> way of figuring out why a thing is disabled and be able to figure out
> what they can do to enable the feature.

I agree.

> >
> > So, let's take one extreme example :
> > My default tasksource will be a couchdb one so I can access my tasks
> > from everywhere.
> > At work, I will have a local-file tasksource with @work tag (and
> > subtags).
> > I will have an import twitter tasksource that will put in my default
> > tasksource all tweets sent to @ploum with the tag #gtg (including mine)
> > (the tag @fromtwitter will be added)
> > I will have a read-only tasksource that display all bugs assigned to me
> > in Launchpad. Those will automatically have the @gtg tags.
> > I will have an export tasksource that send to my phone every tasks witth
> > the tag @shopping, overwritting the old list at each synchronisation
> > (but, as you can see, the tasks stay in the default tasksource).
> >
> 
> That's quite an extreme example.
> 
> > So, what's my plan to do this?
> >
> > Before everything (step 0), I want to change and improve the backends
> > API. For example, the get_all will no longer returns every single task
> > from the dawn of humanity but will instead retrieve opened tasks and
> > tasks that were closed less that X days ago where X is a parameter,
> > common to all backends, that the user can change. This will solve :
> > https://bugs.edge.launchpad.net/gtg/+bug/312700
> > https://bugs.edge.launchpad.net/gtg/+bug/495758
> >
> 
> Better yet, don't have a get_all method: have something that supports
> a streaming approach.

We will refactorize that to use callbacks so the get_all will, in fact,
ask the backend to add each task himself.
> 
> > As a compensation, the backends will also (maybe later) have a "search"
> > function. When receiving a search, the backend will answer with every
> > single tasks matching the search, even old ones.
> >
> 
> Which means you'll need a search API. They are tricky.
> 
> > But now, we can start the implementation.
> >
> > First, we have to make the name of a task unique.
> 
> No we don't. :)

Agreed, I've changed my mind :-)

> 
> > Also, the ID of a tasksource should be unique because that ID is written
> > in the description of tasks.
> >
> 
> Yes.
> 
> > Second step is ensure that tasks could support multiple IDs from
> > differents tasksources.
> >
> 
> Why?

Because a task could be part of multiple tasksources.

> 
> > Third step : add support to assign a specific backends based on tags.
> > Yes, it means that tasks migth be first added to the default backend
> > then removed from it once a tag is added.
> >
> 
> Actually, no. The next step is having a user interface for configuring
> a backend at all.
> 
> Then the next step is having support for _multiple_ tasksources with
> one default write tasksources. This includes things like being able to
> look at a task and see which tasksource it is from. It will also
> include things like disabling features based on backend.
> 
> *Then* you get to having support based on assigning a tasksource to a tag.

Agree but I hate Glade so much that I always put it as the last bit :-D


> 
> > Fourth : we should ensure that removing tags puts well the task in the
> > default backend.
> >
> 
> You mean, moves the task to the default backend? Yes that's important.

Yep

> 
> > Fifth : put a GUI above all of that so the user can choose and configure
> > his own tasksources.
> >
> 
> As mentioned before, I think you should do this sooner. None of the
> earlier steps have any user benefit until you can do this.

Agree but, hey, it's glade man ! ;-)

> 
> >
> > This is the vision of a multi-backends tasks system I developed for more
> > than one year. This is the last big stuff I want to add to GTG before a
> > 1.0 release. Of course, nothing is written in stone. I'm open to critics
> > and proposition (mostly from coding people as this is a quite technical
> > proposition). Also remember that GTG is a community project and that my
> > vision will only be accepted if I can convince the majority of
> > developers that this is a good idea.
> 
> Thank you for putting your vision on the line.
> 
> I realize that I've been a bit critical. This is partly because I have
> a bias against big infrastructural changes, but more because I care
> about GTG and want it to keep doing well.


I appreciate your critics. As you can see, I changed my mind for some
stuffs. I hope I answered your question.

Your contribution to GTG is highly valuable and I'm glad to have you in
our little community. Don't hesitate to send critical mails like that,
it shapes things for a better future. 


Pfew, I answered it !!!

Thanks for that,


Lionel