quickly-talk team mailing list archive

[Didier Roche <didrocks@xxxxxxxxxx>]

Le 07/08/2012 23:41, Alistair Buxton a écrit :
> Hi,
>
> Further to discussion in #quickly earlier, here is my proposal for why
> I believe that hosting service (eg Launchpad or Github or Sourceforge
> etc) and the corresponding VCS stuff should be abstracted from the
> project templates by more than just the simple inheritance we have
> now. This is mostly counters to objections raised...

Hey Alistair, thanks for that email and opening the discussion.

I would restate what Quickly is first and how Rick and I envisioned it 
from the start:
- Quickly is an opinionated tool, to build applications, and to 
particularly help opportunistic developers to create applications [on 
ubuntu].
- The first version of Quickly didn't have any template, we added them 
because we saw the need for more than one boiler plate of code
- The template has consequently been the part which is opinionated.


> Summary:
>
> When making a new project with Quickly, it should pull in two
> templates instead of one. The first template should be selected by the
> user exactly like it is now. The second template should silently
> default to "ubuntu-hosting" and it should contain the implementation
> of Quickly commands which deal with project hosting only, and no
> boilerplate code. The second template can be overridden by the user to
> select a different hosting service.
>
> To a user who is ignorant of the various hosting services, this is
> completely invisible. This does not change the functionality of
> Quickly at all, as it can already support different hosting services.
> This proposal only makes it easier for template developers to support
> different hosting services without duplicating work. In turn that
> should increase the number of templates available for users.

What are you trying to achieve there? Do you think the Quickly developer 
targets cares about the Vcs, the hosting service? Also, you seem to only 
focus on that particular part of the technology. Remember that this just 
one part of the demand: some people asked for different languages, other 
asked for different technologies (gtk/qt/html), different packaging 
system, different build system even! And the good news: we are 
supporting that! Through templates, I don't think the Vcs/hosting story 
should be different.

> I can:
>
> quickly create <any-template>
>       make a launchpad backed <any-template>-type project.
> quickly release
>       ask for my launchpad credentials and upload code to my bzr repo
>       and deb to my PPA
>
> I should also be able to:
>
> quickly create <any-template> --hosting=github
>       make a github backed <any-template>-type project.
> quickly release
>       ask for my github credentials and upload code to my git repo and
>       deb to downloads area.
Options with --<whatever> is something we tried to avoid as much as 
possible in the Quickly design, I want that we continue going that path :)

> Quickly templates and why they should be split into two parts:
>
> Quickly has a template system which serves as a basis for new
> projects. I can fork the base template and change it to use git/github
> or hg/bitbucked instead of bzr/launchpad. That's not a problem even
> under the existing Quickly template system[1]. The problem comes in
> maintaining templates that inherit from base. For every template the
> inherits directly or indirectly from base, I have to maintain an exact
> copy which inherits from my fork. This literally means a 1 line change
> and then keeping the rest of the files in sync forever. That's
> basically not going to happen, so in the current system, the user's
> choice of templates is limited by their choice of hosting service.
Again, most of Quickly users (and the target) doesn't even know what a 
Vcs is. Remember the goal of the project. I think most of git users 
doesn't want to use Quickly for instance.

> [1] see https://github.com/ali1234/quickly-git-templates
>
> Currently the inheritance tree looks like this (fixed width font required):
>
>           ubuntu-app
>          /  |  |  | \
> ubuntu-cli |  |  |  |
>    ubuntu-html |  |  bitbucket-app...
>      ubuntu-flash |
>                   github-app
>                   |  |  |
>          github-cli  |  |
>            github-html  |
>              github-flash
>
> github-cli is an exact copy of ubuntu-cli except for 1 line, but is
> not able to inherit anything from it, and does not even know that it
> exists. Also note that this inheritance tree grows exponentially as
> the number of hosting systems and project types increases. To support
> 3 hosting servies and 3 project types you need 9 templates and a large
> amount of duplicated template code.
>
> Now supposing I make some unique template. Since I like git, I'll base
> it off my forked github templates. The majority of Quickly users who
> are using Launchpad will not be able to use my work. I don't want
> that, but I am not willing to maintain two copies of 99% identical
> code, and so I'm going to just make the version that I need. It
> doesn't have to be this way.
>
>
> How it should work instead:
>
> Hosting type:          Project Type:
>
> ubuntu-hosting              generic-app
>       |                       |  |  |
> github-hosting     generic-cli  |  generic-html
>                            generic-flash
>
> When creating a project you select the hosting type and project type you want:
>
> ubuntu-hosting + generic-app = ubuntu-app
> github-hosting + generic-cli = github-cli
> ... or any other combination you want!
>
> If you didn't ask for a specific hosting type, you would get
> Ubuntu/Launchpad/bzr hosting. So the result would be exactly the same
> as it is now. It is just much less of a headache for template
> developers. To support 3 hosts and 3 project types you only need 6
> "half" templates. They are "half" because 3 of them only deal with
> hosting and 3 only deal with project types. There is no duplicated
> code.

How do you decide which part should be separated? also, what about 
different build system, what about slightly different boiler plate? 
Should we run Generic-app and then ubunut-hosting or the contrary? But 
what if we have some part bound to the boilerplate, then some generic 
code to run, then some part bound to the boilerplate again to run.

For me, this design seems to really bring more flaws that it tries to 
solve and I'm certain we will run into maintainance and hacks quickly 
after creating it. (and again it only "solves" the hosting, doesn't 
address all the other requests)
> Why should it be split this way? Why not abstract build system, or
> packaging system?
>
> To date no template (that I know of, and except for my own
> proof-of-concept fork) has modified the hosting and VCS commands. This
> just goes to show that there is clear separation between the hosting
> system and the project files/boilerplate/build system parts of the
> template.
>
> Changing the build system necessitates changing the project
> boilerplate - that is eg. switching between make and cmake requires
> CMakelists.txt instead of Makefile. Changing the packaging system has
> similar requirements. Changing the hosting service or VCS type does
> not affect the contents of the repository at all.
Right, and people who made the request had exactly the same demand than 
you: for 1 file to change, I have to maintain the whole template fork. 
So, it's not a different problem from what I can see. You can ask the 
same for packaging system.

> Also, if there were a split between project type and build system,
> then not all combinations could work together. A C++ project built by
> a setup.py makes no sense. Same for packaging: a Windows C
> application[2] packaged in an RPM makes no sense. However, any hosting
> service can host any type of project, so any hosting-template should
> be compatible with any project-template.
>
> Even if Quickly did support multiple packaging types (and I'm not
> suggesting it should or shouldn't), I wouldn't want to pick just one.
> I would want to build packages for as many different systems as my
> project can support. Again, that's strongly coupled to the
> project-type, and so this belongs with the other project-specific
> code, not in a separate user-selectable template.

Well, we proposed the -t <template> option. so you could have run 
quickly -t fedora-template package. However, in practice, we didn't see 
that happening. So I proposed on the first hangout to remove it and it 
seems it didn't get any objections. We can design for something in 10 
years, but this thing can never happen. That's why I really think we 
need to focus on what we want to achieve, for the people we want to use 
that tool, and doing an excellent job here.
> [2] Yes, even the existing Quickly implementation could easily support
> a template for developing Windows C applications under Ubuntu.
>
>
>
> What about release/submitubuntu?
>
> Any code hosting provider has a system for hosting compiled binaries
> which can handle debs, tarballs or whatever you want. It's not a PPA,
> but it is functionally equivalent for the purposes of what Quickly
> actually does, which is upload the deb somewhere where other people
> can get at it. This doesn't have to be specific to sites like github
> or sourceforge even. A template could push code to a remote VCS,
> upload the deb, and kick off makerepo, all over ssh to your own
> personal webserver, for a totally self-hosted solution.
>
> That would work with all existing Quickly commands except for one:
> submitubuntu. That is going to be harder to implement for other
> hosting providers, but not impossible. However, since nothing depends
> on submitubuntu, there's no reason why hosting templates other than
> launchpad-hosting would even need to implement it. There's certainly
> nothing in the existing Quickly templates that requires all templates
> to implement all commands...
Right, and as the set of commands will be then different, you need a 
different template.
What about the tutorial? The one which we want to build will have 
screenshot about launchpad, apps.ubuntu.com and other ubuntu facilities 
for the ubuntu-* template. Why introducing an optional hosting system 
which will shadow some commands like submitubuntu, and then, will need 
to modify the tutorial?
I think you really need another template for implementing that properly 
and that we don't make anything specific for Vcs/hosting than any other 
part of the template/boiler plate.


> But VCS abstraction is doomed to failure.
>
> It does not need to be a complete perfect abstraction of all VCS
> systems. It is only necessary to abstract the subset of features which
> Quickly actually uses, which is a fairly small subset. Quickly is
> already a VCS abstraction in this sense because you don't need to do
> any direct VCS commands to use it.
Quickly is using some commands which uses a Vcs, it's not a Vcs 
abstraction by any mean and doesn't want to be one for good reasons.
> "do not make the template more difficult to write/maintain"
>
> It does not. All you have to do is move some commands into a different
> template, one time, for the base template. After that, writing new
> templates is easier if you don't want to use Launchpad, or exactly the
> same if you do.
You have two commands parts, with hooks inside and outside the template, 
you need to do some ping-pong (in some undefined workflow if you need 
more than one call to the "inner" part of the command bound to the 
boiler plate). Yes, this is a maintenance burden IMHO.

> "do not ask additional question to the users"
>
> It does not. User won't even know this feature exists unless they
> specifically request it when creating a project.
>
> Contrast with how it works now: User already has to choose a hoster
> but it is an implicit choice, and it requires the user to find a
> template which has the right combination of host and project type that
> they want. So currently this is a question which Quickly puts to the
> user, but which is difficult for the user to answer (or even realise
> they are even being asked.) Under my proposal the choice would be
> explicit but optional instead of implicit but mandatory, and all
> possible selections would be available for all project types.
You are the second one in 3 years asking to us to be hosted on github or 
using another Vcs. So it doesn't seem to be a fairly common question. 
Most of people using Quickly seems to have no idea about what a Vcs is. 
Again, Quickly is there to make opinionated choices on technology, and 
we will keep that line. Then, we enable people to make different choice 
through templates, which contains a boiler plate. This enable to ensure 
that the combination is known, well tested and under control, which is 
what is clearly needed to give a crystal clear experience to the user.

So we don't ask any question to the user and use what we think is 
important to us. We only ship right now 3 templates by default: 
ubuntu-application, ubuntu-cli, or ubuntu-flash.
We do not ask what vcs do you want, what hosting service do you want. 
You want to create an application -> ubuntu-application, you want to 
create a command line script -> ubuntu-cli, a flash game/app -> 
ubuntu-flash (most of people knows what flash is).
Then, if the user grows on his knowledges, he can find the 
community-templates projects and use another template for a newer project.

> "do not prevent some integration"
>
> This does not prevent any template from implementing any commands that
> it wants to, as long as there are no conflicts, and some bare minimum
> requirements are implemented - eg save, build etc. - things that all
> templates should have anyway.
For that, the plan is to put as many facilities as possible to the 
templatetools libraries to make writing templates easy.
> Final Thought: Make it easy or make it impossible.
>
> If you still don't want to support different hosting systems, then
> fine. In that case it should be moved out of the project templates and
> hardcoded into Quickly core - but doing that would require creating an
> abstraction layer virtually identical to the one I am proposing.
We can support different hosting systems the same way we can support 
different languages/technologies. This is then by creating other 
templates. Also, if you have good code encapsulation, we can put the git 
part to the quickly templatetools library as we will with bzr (and as we 
did with launchpad for instance) so that creating a template will be easier.

We want to make template easier to create and not inherit "all or 
nothing" from another template commands, that's why I envision the new 
commands to be that way:

from quickly.templates import Command

class Create(Command):
def run(self):
'''Core part of the command, piloting other methods'''
self.do_a()
self.do_b()
self.do_c()
self.do_d()

then, do_a, do_b, do_c, will be functional logical part of the command. 
So, if there are bzr commands for instance, they will be in a:

def do_b(self):
templatestools.bzr.save()

def do_c(self):
launchpad= templatestools.launchpad.get_cred()
launchpad.create_project(self.project_name)
launchpad.publish_command(…)

so, if you want to create a new template inheriting from those, you can:

from quickly.templates.ubuntu_application import Create as UbuntuappCreate

class Create(UbuntuappCreate):

def do_b(self):
'''use git'''

def do_c(self):
'''use github'''

As all python functions are virtual, this is what I think is a good 
compromise. We still need a way to import some "bare metals" command 
without modifying them maybe (or juts asking for an subclassing with 
pass in it, no strong opinion on that). At least, it's an explicit 
declaration of what is imported, what is really overriden (like no 
submitubuntu template in this derivate template and a new tutorial 
template).

Now comes to the question of the boiler plate. We can figure something 
out so that you can import the boiler plate from another template 
without having to copy it, and having the command launched in the 
context of the other template, as it's already the case here.

So .commandsconfig will have some metadata for this.

It seems to me to be a less complicated design, which can grow way more 
sanely and harmoniously in the future to the new requests. It enables 
you to achieve your different hosting goal in a new template without 
having to copy the boiler plate and to maintain the whole command.
What do you think?

Thanks,
Didier