← Back to team overview

quickly-talk team mailing list archive

Re: Proposal: Abstract hosting service from project templates.

 

Hi all
 I agree with Didier that the solution should be comprehensive not
just for hosting. There is a problem with existing templates being
monolithic. An example was ubuntu-pygame template. It was enhanced
with a high score system using desktop couch db. When desktop couch db
became unusable the whole template was removed.
 I got fed up with monolithic templates so I wrote a modular quickly
template system. Have a look at
https://code.launchpad.net/~tony-badwolf/quickly/modular. Basically
you could write an "empty" template that gets all its project files
from ubuntu-application template at create time and only contains some
custom code for git.

On 08/08/2012, Didier Roche <didrocks@xxxxxxxxxx> wrote:
> Le 07/08/2012 23:41, Alistair Buxton a écrit :
>> Hi,
>>
>> Further to discussion in #quickly earlier, here is my proposal for why
>> I believe that hosting service (eg Launchpad or Github or Sourceforge
>> etc) and the corresponding VCS stuff should be abstracted from the
>> project templates by more than just the simple inheritance we have
>> now. This is mostly counters to objections raised...
>>
>
> Hey Alistair, thanks for that email and opening the discussion.
>
> I would restate what Quickly is first and how Rick and I envisioned it
> from the start:
> - Quickly is an opinionated tool, to build applications, and to
> particularly help opportunistic developers to create applications [on
> ubuntu].
> - The first version of Quickly didn't have any template, we added them
> because we saw the need for more than one boiler plate of code
> - The template has consequently been the part which is opinionated.
>
>
>>
>> Summary:
>>
>> When making a new project with Quickly, it should pull in two
>> templates instead of one. The first template should be selected by the
>> user exactly like it is now. The second template should silently
>> default to "ubuntu-hosting" and it should contain the implementation
>> of Quickly commands which deal with project hosting only, and no
>> boilerplate code. The second template can be overridden by the user to
>> select a different hosting service.
>>
>> To a user who is ignorant of the various hosting services, this is
>> completely invisible. This does not change the functionality of
>> Quickly at all, as it can already support different hosting services.
>> This proposal only makes it easier for template developers to support
>> different hosting services without duplicating work. In turn that
>> should increase the number of templates available for users.
>
> What are you trying to achieve there? Do you think the Quickly developer
> targets cares about the Vcs, the hosting service? Also, you seem to only
> focus on that particular part of the technology. Remember that this just
> one part of the demand: some people asked for different languages, other
> asked for different technologies (gtk/qt/html), different packaging
> system, different build system even! And the good news: we are
> supporting that! Through templates, I don't think the Vcs/hosting story
> should be different.
>
>> I can:
>>
>> quickly create <any-template>
>>       make a launchpad backed <any-template>-type project.
>> quickly release
>>       ask for my launchpad credentials and upload code to my bzr repo
>>       and deb to my PPA
>>
>> I should also be able to:
>>
>> quickly create <any-template> --hosting=github
>>       make a github backed <any-template>-type project.
>> quickly release
>>       ask for my github credentials and upload code to my git repo and
>>       deb to downloads area.
> Options with --<whatever> is something we tried to avoid as much as
> possible in the Quickly design, I want that we continue going that path :)
>
>> Quickly templates and why they should be split into two parts:
>>
>> Quickly has a template system which serves as a basis for new
>> projects. I can fork the base template and change it to use git/github
>> or hg/bitbucked instead of bzr/launchpad. That's not a problem even
>> under the existing Quickly template system[1]. The problem comes in
>> maintaining templates that inherit from base. For every template the
>> inherits directly or indirectly from base, I have to maintain an exact
>> copy which inherits from my fork. This literally means a 1 line change
>> and then keeping the rest of the files in sync forever. That's
>> basically not going to happen, so in the current system, the user's
>> choice of templates is limited by their choice of hosting service.
> Again, most of Quickly users (and the target) doesn't even know what a
> Vcs is. Remember the goal of the project. I think most of git users
> doesn't want to use Quickly for instance.
>
>> [1] see https://github.com/ali1234/quickly-git-templates
>>
>> Currently the inheritance tree looks like this (fixed width font
>> required):
>>
>>           ubuntu-app
>>          /  |  |  | \
>> ubuntu-cli |  |  |  |
>>    ubuntu-html |  |  bitbucket-app...
>>      ubuntu-flash |
>>                   github-app
>>                   |  |  |
>>          github-cli  |  |
>>            github-html  |
>>              github-flash
>>
>> github-cli is an exact copy of ubuntu-cli except for 1 line, but is
>> not able to inherit anything from it, and does not even know that it
>> exists. Also note that this inheritance tree grows exponentially as
>> the number of hosting systems and project types increases. To support
>> 3 hosting servies and 3 project types you need 9 templates and a large
>> amount of duplicated template code.
>>
>> Now supposing I make some unique template. Since I like git, I'll base
>> it off my forked github templates. The majority of Quickly users who
>> are using Launchpad will not be able to use my work. I don't want
>> that, but I am not willing to maintain two copies of 99% identical
>> code, and so I'm going to just make the version that I need. It
>> doesn't have to be this way.
>>
>>
>> How it should work instead:
>>
>> Hosting type:          Project Type:
>>
>> ubuntu-hosting              generic-app
>>       |                       |  |  |
>> github-hosting     generic-cli  |  generic-html
>>                            generic-flash
>>
>> When creating a project you select the hosting type and project type you
>> want:
>>
>> ubuntu-hosting + generic-app = ubuntu-app
>> github-hosting + generic-cli = github-cli
>> ... or any other combination you want!
>>
>> If you didn't ask for a specific hosting type, you would get
>> Ubuntu/Launchpad/bzr hosting. So the result would be exactly the same
>> as it is now. It is just much less of a headache for template
>> developers. To support 3 hosts and 3 project types you only need 6
>> "half" templates. They are "half" because 3 of them only deal with
>> hosting and 3 only deal with project types. There is no duplicated
>> code.
>
> How do you decide which part should be separated? also, what about
> different build system, what about slightly different boiler plate?
> Should we run Generic-app and then ubunut-hosting or the contrary? But
> what if we have some part bound to the boilerplate, then some generic
> code to run, then some part bound to the boilerplate again to run.
>
> For me, this design seems to really bring more flaws that it tries to
> solve and I'm certain we will run into maintainance and hacks quickly
> after creating it. (and again it only "solves" the hosting, doesn't
> address all the other requests)
>>
>>
>> Why should it be split this way? Why not abstract build system, or
>> packaging system?
>>
>> To date no template (that I know of, and except for my own
>> proof-of-concept fork) has modified the hosting and VCS commands. This
>> just goes to show that there is clear separation between the hosting
>> system and the project files/boilerplate/build system parts of the
>> template.
>>
>> Changing the build system necessitates changing the project
>> boilerplate - that is eg. switching between make and cmake requires
>> CMakelists.txt instead of Makefile. Changing the packaging system has
>> similar requirements. Changing the hosting service or VCS type does
>> not affect the contents of the repository at all.
> Right, and people who made the request had exactly the same demand than
> you: for 1 file to change, I have to maintain the whole template fork.
> So, it's not a different problem from what I can see. You can ask the
> same for packaging system.
>
>> Also, if there were a split between project type and build system,
>> then not all combinations could work together. A C++ project built by
>> a setup.py makes no sense. Same for packaging: a Windows C
>> application[2] packaged in an RPM makes no sense. However, any hosting
>> service can host any type of project, so any hosting-template should
>> be compatible with any project-template.
>>
>> Even if Quickly did support multiple packaging types (and I'm not
>> suggesting it should or shouldn't), I wouldn't want to pick just one.
>> I would want to build packages for as many different systems as my
>> project can support. Again, that's strongly coupled to the
>> project-type, and so this belongs with the other project-specific
>> code, not in a separate user-selectable template.
>
> Well, we proposed the -t <template> option. so you could have run
> quickly -t fedora-template package. However, in practice, we didn't see
> that happening. So I proposed on the first hangout to remove it and it
> seems it didn't get any objections. We can design for something in 10
> years, but this thing can never happen. That's why I really think we
> need to focus on what we want to achieve, for the people we want to use
> that tool, and doing an excellent job here.
>>
>> [2] Yes, even the existing Quickly implementation could easily support
>> a template for developing Windows C applications under Ubuntu.
>>
>>
>>
>> What about release/submitubuntu?
>>
>> Any code hosting provider has a system for hosting compiled binaries
>> which can handle debs, tarballs or whatever you want. It's not a PPA,
>> but it is functionally equivalent for the purposes of what Quickly
>> actually does, which is upload the deb somewhere where other people
>> can get at it. This doesn't have to be specific to sites like github
>> or sourceforge even. A template could push code to a remote VCS,
>> upload the deb, and kick off makerepo, all over ssh to your own
>> personal webserver, for a totally self-hosted solution.
>>
>> That would work with all existing Quickly commands except for one:
>> submitubuntu. That is going to be harder to implement for other
>> hosting providers, but not impossible. However, since nothing depends
>> on submitubuntu, there's no reason why hosting templates other than
>> launchpad-hosting would even need to implement it. There's certainly
>> nothing in the existing Quickly templates that requires all templates
>> to implement all commands...
> Right, and as the set of commands will be then different, you need a
> different template.
> What about the tutorial? The one which we want to build will have
> screenshot about launchpad, apps.ubuntu.com and other ubuntu facilities
> for the ubuntu-* template. Why introducing an optional hosting system
> which will shadow some commands like submitubuntu, and then, will need
> to modify the tutorial?
> I think you really need another template for implementing that properly
> and that we don't make anything specific for Vcs/hosting than any other
> part of the template/boiler plate.
>
>
>> But VCS abstraction is doomed to failure.
>>
>> It does not need to be a complete perfect abstraction of all VCS
>> systems. It is only necessary to abstract the subset of features which
>> Quickly actually uses, which is a fairly small subset. Quickly is
>> already a VCS abstraction in this sense because you don't need to do
>> any direct VCS commands to use it.
> Quickly is using some commands which uses a Vcs, it's not a Vcs
> abstraction by any mean and doesn't want to be one for good reasons.
>>
>> "do not make the template more difficult to write/maintain"
>>
>> It does not. All you have to do is move some commands into a different
>> template, one time, for the base template. After that, writing new
>> templates is easier if you don't want to use Launchpad, or exactly the
>> same if you do.
> You have two commands parts, with hooks inside and outside the template,
> you need to do some ping-pong (in some undefined workflow if you need
> more than one call to the "inner" part of the command bound to the
> boiler plate). Yes, this is a maintenance burden IMHO.
>
>> "do not ask additional question to the users"
>>
>> It does not. User won't even know this feature exists unless they
>> specifically request it when creating a project.
>>
>> Contrast with how it works now: User already has to choose a hoster
>> but it is an implicit choice, and it requires the user to find a
>> template which has the right combination of host and project type that
>> they want. So currently this is a question which Quickly puts to the
>> user, but which is difficult for the user to answer (or even realise
>> they are even being asked.) Under my proposal the choice would be
>> explicit but optional instead of implicit but mandatory, and all
>> possible selections would be available for all project types.
> You are the second one in 3 years asking to us to be hosted on github or
> using another Vcs. So it doesn't seem to be a fairly common question.
> Most of people using Quickly seems to have no idea about what a Vcs is.
> Again, Quickly is there to make opinionated choices on technology, and
> we will keep that line. Then, we enable people to make different choice
> through templates, which contains a boiler plate. This enable to ensure
> that the combination is known, well tested and under control, which is
> what is clearly needed to give a crystal clear experience to the user.
>
> So we don't ask any question to the user and use what we think is
> important to us. We only ship right now 3 templates by default:
> ubuntu-application, ubuntu-cli, or ubuntu-flash.
> We do not ask what vcs do you want, what hosting service do you want.
> You want to create an application -> ubuntu-application, you want to
> create a command line script -> ubuntu-cli, a flash game/app ->
> ubuntu-flash (most of people knows what flash is).
> Then, if the user grows on his knowledges, he can find the
> community-templates projects and use another template for a newer project.
>
>> "do not prevent some integration"
>>
>> This does not prevent any template from implementing any commands that
>> it wants to, as long as there are no conflicts, and some bare minimum
>> requirements are implemented - eg save, build etc. - things that all
>> templates should have anyway.
> For that, the plan is to put as many facilities as possible to the
> templatetools libraries to make writing templates easy.
>> Final Thought: Make it easy or make it impossible.
>>
>> If you still don't want to support different hosting systems, then
>> fine. In that case it should be moved out of the project templates and
>> hardcoded into Quickly core - but doing that would require creating an
>> abstraction layer virtually identical to the one I am proposing.
>>
> We can support different hosting systems the same way we can support
> different languages/technologies. This is then by creating other
> templates. Also, if you have good code encapsulation, we can put the git
> part to the quickly templatetools library as we will with bzr (and as we
> did with launchpad for instance) so that creating a template will be
> easier.
>
> We want to make template easier to create and not inherit "all or
> nothing" from another template commands, that's why I envision the new
> commands to be that way:
>
> from quickly.templates import Command
>
> class Create(Command):
> def run(self):
> '''Core part of the command, piloting other methods'''
> self.do_a()
> self.do_b()
> self.do_c()
> self.do_d()
>
> then, do_a, do_b, do_c, will be functional logical part of the command.
> So, if there are bzr commands for instance, they will be in a:
>
> def do_b(self):
> templatestools.bzr.save()
>
> def do_c(self):
> launchpad= templatestools.launchpad.get_cred()
> launchpad.create_project(self.project_name)
> launchpad.publish_command(…)
>
> so, if you want to create a new template inheriting from those, you can:
>
> from quickly.templates.ubuntu_application import Create as UbuntuappCreate
>
> class Create(UbuntuappCreate):
>
> def do_b(self):
> '''use git'''
>
> def do_c(self):
> '''use github'''
>
> As all python functions are virtual, this is what I think is a good
> compromise. We still need a way to import some "bare metals" command
> without modifying them maybe (or juts asking for an subclassing with
> pass in it, no strong opinion on that). At least, it's an explicit
> declaration of what is imported, what is really overriden (like no
> submitubuntu template in this derivate template and a new tutorial
> template).
>
> Now comes to the question of the boiler plate. We can figure something
> out so that you can import the boiler plate from another template
> without having to copy it, and having the command launched in the
> context of the other template, as it's already the case here.
>
> So .commandsconfig will have some metadata for this.
>
> It seems to me to be a less complicated design, which can grow way more
> sanely and harmoniously in the future to the new requests. It enables
> you to achieve your different hosting goal in a new template without
> having to copy the boiler plate and to maintain the whole command.
> What do you think?
>
> Thanks,
> Didier
>
> _______________________________________________
> Mailing list: https://launchpad.net/~quickly-talk
> Post to     : quickly-talk@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~quickly-talk
> More help   : https://help.launchpad.net/ListHelp
>


References