← Back to team overview

openstack team mailing list archive

Re: Lazy import of modules

 

The LazyPluggable in nova common was based on yours I think.  The class idea is fine with me as long as those features are added.  You just have to make sure that you don't eat the exception in the try, catch, because that is what import class was doing, but it was catching underlying import errors (i.e. a missing dependency) and hiding the underlying error.

Vish

On Jan 13, 2011, at 5:30 PM, Andy Smith wrote:

> While I agree largely with the statements so far, I think the main issue with nova-common is that the swift project and the nova project have pretty much no communication going on between them. I think that is okay for now, the projects are two rather distinct codebases dropped wholesale next to each other without any prior evidence that the developers have anything in common besides python. As things progress and there are more reasons for people to be using both projects at once I expect that will have to start changing.
> 
> As for lazy import, my contribution to the discussion is the LazyPluggable class in utils.py, it is currently different from the others because it actually lazy (rather than just runtime) and because it limits the list of importable modules but could easily have that removed. Between importing objects vs modules that is a pretty easy thing to ignore with a try-catch (try first to import it as a module if that fails split the tail off and try as an object). LazyPluggable currently does that by allowing a tuple to be set as the module, but it is easy enough to make it just try both.
> 
> Re: class vs instance, I think class is still the way to go i the majority of cases but it doesn't matter in practice, if you know you are expecting a class you'll just attempt to instantiate that when you get it, but if the string is to a function rather than a class you can still use that function as a factory without any change to your code.
> 
> Examples of the various use cases in the ideal system:
> 
> # Code expecting a class
> 
> virt_driver = LazyPluggable(FLAGS['virt_driver'])
> 
> class ComputeManager(object):
>   def __init__(self):
>     self.virt = virt_driver()
> 
> # Above works if virt_driver points to:
> # nova.virt.foo.VirtDriver
> 
> class VirtDriver(object):
>   pass
> 
> # or if it points to
> # nova.virt.bar.VirtDriverFactory
> 
> def VirtDriverFactory():
>   return VirtDriver()
> 
> 
> # Code expecting an instance
> 
> db_backend = LazyPluggable(FLAGS['db_backend'])
> 
> class ComputeManager(object):
>   def __init__(self):
>     self.db = db_backend
>     self.db.some_func()
> 
> # Above works if db_backend points to:
> # nova.db.foo
> 
> def some_func():
>   pass
> 
> # or if it points to
> # nova.db.bar.public_api
> 
> class PublicApi(object):
>   def some_func(self):
>     pass
> 
> public_api = PublicApi()
> 
> All that functionality can be integrated into LazyPluggable trivially.
> 
> Why LazyPluggable over import_*? It is declarative, you can place it at the top of your file and people can see all the pluggable services this file interacts with and which flags control them.
> 
> --andy
> 
> On Thu, Jan 13, 2011 at 3:43 PM, Devin Carlen <devin.carlen@xxxxxxxxx> wrote:
> I see no problem with putting lazy loading in common.  Just because it's there doesn't mean swift has to use it. Common simply implies that they are modules used by several, but not necessarily all, projects.
> 
> 
> On Jan 13, 2011, at 3:24 PM, Vishvananda Ishaya wrote:
> 
> > The lazy loading in common seems fine minus one small issue.  If I read it correctly It looks like it is limited to a class or module.   There doesn't seem to be a way to proxy into an object itself.  I'd like to be able to specify a class (or a method) and have it lazy loaded into an object backend.  This is what import_object did automatically that makes it so versatile.  Admittedly it had an annoying bug that ate underlying exceptions, but imo, that is a bug that can be addressed.
> >
> > Vish
> >
> > On Jan 13, 2011, at 3:07 PM, Ewan Mellor wrote:
> >
> >> I can understand that the Swift guys don't want to destabilize their codebase, but Glance and Nova should have some common items, no?  Lazy loading, logging, and I18N support all spring to mind as likely to be common across Glance and Nova.
> >>
> >> Ewan.
> >>
> >>> -----Original Message-----
> >>> From: Jay Pipes [mailto:jaypipes@xxxxxxxxx]
> >>> Sent: 13 January 2011 11:50
> >>> To: Ewan Mellor
> >>> Cc: Todd Willey; openstack@xxxxxxxxxxxxxxxxxxx
> >>> Subject: Re: [Openstack] Lazy import of modules
> >>>
> >>> On Wed, Jan 12, 2011 at 6:43 PM, Ewan Mellor
> >>> <Ewan.Mellor@xxxxxxxxxxxxx> wrote:
> >>>> At the risk of starting to shave yaks: do we want to have an
> >>> openstack-common then?  It seems to be DOA at the moment.
> >>>
> >>> There's little to no agreement on common principles and code between
> >>> the projects, unfortunately. It would be best, IMHO, to just put this
> >>> in the Nova project alone. Swift coders don't have much interest in
> >>> using much of the code in Nova, and for Glance, we've used a bit of
> >>> code from Nova but will not likely be using much more, and may even
> >>> revert some of the Nova-centric stuff (like flags.py) to more
> >>> standards-based approaches used by Swift.
> >>>
> >>> -jay
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~openstack
> >> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~openstack
> >> More help   : https://help.launchpad.net/ListHelp
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 


Follow ups

References