launchpad-dev team mailing list archive

[Jeroen Vermeulen <jtv@xxxxxxxxxxxxx>]

Barry Warsaw wrote:

> A common question is whether to use these to manage transactions.  Jeroen and
> I have debated this one.  It's worked well for me in my Storm-based
> non-Launchpad applications, but Jeroen doesn't like their semantics for this.
> I'll let him take it from there.

I've got nothing against the keyword per se.  The common, simple "free 
my resource" use-case will be convenient.  Using it for that has my 
unreserved blessing.

(Irrelevant gripes: it's a thin layer of sugar; it doesn't scale well 
for more than resource; cleanup requirements should be encapsulated in 
the type not the using code; phew, that's off my chest).

My objections relate to how exception handling is designed into the 
protocol.  It basically assumes that the code inside the "with" block is 
written to suit the resource's wishes, not the other way around.

I would very much like us to stay away from any use of "with" beyond a 
substitute for "finally."  If that becomes impossible, we should figure 
out some very limited usage patterns that work well for us.  The "with" 
blocks should always do something that we can understand and predict. 
The language does not enforce that, and I see that as a maintenance risk.

Rant follows; most people will want to stop reading here.

To me, the way the protocol deals with exceptions smells of 
over-specific design.  An API like that can trick people into 
unnecessary complexity when they should be ignoring the extra knobs and 
dials.  It's like the Unmount / Eject / Safely remove drive options in 
Nautilus: in theory they let you do it just right, but in practice it 
all depends on the specific device and you just get more ways to do it 
wrong.

The justification for this design was transactions: it lets you write 
transaction classes don't need explicit commits or aborts, because they 
can see for themselves if the code block exits normally or with an 
exception.

That is not a design I like.  Maybe it's just because I'm Dutch, but 
commit should be explicit.  That's your finish line.  You may have 
separate exception handling around it depending on special needs in your 
code.  Aborts can safely be implicit, partly because they shouldn't 
raise exceptions anyway, and you may want to do them even when no 
exception comes up.

So the primary use-case is one that may seem sensible, but definitely 
not an approach I'd take.  I'd go for an explicit commit; the "with" 
exit handler would abort if commit was not reached, regardless of 
exceptions.  The design in the PEP makes the finish line invisible, but 
at the same time stakes everything on whether you cross that invisible 
line normally or by exception.  And possibly the type of the exception. 
 And possibly other details of the exception object.  Or maybe it just 
inspects your call tree and figures out what it thinks you wanted.  This 
adds a whole new dimension to a simple API that sits in your 
error-handling paths (traditionally the most numerous and least 
well-tested paths in an application) and that needs to be carefully 
designed and documented, not yet supported by documentation tools AFAIK, 
all justified by a doubtful use-case.

And then on flip side of the coin, the "with" keyword also implicitly 
swallows or propagates exceptions, at the resource's discretion.  Say 
you have a transaction manager that commits if your "with" block exits 
normally, or aborts if it raises.  Now, what does this code do?

with my_transaction_manager():
  with some_resource():
    foo()

Will the transaction abort when foo() raises an exception?  It also 
depends on the handler for some_resource, and how it feels about the 
exception.  To deal with that properly, a resource should probably have 
custom exception types to signal different exit conditions to its own 
"with" handler.  But even then:

def foo():
    with some_resource() as x:
        bar(x)

def bar(x):
    with some_resource() as y:
        x.splat()
        y.splat()

Say one of the splat()s raises a custom exception.  How sure are you 
that it gets to the right cleanup?  It could be handled by y, or by x 
and y both; the language doesn't say.  Which behavior do you want?  That 
depends on the code in foo and bar, but it's actually dictated by 
some_resource.  It's up to some_resource to implement and document 
something sensible: handle-and-raise, handle-and-swallow, raise up to 
the handler for either x or y depending on which raised the exception 
and either swallow there or raise a different exception type, etc. 
Whatever it chooses to do may or may not be what you need in your 
particular piece of code.  Actually the 3rd case is probably the most 
sensible, but the "with" API discourages that one compared to the other 
two.  The author of some_resource may go with one of the easy options 
and come up with a better one later, breaking your code.

So if you use the exception-handling parts of the protocol, "with" is 
basically an alias for multiple different control structures.  It hides 
the choice of control flow for your code inside the type.  We need to be 
very conservative with this, or risk losing sight of our exception handling.


Jeroen