linaro-project-management team mailing list archive

[Christian Robottom Reis <kiko@xxxxxxxxxx>]

Guys, I'm honestly disappointed in the general tone of "we don't control
upstream" that is getting repeated in this thread. I've said this a
number of times, and since it's still going on, I want to make it clear:
our success, and ultimately my job, depend on YOU finding a way to
strengthen our participation upstream. Ignoring the fact that members
expect us to deliver cards in their target quarter is simply not okay.

I find Nico and Amit's exchange on "unresponsive maintainers" really
enlightening. Nico's stating the obvious to me, yet it's not baked into
our mindset: WE ARE THE ARM LINUX UPSTREAM. If a maintainer is
unresponsive or blocks your work, then it's YOUR responsibility to find
a way to get the patch reviewed and either acked or nacked, and if that
takes a month to happen, then you're doing a really poor job.

Unless we take advantage of that unique position Linaro's members have
put us in, this project will fail and we'll have to go back to playing
BSP cleanup or whatever boring jobs we had before.

Yes, Deepak's kernel release alignment suggestion is on the right track;
however, alone it's not going to make things any better. I've been
waiting for a strong plan to come out of this thread, but in its
absence, my solution to this problem runs deeper and comes in steps:

    0. Managing the patches being written and submitted is critical. If
       an engineer who is supposed to write one hasn't pushed in a week,
       it should be a red alert. If nobody has replied to the patch in
       48 hours, it should be a red alert. I have not seen a single
       effort to track this, even manually, by a PM or tech lead. I
       don't care if you don't think you have time for this; it's more
       important than ANYTHING else.

    1. Quarters and kernel cycles are eerily compatible. When we put a
       card on a roadmap, realize you are saying "I plan to land this
       feature in kernel release X". When a card comes up, spend an hour
       looking at historical data and the code involved, and make a
       realistic guess.

       If the guess puts delivery 6, 9 or 12 months away, then you need
       to change the card. Maybe it needs scope pruning. Maybe you need
       3 months to produce a working patch that you'll use as an RFC
       upstream but which a member can use in his BSP. Maybe you're just
       not confident enough and need a call or meeting with a maintainer
       that can advise on the topic. But you should be prepared to work
       hard to hit the quarter when the card goes up.

    2. Plan to deliver in iterations. Each iteration needs a deadline.
       Iterations should be small; week-sized for a reasonable chunk of
       work. You won't publish the deadlines to the TSC or the world,
       but you will follow it religiously inside your team.

       When you miss an iteration, it needs to be handled specially;
       come up with a process to handle the exceptions. Upstream
       unresponsive? Developer distracted with something else? Patch got
       really hairy? Have it so a PM knows exactly how to handle it when
       the exception triggers.

       Yes, you won't know exactly the max number of iterations. But you
       must know the minimum number because that's what you have in your
       plan. And you'll know at which of those iterations the feature
       can be used in a member BSP, even if it'll continue to evolve.

    3. You need sherrifs. Platform has the release manager; the TCWG has
       patch mergers and release managers; the kernel needs people that
       are in charge of blowing the whistle on stale patches, stuck
       discussions and missed deadlines.

       Put in place specific roles and empower people to drive your
       iterations into success. Rotate the roles between PMWG and KWG,
       if you want to spread out the responsibility. Without these
       supporting roles tech leads and PMs bottleneck everything.

    4. When a maintainer asks for a major redesign, sends you off to do
       cleanup, or NACKs what you are implementing, then in no more than
       24h you should find a solution to the issue.

       I want to stress that 90% of the time there is a solution that we
       have missed. Read Nico's email again: think outside the narrow
       concept of "the maintainer", and realize that recent committers
       have a very strong voice. Other maintainers, including Linus,
       some of whom we employ, will give you support in making your
       point. We stand for our membership, and that's a lot of kernel
       credit, but you need to use it.

       If you truly, honestly can't make it happen, you will need to go
       back to the TSC and say "I've got a problem and this card date is
       forfeit." If this happens for all your cards, then go back to
       point 1 above and realize you're doing a poor job at planning.

    5. You /must/ budget cleanup time to strengthen our position in the
       kernel ecosystem. Do code cleanups. Non-critical stuff. Help
       review irrelevant patches. At least one day a week should be
       spent writing and committing code upstream that is not for a
       card, but which makes the kernel better. Each and every kernel
       engineer should be doing this.

Now, it's obvious that the TSC and members will make unrealistic demands
and lock onto dates. You should count on me to ensure that we push back
hard when we can't deliver what's asked for. But in order for me to do
that, I need to feel you're doing your part in getting things done on
time, and I honestly don't feel that way today.

Finally, on the point of data and tools:

On Wed, Mar 07, 2012 at 05:29:12PM -0800, John Stultz wrote:
> That's probably a good indicator. If we have more then one patch being
> submitted, its likely complicated enough to warrant some tracking (ie:
> not a simple fix). 
> 
> That said, patches.linaro.org is nice for exactly this sort of tracking.
> However its a little too broad in the way it does its tracking. I just
> wish there was some way to link patches/patch-bundles to blueprints.

There's a lot of shallow thinking going on in this thread, but John is
actually very insightful where he points out that tracking patchsets is
important. This is fundamental data; unless we measure it we will never
be able to control or predict delivery. I've made a mockup of how we
might track this in a 15-minute spreadsheet:

    https://docs.google.com/spreadsheet/ccc?key=0AnmosMCIa6jxdEF3MVFoTFB0ZnBaRFJuYTdKbFJXTlE

Ilias and Matthias had a call with me this week to discuss, and your
input would be great.

It's your job as leads and PMs to make this change happen. Let's do it.
-- 
Christian Robottom Reis, Engineering VP
Brazil (GMT-3) | [+55] 16 9112 6430 | [+1] 612 216 4935
Linaro.org: Open Source Software for ARM SoCs