openstack-doc-core team mailing list archive

[Tom Fifield <tom@xxxxxxxxxxxxx>]

On 25/06/15 10:03, Lana Brindley wrote:
> Hi core team,
> 
> One of the things that came up as a pain point at Summit was the DocImpact automation. I�ve been doing some thinking about this, and this is where I�m at right now:
> 
> Workflow: Devs create a patch, and at commit time they think "oh yeah, probably there should be some docs about that" and whack in a 'DocImpact' flag. In other words, there's not a lot of thought going in here, it's just an easy thing to do, they get a warm fuzzy, and there�s an assumption that the documentation fairy comes along and takes care of things. In reality, what then happens is some docs triager spends an hour looking at the patch, searching through the docs, then deciding it has nothing to do with openstack-manuals. Often, the actual doc impact (if there even is one) is against docs in the dev repo, not openstack-manuals.
> 
> In short, devs recognise they need docs, and DocImpact is an easy way to feel like they're doing something productive to make that happen. What really happens is that those bugs are largely irrelevant for openstack-manuals, which puts pressure on our core team to triage them effectively. To try and get some perspective on the size of the problem, here are some numbers:
> 
> Of the 508 current openstack-manuals bugs, 214 are the result of the DocImpact script. 51 of these are yet to be triaged.
> Right now, there are 170 patches in-flight with the DocImpact tag. To state the obvious, if all them land, that�s 170 new bugs in our queue.
> 
> So, solutions:
> 
> 1: We have a crack team of docs people (potentially with automation assistance) going through docimpact bugs, cc'ing the original patch authors, and moving them into dev repos where necessarily, and marking the rest invalid. We could team this with a an education campaign on the dev mailing list.
> 
> 2: We ditch docimpact, and force devs to create their own docs bugs (and patches). This would mean fewer devs get on with the job of documentation, but at least what we do get would be well-considered, rather than hastily added to their commit message.
> 
> 3: We adjust docimpact to raise a bug in their own dev repo. So, patch against swift with docimpact raises a bug in the swift bug queue, not the openstack-manuals doc queue. Maintenance overhead then remains with dev, and I bet you any money they would self-police that better than we ever could.
> 
> 4: Some other thing I haven�t thought of yet � ?
> 
> I have a feeling 3 is where the money's at. I took the liberty of quickly checking with an Infra core, and this change should be relatively easy to implement, too, for the record.
> 
> Thoughts?


Combo of modified #1 & modified #3, with a bit of #4 (communication back
on their patch), with a bit of #5 (modified docimpact parameters).


While I fully agree that DocImpact is in am imperfect state, and
developers are falling into the "magic fairy" trap, I want to first go
back to the reason DocImpact exists and understand the large benefit it
does give us, despite misgivings.

Prior to enlisting developer assistance to identify patches that
potentially resulted in the need for some documentation, our team had to
put in an enormous amount of effort to try and work out what was going
on dev-wise. We kinda had an inkling that there were some new features
going in, and there were probably some pretty important bugs being
fixed. To work that out, it involved reading almost every single
blueprint for features, and then trying to read a few hundred random
patches that looked interesting enough that they might result in docs.

What the introduction of DocImpact did was, for the most part, solve
this tracking problem. We successfully leveraged the collective
developer effort to let us know what the interesting things were.

Our developers aren't perfect in doing this, but in general, we get bugs
lodged so we can track major features, ensure our scripts for config
changes are working, and find out rather important updates.

In addition, we are able to fairly apply a priority to the patch that
comes in. Vendor driver? Don't really care: low.
Deprecation/Introduction of a major feature? Important: High. Bug Fix
that seriously changes behaviour: High. See also
https://wiki.openstack.org/wiki/Documentation/HowTo#Doc_Bug_Triaging_Guidelines

Though flawed and in need of a serious kick in the pants, currently
DocImpact provides the only place this tracking occurs.

So, whatever solution we adopt, wherever/however it ends up, we need to
make sure this tracking ability continues - leveraging the effort of
many to filter the firehose a little.


I now think this email is too long - I think I might address my first
line response/actual solution ideas in a separate one instead, standby!





Regards,


Tom