fuel-dev team mailing list archive

[Andrey Korolyov <akorolev@xxxxxxxxxxxx>]

On 03/29/2014 02:06 AM, Andrew Woodward wrote:
>     Firstly, 3.10-lt kernel is an LTS kernel supported by Linux Foundation 
> 
> For example Cisco UCS is not validated on CentOS using 3.10. Cisco won't
> provide support because they didn't validate it. Another example is if a
> vendor provides kernel drives they may be bound to 2.6 and not provide
> packages that cant be bound to 3.10 again because they dont test it. We
> will have issues with this until RHEL is on 3.10 and vendors start to
> test and validate against it.
>  
> 
>     We've already experienced issues with hardware unsupported by 2.6.32
>     kernel
> 
> This is the first that I've heard of hardware problems other than OVS
> and just plain missing drivers that the newer kernels have embeded
> (Which is our bad). Please provide examples.
> 
> If you want to upgrade CentOS 6.x to 3.10 and make it default, you can't
> this is Red Hat's job. This is a limitation that is known to the people
> who select CentOS. If they dont like it, then they can use the fedora-lt
> kernel at their own volition, wait for CentOS 7, or use Ubuntu. Taking
> away the default kernel from CentOS 6.x violates a core assumption users
> are going to make if they select to install it and makes it NOT CentOS.
> 
> Yes I object to using the 3.10 kernel any were for CentOS by default.
> Again, its no longer CentOS and no longer something that we can test
> effectively. We are going down the slope of needing to validate a kernel
> against a myriad of hardware platforms. This is something that we
> absolutely should not be attempting to do. Hardware vendors and Distos
> do this for us with their default kernel for distributions they support.
> 
> again this goes back to what problem are we trying to solve by moving
> the bootstrap kernel to 3.10? I'm still guessing here so you really need
> to come up with a story here so that we can actually solve the problem. 

Hello,

for CentOS 6, there are at least following major issues(or blockers)

- UCS chassis networking
- connectivity with VLAN Neutron mode for at least half of end-user nic
models
- performance for GRE Neutron mode for selected cards
- block device naming

Please remember that MOS went too far away from bare CentOS due to lot
of individual package upgrades to be compared with base distro. There is
no way to ship out-of-box CentOS with all currently supported modes -
Neutron will be broken, Ceph support will be damaged and so on. Also
Centos have nothing to do with binaries provided by RH, it`s just a
volunteer work which is applicable for evaluation but impossible for any
kind of industrial usage, even with recent RH announcements for
supporting it. I agree that it may be not a good idea to switch
everything to 3.x branch at once, but if one will leave selection
between in-distro(though MOS actually uses *completely* unsupported
kernel patchset with fixed related to Neutron interoperability in RH`s
2.6.32 frankenkernel) an latest stable, there is no point to push it
away from, say, master node or boostrap because least number of
hardware-compatible issues will be presented from the beginning. I can
name just one side effect from moving bootstrap to the newer kernel - it
is cciss device path naming which went away in 3.x but still here with
current CentOS kernel.

Keeping focus on the CentOS, please bear in mind that although kernel in
it receiving at least some significant patches, userspace is ancient and
almost never inherits features from upstream, resulting in poor user
practices - for example, shipped glibc leaking under specific
conditions, namespace support is just bare and requires a couple of
overrides via newer package installation etc etc. Again, when RedHat
actually does guarantee that their binaries will work as promised,
CentOS community simply not able to do the same, so matter of amount of
overrides is just a sight to the gap between evaluated and production
distro, not an actual production. Sorry for a long post, but I really do
not like exchanging software roles in view.

> 
> It sounds like you are looking to improve UX when installing, great!.
> However this is the wrong way. We can either continue to have this A/B
> split between the kernel that is used by Ubuntu and the kernel we are
> using for CentOS (no moving to 3.10 in CentOS does not resolve this
> issue) or we can fix this by improving the approach.
> 
> 1) I think that if we just want to improve the bootstrap kernel, then we
> should switch bootstrap to Ubuntu, it gets a 3.8 kernel
> 2) If we want to solve this A/B issue then we need to prepare both
> CentOS and Ubuntu bootstrap images and allow the user to switch. It's
> the only complete solution and I'd guess that most of the time they will
> only use one, or the other. (Unless they test one and then switch to the
> other).
> 
> 
> 
> On Fri, Mar 28, 2014 at 2:17 PM, Vladimir Kuklin <vkuklin@xxxxxxxxxxxx
> <mailto:vkuklin@xxxxxxxxxxxx>> wrote:
> 
>     Andrew, Dmitry N,
> 
>     Firstly, 3.10-lt kernel is an LTS kernel supported by Linux
>     Foundation, this is not a piece of some short-term supported code.
>     Moreover, this kernel has a support of much more hardware as it is
>     not so ancient as 2.6.32 kernel is. We've already experienced issues
>     with hardware unsupported by 2.6.32 kernel. This makes our users
>     pretty unhappy, that they cannot use their newer hardware, even
>     though it is supported by 3.10 and Ubuntu 3.x kernels. Thus,
>     changing bootstrap kernel allows users to use their newer hardware
>     without breaking things as user is still able to choose the kernel
>     for main OS installation. Thus we faced the fact that we need either
>     to provide bootstrap with the newest LTS kernel or create several
>     bootstrap images, which looks much more complicated. We chose the
>     compromise. This is obviously a mistake, that we did not dicsuss
>     this in this ML. So, if you have any objections and you think that
>     we should revert this change, please, provide you arguments. In my
>     opinion, changing only the bootstrap image kernel is not a big deal. 
> 
> 
> 
> 
>     On Fri, Mar 28, 2014 at 10:31 PM, Dmitriy Novakovskiy
>     <dnovakovskiy@xxxxxxxxxxxx <mailto:dnovakovskiy@xxxxxxxxxxxx>> wrote:
> 
>         +1 to Andrew's points. I've also been dealing with a bulk of
>         vendors' pain - plugins (Cinder, Neutron) not working properly
>         due to our CentOS being not actually CentOS (kernel+qemu versions).
> 
>         ---
>         Regards,
>         Dmitriy
> 
> 
>         On Fri, Mar 28, 2014 at 11:22 AM, Andrew Woodward
>         <xarses@xxxxxxxxx <mailto:xarses@xxxxxxxxx>> wrote:
> 
>                 Right now master node has 2.6 kernel, bootstrap has 3.10
>                 and target node has an option in settings with 2.6
>                 kernel as default.
> 
> 
>             WHAT? Why is the bootstrap 3.10 already? I already spoke to
>             the issues that some users have had about this. This should
>             have been discussed on the ML before making a change like this.
> 
>             First off, I'm sorry but making the 3.10 the default kernel
>             for our "CentOS 6.x" installs inherently makes it NOT
>             CentOS, it becomes Fedora 18. If we are going to do this,
>             then we might as well switch from CentOS base to a proper
>             Fedora base. At the same time, making 3.10 the default
>             kernel impacts vendor support. Vendors that might have
>             supported CentOS 6.x now wont. I've already been in on a
>             conversation with with a user, and separately two vendors
>             that are having a hard time using our "CentOS" because we
>             don't have a stock kernel. This removes support-ability from
>             the list of reasons to use CentOS, which is one of the core
>             reasons I've seen organizations select it.
> 
>             Second, what is the goal here? 
> 
>             * if we are trying reduce issues from master -> bootstrap ->
>             target node? switching kernel around? If this is the case
>             then we really should just change master and bootstrap to
>             Ubuntu (which I've been considering raising anyways).
> 
>             * If we are trying to reduce the OVS issues (or other old
>             kernel issues), then the current workup is correct, the user
>             should have to explicitly choose a kernel that wasn't part
>             of the base distribution, this is important as is must be
>             their choice and not default to enter into a mode that
>             reduces their support. 
> 
>             * If we are trying to get CentOS to a more modern kernel,
>             then we really just have to wait for CentOS 7 otherwise we
>             are really better off tracking Fedora instead of CentOS
> 
>             Third, this is changing our user story of "No vendor
>             lock-in" to "You're stuck with Mirantis". We really need to
>             take a step back an evaluate where we are going. If we want
>             to maintain our on proprietary distro then we need to stop
>             beating around and do it, and do it well. If we are going to
>             fight vendor lock-in, then we need to be more cautions about
>             what distro packages we are modifying and why. We also need
>             to not replace distro packages but instead maintain a clean
>             distro repos and a MOS repo, this will grant us a clear line
>             of what we are doing and better allow others to participate
>             (like bringing in other distros, vendor compatibility, and
>             others)
> 
> 
>             On Fri, Mar 28, 2014 at 5:08 AM, Matthew Mosesohn
>             <mmosesohn@xxxxxxxxxxxx <mailto:mmosesohn@xxxxxxxxxxxx>> wrote:
> 
>                 I propose that since we do kernel-lt 3.10.30 as a
>                 default for installs
>                 and bootstrap, we should default to it for Fuel master
>                 node as well.
>                 I believe we should keep the 2.6.32 kernel around just
>                 in case there
>                 is a regression in using the newer kernel, but maybe we
>                 can drop it
>                 after a release cycle or two if there are no major issues.
> 
>                 The build tiem option I mentioned to you would be to
>                 optionally
>                 generate a bootstrap image with the 2.6.32 kernel during
>                 ISO build. It
>                 isn't likely we will need to build it, but it should be
>                 easy to swap
>                 back if the need arises.
> 
>                 On Fri, Mar 28, 2014 at 4:03 PM, Dmitry Pyzhov
>                 <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx>> wrote:
>                 > Guys,
>                 >
>                 > we have two kernels: default 2.6.32 and kernel-lt
>                 3.10.30. And we have three
>                 > types of CentOS nodes: master node, bootstrap, target
>                 node.
>                 >
>                 > Right now master node has 2.6 kernel, bootstrap has
>                 3.10 and target node has
>                 > an option in settings with 2.6 kernel as default.
>                 >
>                 > There is a suggestion from our team members to use
>                 3.10 kernel everywhere
>                 > and add build-time option for 2.6.
>                 >
>                 > Any concerns?
>                 >
>                 > --
>                 > Mailing list: https://launchpad.net/~fuel-dev
>                 > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>                 <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>                 > Unsubscribe : https://launchpad.net/~fuel-dev
>                 > More help   : https://help.launchpad.net/ListHelp
>                 >
> 
>                 --
>                 Mailing list: https://launchpad.net/~fuel-dev
>                 Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>                 <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>                 Unsubscribe : https://launchpad.net/~fuel-dev
>                 More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
>             -- 
>             Andrew
>             Mirantis
>             Ceph community
> 
>             --
>             Mailing list: https://launchpad.net/~fuel-dev
>             Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>             <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>             Unsubscribe : https://launchpad.net/~fuel-dev
>             More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
>         --
>         Mailing list: https://launchpad.net/~fuel-dev
>         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>         Unsubscribe : https://launchpad.net/~fuel-dev
>         More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
>     -- 
>     Yours Faithfully,
>     Vladimir Kuklin,
>     Fuel Library Tech Lead,
>     Mirantis, Inc.
>     +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
>     +7 (926) 702-39-68 <tel:%2B7%20%28926%29%20702-39-68>
>     Skype kuklinvv
>     45bk3, Vorontsovskaya Str.
>     Moscow, Russia,
>     www.mirantis.com <http://www.mirantis.ru/>
>     www.mirantis.ru <http://www.mirantis.ru/>
>     vkuklin@xxxxxxxxxxxx <mailto:vkuklin@xxxxxxxxxxxx>
> 
> 
> 
> 
> -- 
> Andrew
> Mirantis
> Ceph community
> 
>