yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #61598
[Bug 1665548] [NEW] nova fail to evacuate as wrong onSharedStorarge parameter
Public bug reported:
Nova API removed onSharedStorage parameter after v2.14, but we still
have many users use API under v2.14. And we found that nova 100% fail to
evacuate a server via novaclient, the error log as follows:
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher [req-b34b79d6-20ed-4488-b8d5-98f4007eb12e 681041a7364d4852930021d009c8dc2b bafaf53fac4346b9bcd6a77cf964b7af - - -] Exception during message handling: Invalid state of instance files on shared storage
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher payload)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2885, in _do_rebuild_instance
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher _("Invalid state of instance files on shared"
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher
I dug into the API source, and I found that the onSharedStorage will
force to be converted to False value, even if in fact its value is None:
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/evacuate.py#L43-L48
In other word, If we didn't pass this arguments, we will actually pass
the False value. If we use shared-storage, it will not match the value
getting from the backend driver.
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
- Nova API removed onSharedStorage parameter after v2.14, but we have many
- user to API under v2.14. And we found that nova 100% fail to evacuate a
- server via novaclient, the error log as follows:
+ Nova API removed onSharedStorage parameter after v2.14, but we still
+ have many users use API under v2.14. And we found that nova 100% fail to
+ evacuate a server via novaclient, the error log as follows:
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher [req-b34b79d6-20ed-4488-b8d5-98f4007eb12e 681041a7364d4852930021d009c8dc2b bafaf53fac4346b9bcd6a77cf964b7af - - -] Exception during message handling: Invalid state of instance files on shared storage
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher payload)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2885, in _do_rebuild_instance
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher _("Invalid state of instance files on shared"
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher
- I dig into the API source, and I found that the onSharedStorage will
- force to be converted to False, even if in fact its value is None:
+ I dug into the API source, and I found that the onSharedStorage will
+ force to be converted to False value, even if in fact its value is None:
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/evacuate.py#L43-L48
- In other words, If we didn't pass this arguments, we will actually pass
+ In other word, If we didn't pass this arguments, we will actually pass
the False value. If we use shared-storage, it will not match the value
- from backend driver.
+ getting from the backend driver.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1665548
Title:
nova fail to evacuate as wrong onSharedStorarge parameter
Status in OpenStack Compute (nova):
New
Bug description:
Nova API removed onSharedStorage parameter after v2.14, but we still
have many users use API under v2.14. And we found that nova 100% fail
to evacuate a server via novaclient, the error log as follows:
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher [req-b34b79d6-20ed-4488-b8d5-98f4007eb12e 681041a7364d4852930021d009c8dc2b bafaf53fac4346b9bcd6a77cf964b7af - - -] Exception during message handling: Invalid state of instance files on shared storage
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher payload)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2809, in rebuild_instance
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2853, in _do_rebuild_instance_with_claim
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs)
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2885, in _do_rebuild_instance
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher _("Invalid state of instance files on shared"
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage
2017-02-16 20:22:48.106 6144 ERROR oslo_messaging.rpc.dispatcher
I dug into the API source, and I found that the onSharedStorage will
force to be converted to False value, even if in fact its value is
None:
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/evacuate.py#L43-L48
In other word, If we didn't pass this arguments, we will actually pass
the False value. If we use shared-storage, it will not match the value
getting from the backend driver.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1665548/+subscriptions
Follow ups