yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1818544] [NEW] openstack instance not able to scp files to outside network

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sohny <sohnythomas@xxxxxxxx>
Date: Mon, 04 Mar 2019 16:54:25 -0000
Reply-to: Bug 1818544 <1818544@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have a openstack setup which has newton version installed on CentOS
7.3. I am able to successfuly create VMs and associate floating Ips.
Successfuly able to ssh into and out of VM from external network but I
am not able to SCP any file out of the VM instance . SCP into the VM
instance is fine. SCP btw VM instances are also fine

Below is a 70kb file for which transfer is still going on after 2 hours.
At the destination only 10kb has been copied.

[root@test-server ~]# scp  /var/cache/jenkins/war/WEB-
INF/lib/remoting-3.29.jar
dscadmin@1.20.28.146:/home/dscadmin/jenkins/remoting-3.29.jar
100%  771KB  35.0KB/s   00:22    [


System SPECs:

[root@newton-1 neutron]# openstack --version
openstack 3.2.1

[root@newton-1 neutron]# rpm -qa|grep neutron
openstack-neutron-common-9.4.1-1.el7.noarch
openstack-neutron-openvswitch-9.4.1-1.el7.noarch
puppet-neutron-9.5.0-1.el7.noarch
openstack-neutron-ml2-9.4.1-1.el7.noarch
python-neutron-9.4.1-1.el7.noarch
openstack-neutron-metering-agent-9.4.1-1.el7.noarch
python-neutron-lib-0.4.0-1.el7.noarch
openstack-neutron-9.4.1-1.el7.noarch
python2-neutronclient-6.0.0-2.el7.noarch

[root@newton-1 neutron]# uname -r
3.10.0-514.26.2.el7.x86_64


Following are some log files i tracked for this at /var/log/neutron

[root@newton-1 neutron]# tail -f server.log
2019-03-04 11:54:28.737 17152 INFO neutron.wsgi [req-de76d2a0-72c1-4d90-abf9-42e03aa9c76c bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=113e2ce7-475b-43ea-9765-7cce7565e639&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.035162
2019-03-04 11:54:28.749 17152 INFO neutron.wsgi [req-af4b081a-c900-4cd4-ad34-f64d19b57048 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.60.8&port_id=171c329c-e945-4bf4-bc05-82177a776c72 HTTP/1.1" 200 217 0.009857
2019-03-04 11:54:28.801 17152 INFO neutron.wsgi [req-44ab23c8-84dd-4176-865b-c47a1bc288c0 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/subnets.json?id=83a71783-dcc3-4d8a-8560-89972e03bab5 HTTP/1.1" 200 863 0.049811
2019-03-04 11:54:28.837 17152 INFO neutron.wsgi [req-bb98b71c-f5ca-4be1-bff9-bd7795c38ffa bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=bab474bb-88b7-490f-8c5c-19b08d758a02&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.034647
2019-03-04 11:54:28.849 17152 INFO neutron.wsgi [req-ce60d323-075b-4b30-9104-e708c22c44b8 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.61.9&port_id=5864f529-8cb6-4d30-b3a3-c76be8750d3a HTTP/1.1" 200 217 0.009498
2019-03-04 11:54:28.901 17152 INFO neutron.wsgi [req-384ae67c-c463-4815-be96-8b1710d88d21 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/subnets.json?id=95e5bc60-0309-401c-87a9-b6596957f8a6 HTTP/1.1" 200 863 0.050315
2019-03-04 11:54:28.940 17152 INFO neutron.wsgi [req-6fb418f2-1640-4daf-ad98-a9a8e870f6bd bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=3950ea82-c375-4fcc-bc19-9e01d92681fd&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.037263
2019-03-04 11:54:28.951 17152 INFO neutron.wsgi [req-5e7b3c25-3c32-4b97-a595-3b548daec6ae bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.63.5&port_id=983c2442-ecab-49aa-988a-775fc70a9d10 HTTP/1.1" 200 217 0.009744
2019-03-04 11:54:29.000 17152 INFO neutron.wsgi [req-3e7fd50b-9fb1-4c72-9072-a0aa027f547c bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:29] "GET /v2.0/subnets.json?id=24c11aff-9785-4ed2-b33c-b98b037c24dc HTTP/1.1" 200 863 0.046999
2019-03-04 11:54:29.037 17152 INFO neutron.wsgi [req-2d71b123-b2aa-47d6-95db-bb254032fd7f bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:29] "GET /v2.0/ports.json?network_id=bcfd0431-aec1-42b0-98ad-b7155c836b97&device_owner=network%3Adhcp HTTP/1.1" 200 1118 0.035438

[root@newton-1 neutron]# tail -f dhcp-agent.log
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     'value': value}])
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 912, in execute
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     log_fail_as_error=log_fail_as_error, **kwargs)
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 148, in execute
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     raise ProcessExecutionError(msg, returncode=returncode)
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra: No such file or directory
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [-] Finished network d2b60324-af2b-458e-bc98-21026e00c953 dhcp configuration
2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [req-c7d6f7cb-4229-4143-8b50-a1fb3af11b13 - - - - -] Synchronizing state complete


Let me know if anymore info is required. This is a really weird scenario
which I am able to reproduce with any instance created on this openstack
host.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1818544

Title:
  openstack instance not able to scp files to outside network

Status in neutron:
  New

Bug description:
  I have a openstack setup which has newton version installed on CentOS
  7.3. I am able to successfuly create VMs and associate floating Ips.
  Successfuly able to ssh into and out of VM from external network but I
  am not able to SCP any file out of the VM instance . SCP into the VM
  instance is fine. SCP btw VM instances are also fine

  Below is a 70kb file for which transfer is still going on after 2
  hours. At the destination only 10kb has been copied.

  [root@test-server ~]# scp  /var/cache/jenkins/war/WEB-
  INF/lib/remoting-3.29.jar
  dscadmin@1.20.28.146:/home/dscadmin/jenkins/remoting-3.29.jar
  100%  771KB  35.0KB/s   00:22    [

  
  System SPECs:

  [root@newton-1 neutron]# openstack --version
  openstack 3.2.1

  [root@newton-1 neutron]# rpm -qa|grep neutron
  openstack-neutron-common-9.4.1-1.el7.noarch
  openstack-neutron-openvswitch-9.4.1-1.el7.noarch
  puppet-neutron-9.5.0-1.el7.noarch
  openstack-neutron-ml2-9.4.1-1.el7.noarch
  python-neutron-9.4.1-1.el7.noarch
  openstack-neutron-metering-agent-9.4.1-1.el7.noarch
  python-neutron-lib-0.4.0-1.el7.noarch
  openstack-neutron-9.4.1-1.el7.noarch
  python2-neutronclient-6.0.0-2.el7.noarch

  [root@newton-1 neutron]# uname -r
  3.10.0-514.26.2.el7.x86_64

  
  Following are some log files i tracked for this at /var/log/neutron

  [root@newton-1 neutron]# tail -f server.log
  2019-03-04 11:54:28.737 17152 INFO neutron.wsgi [req-de76d2a0-72c1-4d90-abf9-42e03aa9c76c bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=113e2ce7-475b-43ea-9765-7cce7565e639&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.035162
  2019-03-04 11:54:28.749 17152 INFO neutron.wsgi [req-af4b081a-c900-4cd4-ad34-f64d19b57048 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.60.8&port_id=171c329c-e945-4bf4-bc05-82177a776c72 HTTP/1.1" 200 217 0.009857
  2019-03-04 11:54:28.801 17152 INFO neutron.wsgi [req-44ab23c8-84dd-4176-865b-c47a1bc288c0 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/subnets.json?id=83a71783-dcc3-4d8a-8560-89972e03bab5 HTTP/1.1" 200 863 0.049811
  2019-03-04 11:54:28.837 17152 INFO neutron.wsgi [req-bb98b71c-f5ca-4be1-bff9-bd7795c38ffa bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=bab474bb-88b7-490f-8c5c-19b08d758a02&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.034647
  2019-03-04 11:54:28.849 17152 INFO neutron.wsgi [req-ce60d323-075b-4b30-9104-e708c22c44b8 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.61.9&port_id=5864f529-8cb6-4d30-b3a3-c76be8750d3a HTTP/1.1" 200 217 0.009498
  2019-03-04 11:54:28.901 17152 INFO neutron.wsgi [req-384ae67c-c463-4815-be96-8b1710d88d21 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/subnets.json?id=95e5bc60-0309-401c-87a9-b6596957f8a6 HTTP/1.1" 200 863 0.050315
  2019-03-04 11:54:28.940 17152 INFO neutron.wsgi [req-6fb418f2-1640-4daf-ad98-a9a8e870f6bd bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=3950ea82-c375-4fcc-bc19-9e01d92681fd&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.037263
  2019-03-04 11:54:28.951 17152 INFO neutron.wsgi [req-5e7b3c25-3c32-4b97-a595-3b548daec6ae bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.63.5&port_id=983c2442-ecab-49aa-988a-775fc70a9d10 HTTP/1.1" 200 217 0.009744
  2019-03-04 11:54:29.000 17152 INFO neutron.wsgi [req-3e7fd50b-9fb1-4c72-9072-a0aa027f547c bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:29] "GET /v2.0/subnets.json?id=24c11aff-9785-4ed2-b33c-b98b037c24dc HTTP/1.1" 200 863 0.046999
  2019-03-04 11:54:29.037 17152 INFO neutron.wsgi [req-2d71b123-b2aa-47d6-95db-bb254032fd7f bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:29] "GET /v2.0/ports.json?network_id=bcfd0431-aec1-42b0-98ad-b7155c836b97&device_owner=network%3Adhcp HTTP/1.1" 200 1118 0.035438

  [root@newton-1 neutron]# tail -f dhcp-agent.log
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     'value': value}])
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 912, in execute
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     log_fail_as_error=log_fail_as_error, **kwargs)
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 148, in execute
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     raise ProcessExecutionError(msg, returncode=returncode)
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra: No such file or directory
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
  2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
  2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [-] Finished network d2b60324-af2b-458e-bc98-21026e00c953 dhcp configuration
  2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [req-c7d6f7cb-4229-4143-8b50-a1fb3af11b13 - - - - -] Synchronizing state complete


  
  Let me know if anymore info is required. This is a really weird scenario which I am able to reproduce with any instance created on this openstack host.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1818544/+subscriptions
Follow ups

[Bug 1818544] Re: openstack instance not able to scp files to outside network
From: Sohny, 2019-03-04