Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1614817

Summary: Error: Systemd start for neutron-destroy-patch-ports failed! No module named compat
Product: Red Hat OpenStack Reporter: Filip Hubík <fhubik>
Component: openvswitchAssignee: Flavio Leitner <fleitner>
Status: CLOSED ERRATA QA Contact: Filip Hubík <fhubik>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: apevec, bcafarel, bhaley, chrisw, fhubik, fleitner, jschluet, mburns, rhos-maint, srevivo, tredaelli, tvignaud, twilson
Target Milestone: betaKeywords: Automation, AutomationBlocker, Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:51:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log
none
/var/tmp/packstack/20180810-085725-cBTrNd/openstack-setup.log
none
Add compat module to ovs package in setup.py none

Description Filip Hubík 2018-08-10 13:35:28 UTC
Created attachment 1475020 [details]
/var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log

Description of problem:

Packstack deployment fails with error:
...
Preparing Puppet manifests                           [ DONE ]
Copying Puppet modules and manifests                 [ DONE ]
Applying 172.16.1.19_controller.pp
172.16.1.19_controller.pp:                           [ DONE ]
Applying 172.16.1.19_network.pp
172.16.1.19_network.pp:                           [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 172.16.1.19_network.pp
Error: Systemd start for neutron-destroy-patch-ports failed!
You will find full trace in log /var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log
Please check log file /var/tmp/packstack/20180810-085725-cBTrNd/openstack-setup.log for more information
Additional information:
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * Warning: NetworkManager is active on 172.16.1.19. OpenStack networking currently does not work on systems that have the Network Manager service enabled.
 * File /root/keystonerc_admin has been created on OpenStack client host 172.16.1.19. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://172.16.1.19/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * Note temporary directory /var/tmp/packstack/0ebbed1dbe2e4207991645dacbcf5238 on host 172.16.1.19 was not deleted for debugging purposes.

MSG:

non-zero return code

Error(s) in /var/tmp/packstack/20180810-085725-cBTrNd/manifests/172.16.1.19_network.pp.log:
...
debug: Runing journalctl command to get logs for systemd start failure: journalctl -n 50 --since '5 minutes ago' -u neutron-destroy-patch-ports --no-pager
Debug: Executing: 'journalctl -n 50 --since '5 minutes ago' -u neutron-destroy-patch-ports --no-pager'
Error: Systemd start for neutron-destroy-patch-ports failed!
journalctl log for neutron-destroy-patch-ports:
-- Logs begin at Fri 2018-08-10 08:51:57 EDT, end at Fri 2018-08-10 09:14:56 EDT. --
systemd[1]: Starting OpenStack Neutron Destroy Patch Ports...
python[5110]: /usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version!
python[5110]: RequestsDependencyWarning)
python[5110]: PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
python[5110]: PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)
python[5110]: PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
python[5110]: PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4)
python[5110]: Traceback (most recent call last):
python[5110]: File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
python[5110]: "__main__", fname, loader, pkg_name)
python[5110]: File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
python[5110]: exec code in run_globals
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 83, in <module>
python[5110]: main()
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 78, in main
python[5110]: port_cleaner = PatchPortCleaner(cfg.CONF)
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 44, in __init__
python[5110]: for bridge in mappings.values()]
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 218, in __init__
python[5110]: super(OVSBridge, self).__init__()
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 117, in __init__
python[5110]: self.ovsdb = ovsdb_api.from_config(self)
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 30, in from_config
python[5110]: ovsdb_api.interface_map[iface_name or cfg.CONF.OVS.ovsdb_interface])
python[5110]: File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 73, in import_module
python[5110]: __import__(import_str)
python[5110]: File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 17, in <module>
python[5110]: from ovsdbapp.backend.ovs_idl import command
python[5110]: File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 16, in <module>
python[5110]: from ovsdbapp.backend.ovs_idl import command as cmd
python[5110]: File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 21, in <module>
python[5110]: from ovsdbapp.backend.ovs_idl import idlutils
python[5110]: File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 21, in <module>
python[5110]: from ovs.db import idl
python[5110]: File "/usr/lib64/python2.7/site-packages/ovs/db/idl.py", line 25, in <module>
python[5110]: from ovs.db import custom_index
python[5110]: File "/usr/lib64/python2.7/site-packages/ovs/db/custom_index.py", line 12, in <module>
python[5110]: from ovs.compat import sortedcontainers
python[5110]: ImportError: No module named compat
systemd[1]: neutron-destroy-patch-ports.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start OpenStack Neutron Destroy Patch Ports.
systemd[1]: Unit neutron-destroy-patch-ports.service entered failed state.
systemd[1]: neutron-destroy-patch-ports.service failed.
...

Version-Release number of selected component (if applicable):
2018-08-10.2

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP14 using packstack, InfraRed, topology AIO
2. Packstack deployments fails

Additional info:
openstack-packstack-13.0.0-0.20180731101322.73e0b07.el7ost.noarch
openstack-packstack-puppet-13.0.0-0.20180731101322.73e0b07.el7ost.noarch
python-neutron-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
openstack-neutron-openvswitch-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
python2-neutron-lib-1.18.0-0.20180725152803.67865c7.el7ost.noarch
puppet-neutron-13.2.0-0.20180805213727.2a42bb2.el7ost.noarch
openstack-neutron-common-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
openstack-neutron-13.0.0-0.20180803061300.765fd4d.el7ost.noarch
python2-neutronclient-6.9.0-0.20180803115753.d090ea2.el7ost.noarch

Comment 1 Filip Hubík 2018-08-10 13:36:12 UTC
Created attachment 1475021 [details]
/var/tmp/packstack/20180810-085725-cBTrNd/openstack-setup.log

Comment 2 Filip Hubík 2018-08-10 13:38:51 UTC
Created attachment 1475022 [details]
answer file

Comment 5 Terry Wilson 2018-08-10 15:03:59 UTC
Created attachment 1475105 [details]
Add compat module to ovs package in setup.py

This is happening because the python/setup.py forgot to include the compat/compat.sortedcontainers directory. I tested a quick build with the patch applied and it added the appropriate directories to the RPM. I'll go ahead and propose the patch upstream as well.

Comment 6 Terry Wilson 2018-08-10 15:14:42 UTC
Patch proposed upstream at https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350966.html

Comment 11 Terry Wilson 2018-08-10 20:17:26 UTC
The patch has been merged upstream in both master and branch-2.10.

Comment 12 Flavio Leitner 2018-08-10 21:54:40 UTC
openvswitch2.0 package including the package is available:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=17734563

Please verify.
fbl

Comment 13 Terry Wilson 2018-08-10 23:36:47 UTC
After installing the fixed rpm from, doing from ovs.compat import sortedcontainers succeeds.

Comment 15 Filip Hubík 2018-08-14 11:52:36 UTC
I can not confirm that this issue is fixed by openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp (it can be kept in version openvswitch2.10-2.10.0-0.20180808gitfaf64fb.el7fdp)
but it is fixed by update of python-openvswitch2.10 to version python-openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.

I adjusted "Fixed in Version" field.

Comment 16 Assaf Muller 2018-08-14 14:28:18 UTC
*** Bug 1615892 has been marked as a duplicate of this bug. ***

Comment 18 Filip Hubík 2018-08-20 09:46:51 UTC
I can verify this is not issue since puddle 2018-08-17.2 where fixed package is included:

packtack-aio $ rpm -qa | grep openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fd
python-openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64

Comment 24 errata-xmlrpc 2019-01-11 11:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045