Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1165692
Summary: | After auto-heal, entitlement certs having "containerImage" content type is not updated in hostname directories | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Rehana <redakkan> | ||||||
Component: | subscription-manager | Assignee: | candlepin-bugs | ||||||
Status: | CLOSED WORKSFORME | QA Contact: | Shwetha Kallesh <skallesh> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | high | ||||||||
Version: | 7.1 | CC: | alikins, bcourt, crog, jsefler, lmiksik, mgrepl, mmalik, redakkan, skallesh, wpoteat | ||||||
Target Milestone: | rc | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | subscription-manager-1.15.5-1 | Doc Type: | Bug Fix | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2015-09-30 16:12:59 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1121117 | ||||||||
Attachments: |
|
Description
Rehana
2014-11-19 13:52:57 UTC
This seems to work with 1.15.5-1 Failed QA [root@dhcp35-98 ~]# subscription-manager version server type: Red Hat Subscription Management subscription management server: 0.9.26.11-1 subscription management rules: 5.12 subscription-manager: 1.15.9-1.el7 python-rhsm: 1.15.3-1.el7 [root@dhcp35-98 ~]# service rhsmcertd restart Redirecting to /bin/systemctl restart rhsmcertd.service [root@dhcp35-98 ~]# subscription-manager list --consumed +-------------------------------------------+ Consumed Subscriptions +-------------------------------------------+ Subscription Name: Employee SKU Provides: JBoss Enterprise Web Platform Red Hat Enterprise Virtualization for IBM Power Red Hat Enterprise Linux for Power, big endian - Extended Update Support Red Hat Certificate System MRG Management Oracle Java (for RHEL Compute Node) - Extended Update Support Red Hat OpenShift Enterprise Infrastructure Beta Red Hat Enterprise Linux for Real Time Red Hat OpenStack Red Hat Hardware Certification Test Suite Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node) Red Hat Certificate System with Advanced Access Red Hat JBoss A-MQ Clients Red Hat Enterprise Linux 7 for HPC Compute Node High Touch Beta JBoss Enterprise Application Platform Red Hat OpenShift Enterprise JBoss FUSE add-on MRG Grid Execute Red Hat Enterprise Linux Server Oracle Java (for RHEL Workstation) Red Hat Enterprise Linux for Power, big endian Red Hat Enterprise Linux EUS Compute Node Red Hat Ceph Storage MON Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER) Red Hat Software Collections (for RHEL Workstation) Red Hat OpenShift Enterprise Application Node Beta Red Hat Enterprise Linux Scalable File System (for RHEL Workstation) JBoss Enterprise Application Platform - ELS Red Hat Enterprise Linux 7 for IBM POWER High Touch Beta Red Hat OpenShift Enterprise JBoss A-MQ add-on Red Hat Enterprise Linux EUS Compute Node High Performance Networking Red Hat Gluster Storage Server for On-premise Red Hat Enterprise Linux Atomic Host HTB Red Hat Gluster Storage Nagios Server Red Hat Ceph Storage Calamari Red Hat OpenStack Beta Red Hat Enterprise Linux for IBM z Systems - Extended Update Support Red Hat Directory Server MRG Realtime Red Hat Enterprise MRG Messaging 3 for RHEL 7 Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support Red Hat OpenShift Enterprise Infrastructure Red Hat Enterprise Linux High Availability (for RHEL Server) Red Hat Enterprise Linux EUS Compute Node Scalable File System Red Hat Enterprise Linux for Power, little endian Red Hat Enterprise Linux 7 for IBM z Systems High Touch Beta JBoss Enterprise Web Server Red Hat Developer Toolset (for RHEL Workstation) Red Hat Enterprise Linux Server - AUS Red Hat Enterprise Linux for SAP Red Hat Enterprise Linux for IBM z Systems Red Hat Enterprise Linux Atomic Host Red Hat Enterprise Linux 7 Desktop High Touch Beta Oracle Java (for RHEL Server) - Extended Update Support Red Hat Storage Red Hat Software Collections (for RHEL Server) Red Hat OpenShift Enterprise Application Node Red Hat Enterprise Linux Scalable File System (for RHEL Server) Red Hat OpenShift Enterprise Red Hat OpenStack Beta Certification Test Suite Red Hat Enterprise Linux High Performance Networking (for RHEL Server) Kernel Derivative Works for HPC for Power Systems Red Hat Enterprise Linux Workstation Red Hat Gluster Storage Management Console (for RHEL Server) Oracle Java (for RHEL Client) Red Hat Enterprise Linux High Availability (for RHEL Server) - AUS Red Hat Enterprise MRG Messaging Red Hat Beta Red Hat Enterprise Linux Atomic Host Beta MRG Grid Red Hat Enterprise Linux 7 Workstation High Touch Beta Red Hat Enterprise Linux Load Balancer (for RHEL Server) Red Hat Software Collections Beta (for RHEL Client) Red Hat S-JIS Support (for RHEL Server) Red Hat Enterprise Virtualization Red Hat Container Development Kit Red Hat Enterprise Linux Server - Extended Update Support Red Hat Enterprise Linux 7 Resilient Storage High Touch Beta Red Hat Enterprise Linux Load Balancer (for RHEL Server) - AUS Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support Red Hat Enterprise Linux 7 Server High Touch Beta Red Hat Enterprise Linux 6 Server HTB Red Hat Container Images Red Hat Cloud Infrastructure Red Hat Software Collections Beta (for RHEL Workstation) Oracle Java (for RHEL Compute Node) Red Hat CloudForms Red Hat Developer Toolset (for RHEL Server EUS) Red Hat Enterprise Linux Desktop Red Hat Enterprise Linux 7 Load Balancer High Touch Beta Red Hat Enterprise Linux Resilient Storage (for RHEL Server) Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - AUS Red Hat Developer Toolset (for RHEL Server) Red Hat Enterprise Linux Server - Extended Life Cycle Support Red Hat Ceph Storage Red Hat Container Images Beta Red Hat Enterprise Linux Server for ARM Beta Red Hat OpenShift Enterprise Client Tools Beta Red Hat OpenShift Enterprise Client Tools Red Hat Enterprise Linux for SAP Hana Red Hat EUCJP Support (for RHEL Server) Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support Red Hat Enterprise Linux 7 High Availability High Touch Beta Oracle Java (for RHEL Server) - AUS Red Hat Software Collections Beta (for RHEL Server) Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node) Red Hat Enterprise Linux Scalable File System (for RHEL Server) - AUS Red Hat Container Images HTB Red Hat Enterprise Linux 6 Workstation HTB Red Hat Enterprise Linux for Scientific Computing Red Hat Enterprise Linux Server for ARM Development Preview Kernel Derivative Works for Bluegene/Q Red Hat OpenShift Enterprise JBoss EAP add-on Beta Red Hat Ceph Storage OSD Red Hat OpenShift Enterprise JBoss EAP add-on SKU: ES0113909 Contract: 10169793 Account: 477931 Serial: 6967812494487310609 Pool ID: 8a85f9823e3d5e43013e3ddd4e9509c4 Provides Management: Yes Active: True Quantity Used: 1 Service Level: Self-Support Service Type: L1-L3 Status Details: Subscription is current Subscription Type: Standard Starts: 04/24/2013 Ends: 01/01/2022 System Type: Virtual [root@dhcp35-98 ~]# rct cc /etc/pki/entitlement/6967812494487310609.pem | grep "container" Type: containerimage Label: rhel-6-server-beta-containers URL: /content/beta/rhel/server/6/6Server/x86_64/containers Type: containerimage Label: rhel-6-server-containers URL: /content/dist/rhel/server/6/6Server/x86_64/containers Type: containerimage Label: rhel-6-server-htb-containers URL: /content/htb/rhel/server/6/6Server/x86_64/containers Type: containerimage Label: rhel-7-server-beta-containers URL: /content/beta/rhel/server/7/x86_64/containers Type: containerimage Label: rhel-7-server-containers URL: /content/dist/rhel/server/7/7Server/x86_64/containers Type: containerimage Label: rhel-7-server-htb-containers URL: /content/htb/rhel/server/7/x86_64/containers [root@dhcp35-98 ~]# cat /etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf [main] enabled = 1 registry_hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com [root@dhcp35-98 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/ [root@dhcp35-98 ~]# ls /etc/docker/certs.d/cdn.redhat.com/ redhat-entitlement-authority.crt [root@dhcp35-98 ~]# ls /etc/docker/certs.d/access.redhat.com/ ^^ no entitlements certs are listed in the hostname directories -------------------------------------------------------------------------------- Now I remove the subscription attached above and manually attach the same pool [root@dhcp35-98 ~]# subscription-manager remove --all 1 subscription removed at the server. 1 local certificate has been deleted. [root@dhcp35-98 ~]# subscription-manager attach --pool 8a85f9823e3d5e43013e3ddd4e9509c4 Successfully attached a subscription for: Employee SKU [root@dhcp35-98 ~]# ls /etc/docker/certs.d/access.redhat.com/ 3606093036906710716.cert 3606093036906710716.key [root@dhcp35-98 ~]# ls /etc/docker/certs.d/cdn.redhat.com/ 3606093036906710716.cert 3606093036906710716.key redhat-entitlement-authority.crt [root@dhcp35-98 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/ 3606093036906710716.cert 3606093036906710716.key ^^^ hostname directories are updated with ent certs any rhsm.log's from when rhsmcertd auto-healed? Having trouble reproducing the failure, attaching an employee sku from access.stage, and then letting rhsmd pick it up creates the container certificate links for me. Logs from rhsmd runs where that fails would get me closer. Created attachment 1061368 [details]
rhsm.log for updation of certs via auto-heal
Auto-heal:
[root@dhcp35-140 ~]# subscription-manager register
Registering to: subscription.rhn.stage.redhat.com/subscription:443
Username: qa
Password:
The system has been registered with ID: 86e8c677-f6c9-4a49-b2ad-c73d93d4c85a
[root@dhcp35-140 ~]# service rhsmcertd restart
Redirecting to /bin/systemctl restart rhsmcertd.service
[root@dhcp35-140 ~]# subscription-manager list --consumed
+-------------------------------------------+
Consumed Subscriptions
+-------------------------------------------+
Subscription Name: Employee SKU
Provides: JBoss Enterprise Web Platform
Oracle Java (for Middleware)
Red Hat Enterprise Virtualization for IBM Power
Red Hat Enterprise Linux for Power, big endian - Extended Update Support
Red Hat Certificate System
MRG Management
Oracle Java (for RHEL Compute Node) - Extended Update Support
Red Hat OpenShift Enterprise Infrastructure Beta
Red Hat Enterprise Linux for Real Time
Red Hat OpenStack
Red Hat Hardware Certification Test Suite
Red Hat Certificate System with Advanced Access
Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
Red Hat JBoss A-MQ Clients
Red Hat Enterprise Linux 7 for HPC Compute Node High Touch Beta
JBoss Enterprise Application Platform
Red Hat OpenShift Enterprise JBoss FUSE add-on
MRG Grid Execute
Oracle Java (for RHEL Workstation)
Red Hat Enterprise Linux Server
Red Hat Enterprise Linux for Power, big endian
Red Hat Enterprise Linux EUS Compute Node
Red Hat Ceph Storage MON
Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER)
Red Hat Software Collections (for RHEL Workstation)
Red Hat OpenShift Enterprise Application Node Beta
Red Hat Enterprise Linux Scalable File System (for RHEL Workstation)
JBoss Enterprise Application Platform - ELS
Red Hat Enterprise Linux 7 for IBM POWER High Touch Beta
Red Hat OpenShift Enterprise JBoss A-MQ add-on
Red Hat Enterprise Linux EUS Compute Node High Performance Networking
Red Hat Gluster Storage Server for On-premise
Atomic Enterprise Platform Early Access
Red Hat Gluster Storage Nagios Server
Red Hat Enterprise Linux Atomic Host HTB
Red Hat Ceph Storage Calamari
Red Hat Enterprise Linux for IBM z Systems - Extended Update Support
Red Hat OpenStack Beta
Red Hat Directory Server
MRG Realtime
Red Hat Enterprise MRG Messaging 3 for RHEL 7
Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
Red Hat OpenShift Enterprise Infrastructure
Red Hat Enterprise Linux High Availability (for RHEL Server)
Red Hat Enterprise Linux EUS Compute Node Scalable File System
Red Hat Enterprise Linux for Power, little endian
Red Hat Enterprise Linux 7 for IBM z Systems High Touch Beta
Red Hat Developer Toolset (for RHEL Workstation)
JBoss Enterprise Web Server
Red Hat Enterprise Linux Server - AUS
Red Hat Enterprise Linux for SAP
Red Hat Enterprise Linux for IBM z Systems
Red Hat Enterprise Linux Atomic Host
Red Hat Storage
Oracle Java (for RHEL Server) - Extended Update Support
Red Hat Enterprise Linux 7 Desktop High Touch Beta
Red Hat Software Collections (for RHEL Server)
Red Hat OpenShift Enterprise Application Node
Red Hat Enterprise Linux Scalable File System (for RHEL Server)
Red Hat OpenShift Enterprise
Red Hat OpenStack Beta Certification Test Suite
Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
Kernel Derivative Works for HPC for Power Systems
Red Hat Enterprise Linux Workstation
Red Hat Gluster Storage Management Console (for RHEL Server)
Oracle Java (for RHEL Client)
Red Hat Enterprise Linux High Availability (for RHEL Server) - AUS
Red Hat Enterprise MRG Messaging
Red Hat Beta
MRG Grid
Red Hat Enterprise Linux Atomic Host Beta
Red Hat Enterprise Linux 7 Workstation High Touch Beta
Red Hat Software Collections Beta (for RHEL Client)
Red Hat Enterprise Linux Load Balancer (for RHEL Server)
Red Hat S-JIS Support (for RHEL Server)
Red Hat Enterprise Virtualization
Red Hat Container Development Kit
Red Hat Enterprise Linux Server - Extended Update Support
Red Hat Enterprise Linux 7 Resilient Storage High Touch Beta
Red Hat Enterprise Linux Load Balancer (for RHEL Server) - AUS
Red Hat Enterprise Linux 7 Server High Touch Beta
Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
Red Hat Enterprise Linux 6 Server HTB
Red Hat Container Images
Red Hat Cloud Infrastructure
Red Hat Software Collections Beta (for RHEL Workstation)
Oracle Java (for RHEL Compute Node)
Red Hat CloudForms
Red Hat Developer Toolset (for RHEL Server EUS)
Red Hat Enterprise Linux Desktop
Red Hat Enterprise Linux 7 Load Balancer High Touch Beta
Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
Red Hat Developer Toolset (for RHEL Server)
Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - AUS
Red Hat Enterprise Linux Server - Extended Life Cycle Support
Red Hat Ceph Storage
Red Hat Container Images Beta
Red Hat Enterprise Linux Server for ARM Beta
Red Hat OpenShift Enterprise Beta
Red Hat OpenShift Enterprise Client Tools Beta
Red Hat OpenShift Enterprise Client Tools
Red Hat Enterprise Linux for SAP Hana
Red Hat EUCJP Support (for RHEL Server)
Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
Red Hat Enterprise Linux 7 High Availability High Touch Beta
Oracle Java (for RHEL Server) - AUS
Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
Red Hat Software Collections Beta (for RHEL Server)
Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node)
Red Hat Enterprise Linux Scalable File System (for RHEL Server) - AUS
Red Hat Enterprise Linux 6 Workstation HTB
Red Hat Container Images HTB
Red Hat Enterprise Linux for Scientific Computing
Red Hat Enterprise Linux Server for ARM Development Preview
Kernel Derivative Works for Bluegene/Q
Red Hat OpenShift Enterprise JBoss EAP add-on Beta
Red Hat OpenShift Enterprise JBoss EAP add-on
Red Hat Ceph Storage OSD
SKU: ES0113909
Contract: 10169793
Account: 477931
Serial: 9184007015206335191
Pool ID: 8a85f9823e3d5e43013e3ddd4e9509c4
Provides Management: Yes
Active: True
Quantity Used: 1
Service Level: Self-Support
Service Type: L1-L3
Status Details: Subscription is current
Subscription Type: Standard
Starts: 04/24/2013
Ends: 01/01/2022
System Type: Virtual
[root@dhcp35-140 ~]# rct cc /etc/pki/entitlement/9184007015206335191.pem | grep "container"
Type: containerimage
Label: rhel-6-server-beta-containers
URL: /content/beta/rhel/server/6/6Server/x86_64/containers
Type: containerimage
Label: rhel-6-server-containers
URL: /content/dist/rhel/server/6/6Server/x86_64/containers
Type: containerimage
Label: rhel-6-server-htb-containers
URL: /content/htb/rhel/server/6/6Server/x86_64/containers
Type: containerimage
Label: rhel-7-server-aep-beta-containers
URL: /content/beta/rhel/server/7/$basearch/aep/containers
Type: containerimage
Label: rhel-7-server-beta-containers
URL: /content/beta/rhel/server/7/x86_64/containers
Type: containerimage
Label: rhel-7-server-containers
URL: /content/dist/rhel/server/7/7Server/x86_64/containers
Type: containerimage
Label: rhel-7-server-htb-containers
URL: /content/htb/rhel/server/7/x86_64/containers
Type: containerimage
Label: rhel-server-7-ose-beta-containers
URL: /content/beta/rhel/server/7/$basearch/ose/3/containers
[root@dhcp35-140 ~]# cat /etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf
[main]
enabled = 1
registry_hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/cdn.redhat.com/
redhat-entitlement-authority.crt
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/access.redhat.com/
[root@dhcp35-140 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.9.51.4-1
subscription management rules: 5.15
subscription-manager: 1.15.9-3.el7
python-rhsm: 1.15.4-2.el7
Created attachment 1061370 [details]
rhsm.log for updation of certs by attaching subscription manually
Now remove the subscription attached by rhsmcertd process and attach the same subscription manually , certs are updated
[root@dhcp35-140 ~]# subscription-manager list --consumed
+-------------------------------------------+
Consumed Subscriptions
+-------------------------------------------+
Subscription Name: Employee SKU
Provides: JBoss Enterprise Web Platform
Oracle Java (for Middleware)
Red Hat Enterprise Virtualization for IBM Power
Red Hat Enterprise Linux for Power, big endian - Extended Update Support
Red Hat Certificate System
MRG Management
Oracle Java (for RHEL Compute Node) - Extended Update Support
Red Hat OpenShift Enterprise Infrastructure Beta
Red Hat Enterprise Linux for Real Time
Red Hat OpenStack
Red Hat Hardware Certification Test Suite
Red Hat Certificate System with Advanced Access
Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
Red Hat JBoss A-MQ Clients
Red Hat Enterprise Linux 7 for HPC Compute Node High Touch Beta
JBoss Enterprise Application Platform
Red Hat OpenShift Enterprise JBoss FUSE add-on
MRG Grid Execute
Oracle Java (for RHEL Workstation)
Red Hat Enterprise Linux Server
Red Hat Enterprise Linux for Power, big endian
Red Hat Enterprise Linux EUS Compute Node
Red Hat Ceph Storage MON
Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER)
Red Hat Software Collections (for RHEL Workstation)
Red Hat OpenShift Enterprise Application Node Beta
Red Hat Enterprise Linux Scalable File System (for RHEL Workstation)
JBoss Enterprise Application Platform - ELS
Red Hat Enterprise Linux 7 for IBM POWER High Touch Beta
Red Hat OpenShift Enterprise JBoss A-MQ add-on
Red Hat Enterprise Linux EUS Compute Node High Performance Networking
Red Hat Gluster Storage Server for On-premise
Atomic Enterprise Platform Early Access
Red Hat Gluster Storage Nagios Server
Red Hat Enterprise Linux Atomic Host HTB
Red Hat Ceph Storage Calamari
Red Hat Enterprise Linux for IBM z Systems - Extended Update Support
Red Hat OpenStack Beta
Red Hat Directory Server
MRG Realtime
Red Hat Enterprise MRG Messaging 3 for RHEL 7
Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
Red Hat OpenShift Enterprise Infrastructure
Red Hat Enterprise Linux High Availability (for RHEL Server)
Red Hat Enterprise Linux EUS Compute Node Scalable File System
Red Hat Enterprise Linux for Power, little endian
Red Hat Enterprise Linux 7 for IBM z Systems High Touch Beta
Red Hat Developer Toolset (for RHEL Workstation)
JBoss Enterprise Web Server
Red Hat Enterprise Linux Server - AUS
Red Hat Enterprise Linux for SAP
Red Hat Enterprise Linux for IBM z Systems
Red Hat Enterprise Linux Atomic Host
Red Hat Storage
Oracle Java (for RHEL Server) - Extended Update Support
Red Hat Enterprise Linux 7 Desktop High Touch Beta
Red Hat Software Collections (for RHEL Server)
Red Hat OpenShift Enterprise Application Node
Red Hat Enterprise Linux Scalable File System (for RHEL Server)
Red Hat OpenShift Enterprise
Red Hat OpenStack Beta Certification Test Suite
Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
Kernel Derivative Works for HPC for Power Systems
Red Hat Enterprise Linux Workstation
Red Hat Gluster Storage Management Console (for RHEL Server)
Oracle Java (for RHEL Client)
Red Hat Enterprise Linux High Availability (for RHEL Server) - AUS
Red Hat Enterprise MRG Messaging
Red Hat Beta
MRG Grid
Red Hat Enterprise Linux Atomic Host Beta
Red Hat Enterprise Linux 7 Workstation High Touch Beta
Red Hat Software Collections Beta (for RHEL Client)
Red Hat Enterprise Linux Load Balancer (for RHEL Server)
Red Hat S-JIS Support (for RHEL Server)
Red Hat Enterprise Virtualization
Red Hat Container Development Kit
Red Hat Enterprise Linux Server - Extended Update Support
Red Hat Enterprise Linux 7 Resilient Storage High Touch Beta
Red Hat Enterprise Linux Load Balancer (for RHEL Server) - AUS
Red Hat Enterprise Linux 7 Server High Touch Beta
Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
Red Hat Enterprise Linux 6 Server HTB
Red Hat Container Images
Red Hat Cloud Infrastructure
Red Hat Software Collections Beta (for RHEL Workstation)
Oracle Java (for RHEL Compute Node)
Red Hat CloudForms
Red Hat Developer Toolset (for RHEL Server EUS)
Red Hat Enterprise Linux Desktop
Red Hat Enterprise Linux 7 Load Balancer High Touch Beta
Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
Red Hat Developer Toolset (for RHEL Server)
Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - AUS
Red Hat Enterprise Linux Server - Extended Life Cycle Support
Red Hat Ceph Storage
Red Hat Container Images Beta
Red Hat Enterprise Linux Server for ARM Beta
Red Hat OpenShift Enterprise Beta
Red Hat OpenShift Enterprise Client Tools Beta
Red Hat OpenShift Enterprise Client Tools
Red Hat Enterprise Linux for SAP Hana
Red Hat EUCJP Support (for RHEL Server)
Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
Red Hat Enterprise Linux 7 High Availability High Touch Beta
Oracle Java (for RHEL Server) - AUS
Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
Red Hat Software Collections Beta (for RHEL Server)
Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node)
Red Hat Enterprise Linux Scalable File System (for RHEL Server) - AUS
Red Hat Enterprise Linux 6 Workstation HTB
Red Hat Container Images HTB
Red Hat Enterprise Linux for Scientific Computing
Red Hat Enterprise Linux Server for ARM Development Preview
Kernel Derivative Works for Bluegene/Q
Red Hat OpenShift Enterprise JBoss EAP add-on Beta
Red Hat OpenShift Enterprise JBoss EAP add-on
Red Hat Ceph Storage OSD
SKU: ES0113909
Contract: 10169793
Account: 477931
Serial: 9184007015206335191
Pool ID: 8a85f9823e3d5e43013e3ddd4e9509c4
Provides Management: Yes
Active: True
Quantity Used: 1
Service Level: Self-Support
Service Type: L1-L3
Status Details: Subscription is current
Subscription Type: Standard
Starts: 04/24/2013
Ends: 01/01/2022
System Type: Virtual
[root@dhcp35-140 ~]# subscription-manager remove --serial 9184007015206335191
Serial numbers successfully removed at the server:
9184007015206335191
1 local certificate has been deleted.
[root@dhcp35-140 ~]# subscription-manager attach --pool 8a85f9823e3d5e43013e3ddd4e9509c4
Successfully attached a subscription for: Employee SKU
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
4664871445946251217.cert 4664871445946251217.key
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/cdn.redhat.com/
4664871445946251217.cert 4664871445946251217.key redhat-entitlement-authority.crt
[root@dhcp35-140 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.9.51.4-1
subscription management rules: 5.15
subscription-manager: 1.15.9-3.el7
python-rhsm: 1.15.4-2.el7
Please reproduce and report back if there is a difference in the end state between an auto-heal and manual attach. The log in comment 7 (where it fails) shows: 2015-08-11 14:22:45,543 [DEBUG] rhsmcertd-worker:13430 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x254fc10>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x3595310>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x359a4d0>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x359a550>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x359bd10>] 2015-08-11 14:22:45,543 [DEBUG] rhsmcertd-worker:13430 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com 2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:141 - Exiting plugin 2015-08-11 14:22:45,543 [DEBUG] rhsmcertd-worker:13430 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com 2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:141 - Exiting plugin 2015-08-11 14:22:45,544 [DEBUG] rhsmcertd-worker:13430 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com 2015-08-11 14:22:45,544 [WARNING] rhsmcertd-worker:13430 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-08-11 14:22:45,544 [WARNING] rhsmcertd-worker:13430 @container.py:141 - Exiting plugin The code in src/subscription_manager/plugin/container.py in sync() logs those messages. That code is checking that /etc/docker/certs.d exists, and not populating the container certs. Those paths should be created by the 'subscription-manager-plugin-container' package, the 'rpm -ql': /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/rhsm/ca/redhat-entitlement-authority.pem /etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf /usr/share/rhsm-plugins/container_content.py /usr/share/rhsm-plugins/container_content.pyc /usr/share/rhsm-plugins/container_content.pyo /usr/share/rhsm/subscription_manager/plugin/container.py /usr/share/rhsm/subscription_manager/plugin/container.pyc /usr/share/rhsm/subscription_manager/plugin/container.pyo Do those paths (/etc/docker/certs.d), exist when rhsmd runs? Is /etc/docker/certs.d a directory? (Verify that it is _not_ a symlink or a broken symlink) Are they being removed (or symlinked) as part of testing? Note: The subman version referenced in the original report (comment 0) was before the /etc/docker/certs.d/ paths were adding to subscription-manager-plugin-container, but the versions referenced in later comments do have it. (Any version later than 1.13.18) I agree with comment 11 that the reason auto-heal fails to copy the container entitlements to the registry_hostnames directory is because the /etc/docker/certs.d/ directory appeared to not exist. The mystery is why did it appear not to exist? If the latest docker or subscription-manager-plugin-container is installed then the existence of /etc/docker/certs.d/ is true. Unfortunately.... I can reproduce this bug despite the preexistence of /etc/docker/certs.d/ .... I suspect an selinux denial is blocking this test... [root@jsefler-7 ~]# rpm -q docker docker-selinux selinux-policy subscription-manager-plugin-container docker-1.7.1-115.el7.x86_64 docker-selinux-1.7.1-115.el7.x86_64 selinux-policy-3.13.1-47.el7.noarch subscription-manager-plugin-container-1.15.9-7.el7.x86_64 [root@jsefler-7 ~]# getenforce Enforcing [root@jsefler-7 ~]# rpm -q --whatprovides /etc/docker/certs.d/ docker-1.7.1-115.el7.x86_64 [root@jsefler-7 ~]# ls -l /etc/docker/ total 4 drwxr-xr-x. 5 root root 60 Sep 10 12:59 certs.d -rw-------. 1 root root 281 Aug 18 18:42 key.json [root@jsefler-7 ~]# ls -l /etc/docker/certs.d/ total 0 drwxr-xr-x. 2 root root 45 Sep 10 12:19 cdn.redhat.com drwxr-xr-x. 2 root root 26 Sep 10 12:42 redhat.com drwxr-xr-x. 2 root root 26 Sep 10 12:42 redhat.io [root@jsefler-7 ~]# subscription-manager register --serverurl=subscription.rhn.stage.redhat.com:443/subscription --username=stage_auto_testuser1 Registering to: subscription.rhn.stage.redhat.com:443/subscription Password: The system has been registered with ID: 091c9bf9-8d9b-4a52-bb5a-880843d114a8 [root@jsefler-7 ~]# subscription-manager list --installed +-------------------------------------------+ Installed Product Status +-------------------------------------------+ Product Name: Red Hat Enterprise Linux Atomic Host Product ID: 271 Version: 7 Arch: x86_64 Status: Not Subscribed Status Details: Not supported by a valid subscription. Starts: Ends: Product Name: Red Hat Enterprise Linux Server Product ID: 69 Version: 7.2 Beta Arch: x86_64 Status: Not Subscribed Status Details: Not supported by a valid subscription. Starts: Ends: [root@jsefler-7 ~]# subscription-manager auto-attach --show Auto-attach preference: enabled [root@jsefler-7 ~]# systemctl restart rhsmcertd.service [root@jsefler-7 ~]# sleep 180 [root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log 2015-09-10 13:16:39,663 [DEBUG] rhsmcertd-worker:4151 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin 2015-09-10 13:16:39,664 [DEBUG] rhsmcertd-worker:4151 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x2b81f50> 2015-09-10 13:16:39,664 [INFO] rhsmcertd-worker:4151 @container_content.py:43 - Updating container content. 2015-09-10 13:16:39,665 [INFO] rhsmcertd-worker:4151 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com 2015-09-10 13:16:39,665 [DEBUG] rhsmcertd-worker:4151 @__init__.py:85 - Searching for content of type: containerimage 2015-09-10 13:16:39,665 [DEBUG] rhsmcertd-worker:4151 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x28e1bd0>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x28e1c10>] 2015-09-10 13:16:39,666 [DEBUG] rhsmcertd-worker:4151 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com 2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:141 - Exiting plugin 2015-09-10 13:16:39,666 [DEBUG] rhsmcertd-worker:4151 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com 2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:141 - Exiting plugin 2015-09-10 13:16:39,667 [DEBUG] rhsmcertd-worker:4151 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com 2015-09-10 13:16:39,667 [WARNING] rhsmcertd-worker:4151 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-09-10 13:16:39,667 [WARNING] rhsmcertd-worker:4151 @container.py:141 - Exiting plugin BANG! For some unknown reason the existence of /etc/docker/certs.d/ is not accessible to rhsmcertd-worker running as a service. HOWEVER if I run rhsmcertd-worker manually, it works (Syncing container certificates is successful)... [root@jsefler-7 ~]# /usr/libexec/rhsmcertd-worker --autoheal Updating entitlement certificates & repositories Installed Products status: 0 updates: [] exceptions: Total updates: 1 Found (local) serial# [] Expected (UEP) serial# [4866972821415011331] Added (new) [sn:4866972821415011331 (Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Oracle Java (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Server - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Server,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Atomic Host,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat EUCJP Support (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Oracle Java (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Software Collections (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Beta,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Developer Toolset (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Atomic Host Beta,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Software Collections Beta (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Container Images,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat Container Images Beta,) @ /etc/pki/entitlement/4866972821415011331.pem] [sn:4866972821415011331 (Red Hat S-JIS Support (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem] Deleted (rogue): <NONE> Total updates: 0 Found (local) serial# [4866972821415011331L] Expected (UEP) serial# [4866972821415011331] Added (new) <NONE> Deleted (rogue): <NONE> [root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log 2015-09-10 13:24:49,725 [DEBUG] rhsmcertd-worker:4423 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin 2015-09-10 13:24:49,726 [DEBUG] rhsmcertd-worker:4423 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x164d850> 2015-09-10 13:24:49,726 [INFO] rhsmcertd-worker:4423 @container_content.py:43 - Updating container content. 2015-09-10 13:24:49,726 [INFO] rhsmcertd-worker:4423 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com 2015-09-10 13:24:49,727 [DEBUG] rhsmcertd-worker:4423 @__init__.py:85 - Searching for content of type: containerimage 2015-09-10 13:24:49,727 [DEBUG] rhsmcertd-worker:4423 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x13acc50>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x13acc90>] 2015-09-10 13:24:49,728 [DEBUG] rhsmcertd-worker:4423 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com 2015-09-10 13:24:49,728 [INFO] rhsmcertd-worker:4423 @container.py:164 - Copying: /etc/pki/entitlement/4866972821415011331.pem -> /etc/docker/certs.d/registry.access.redhat.com/4866972821415011331.cert 2015-09-10 13:24:49,729 [INFO] rhsmcertd-worker:4423 @container.py:170 - Copying: /etc/pki/entitlement/4866972821415011331-key.pem -> /etc/docker/certs.d/registry.access.redhat.com/4866972821415011331.key 2015-09-10 13:24:49,730 [DEBUG] rhsmcertd-worker:4423 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com 2015-09-10 13:24:49,730 [INFO] rhsmcertd-worker:4423 @container.py:164 - Copying: /etc/pki/entitlement/4866972821415011331.pem -> /etc/docker/certs.d/cdn.redhat.com/4866972821415011331.cert 2015-09-10 13:24:49,730 [INFO] rhsmcertd-worker:4423 @container.py:170 - Copying: /etc/pki/entitlement/4866972821415011331-key.pem -> /etc/docker/certs.d/cdn.redhat.com/4866972821415011331.key 2015-09-10 13:24:49,731 [DEBUG] rhsmcertd-worker:4423 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com 2015-09-10 13:24:49,732 [INFO] rhsmcertd-worker:4423 @container.py:164 - Copying: /etc/pki/entitlement/4866972821415011331.pem -> /etc/docker/certs.d/access.redhat.com/4866972821415011331.cert 2015-09-10 13:24:49,732 [INFO] rhsmcertd-worker:4423 @container.py:170 - Copying: /etc/pki/entitlement/4866972821415011331-key.pem -> /etc/docker/certs.d/access.redhat.com/4866972821415011331.key 2015-09-10 13:24:49,733 [DEBUG] rhsmcertd-worker:4423 @plugins.py:769 - Running update_content_hook in ostree_content.OstreeContentPlugin 2015-09-10 13:24:49,733 [DEBUG] rhsmcertd-worker:4423 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x164df90> NOTICE above that manually executing rhsmcertd-worker will successfully syncing container certificates to /etc/docker/certs.d/ but allowing the rhsmcertd service to do it fails. UNFORTUNATELY I did not find an denial in /var/log/audit/audit.log during the test. Last attempt with Permissive... [root@jsefler-7 ~]# subscription-manager unsubscribe --all 1 subscription removed at the server. 1 local certificate has been deleted. [root@jsefler-7 ~]# setenforce 0 [root@jsefler-7 ~]# getenforce Permissive [root@jsefler-7 ~]# systemctl restart rhsmcertd.service [root@jsefler-7 ~]# sleep 180 [root@jsefler-7 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/ 254917606347016976.cert 254917606347016976.key [root@jsefler-7 ~]# ls /etc/docker/certs.d/access.redhat.com/ 254917606347016976.cert 254917606347016976.key [root@jsefler-7 ~]# subscription-manager list --consumed | grep Serial Serial: 254917606347016976 YUP, turning off selinux definitely helped avoid the failure. (In reply to John Sefler from comment #12) > I agree with comment 11 that the reason auto-heal fails to copy the > container entitlements to the registry_hostnames directory is because the > /etc/docker/certs.d/ directory appeared to not exist. The mystery is why > did it appear not to exist? If the latest docker or > subscription-manager-plugin-container is installed then the existence of > /etc/docker/certs.d/ is true. > > Unfortunately.... > I can reproduce this bug despite the preexistence of /etc/docker/certs.d/ > .... > I suspect an selinux denial is blocking this test... > Ah. Interesting. Got any logs with the AVC denials? UNFORTUNATELY I did not find any denial in /var/log/audit/audit.log during the test. (for reference) /etc/docker/certs.d is system_u:object_r:cert_t:s0 /etc/pki/consumer unconfined_u:object_r:cert_t:s0 /etc/pki/entitlement unconfined_u:object_r:cert_t:s0 /usr/bin/rhsmcertd is system_u:object_r:rhsmcertd_exec_t:s0 /usr/libexec/rhsmcertd-worker system_u:object_r:bin_t:s0 (So I believe that means that running rhsmcertd-worker via rhsmcertd will have different context that runing rhsmcertd-worker directly, so that doesn't contradict a selinux cause) could this be related to https://bugzilla.redhat.com/show_bug.cgi?id=1262812 Please retest with selinux-policy-3.13.1-51.el7.noarch as verified in bug 1262812 to determine if this fixes our bug. Retesting with versions... [root@jsefler-7 ~]# rpm -q docker docker-selinux selinux-policy subscription-manager-plugin-container docker-1.8.2-2.el7.x86_64 docker-selinux-1.8.2-2.el7.x86_64 selinux-policy-3.13.1-52.el7.noarch subscription-manager-plugin-container-1.15.9-11.el7.x86_64 Unfortunately, I continue to see the same behavior as outlined in comment 12. The fix for Bug 1262812 did not fix this bug too. Seeking NEEDINFO from the selinux-policy experts for help resolving this bug that occurs when selinux is enforcing and does not occur when selinux is permissive. Comment 15 is our current clue for why the rhsmcertd service fails to recognize the existence of directory /etc/docker/certs.d/ for writing files. Do any SELinux denials appear in enforcing mode? # ausearch -m avc -m user_avc -m selinux_err -i -ts today Unfortuinately, NO denials are logged when setenforce Enforcing... [root@jsefler-7 ~]# ausearch -m avc -m user_avc -m selinux_err -i -ts today <no matches> Here is what is written to audit.log at the time of failure... [root@jsefler-7 ~]# tail -f /var/log/audit/audit.log type=USER_ACCT msg=audit(1443551281.197:462903): pid=32478 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=CRED_ACQ msg=audit(1443551281.197:462904): pid=32478 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=LOGIN msg=audit(1443551281.200:462905): pid=32478 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=25045 res=1 type=USER_START msg=audit(1443551281.234:462906): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=CRED_REFR msg=audit(1443551281.235:462907): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=CRED_DISP msg=audit(1443551281.477:462908): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=USER_END msg=audit(1443551281.485:462909): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grant Here what is written to audit.log at the time of failure...ors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' Here is what is written to rhsm.log at the time of failure (while Enforcing)... [root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log 2015-09-29 14:28:32,195 [DEBUG] rhsmcertd-worker:32507 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin 2015-09-29 14:28:32,195 [DEBUG] rhsmcertd-worker:32507 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x2832f10> 2015-09-29 14:28:32,196 [INFO] rhsmcertd-worker:32507 @container_content.py:43 - Updating container content. 2015-09-29 14:28:32,196 [INFO] rhsmcertd-worker:32507 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com 2015-09-29 14:28:32,196 [DEBUG] rhsmcertd-worker:32507 @__init__.py:85 - Searching for content of type: containerimage 2015-09-29 14:28:32,197 [DEBUG] rhsmcertd-worker:32507 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832090>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832d10>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832d90>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832dd0>] 2015-09-29 14:28:32,198 [DEBUG] rhsmcertd-worker:32507 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com 2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:141 - Exiting plugin 2015-09-29 14:28:32,198 [DEBUG] rhsmcertd-worker:32507 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com 2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:141 - Exiting plugin 2015-09-29 14:28:32,198 [DEBUG] rhsmcertd-worker:32507 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com 2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/ 2015-09-29 14:28:32,199 [WARNING] rhsmcertd-worker:32507 @container.py:141 - Exiting plugin And when I setenforce Permissive, notice that there are no more warnings from rhsmcertd-worker... (This is the behavior we want when selinux is Enforcing) [root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log 2015-09-29 14:38:54,090 [DEBUG] rhsmcertd-worker:32748 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin 2015-09-29 14:38:54,090 [DEBUG] rhsmcertd-worker:32748 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x14a2290> 2015-09-29 14:38:54,091 [INFO] rhsmcertd-worker:32748 @container_content.py:43 - Updating container content. 2015-09-29 14:38:54,092 [INFO] rhsmcertd-worker:32748 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com 2015-09-29 14:38:54,092 [DEBUG] rhsmcertd-worker:32748 @__init__.py:85 - Searching for content of type: containerimage 2015-09-29 14:38:54,093 [DEBUG] rhsmcertd-worker:32748 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x11f4a10>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x1219b50>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x1219bd0>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x1219c10>] 2015-09-29 14:38:54,093 [DEBUG] rhsmcertd-worker:32748 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com 2015-09-29 14:38:54,094 [INFO] rhsmcertd-worker:32748 @container.py:164 - Copying: /etc/pki/entitlement/1439947531996000437.pem -> /etc/docker/certs.d/registry.access.redhat.com/1439947531996000437.cert 2015-09-29 14:38:54,094 [INFO] rhsmcertd-worker:32748 @container.py:170 - Copying: /etc/pki/entitlement/1439947531996000437-key.pem -> /etc/docker/certs.d/registry.access.redhat.com/1439947531996000437.key 2015-09-29 14:38:54,095 [DEBUG] rhsmcertd-worker:32748 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com 2015-09-29 14:38:54,095 [INFO] rhsmcertd-worker:32748 @container.py:164 - Copying: /etc/pki/entitlement/1439947531996000437.pem -> /etc/docker/certs.d/cdn.redhat.com/1439947531996000437.cert 2015-09-29 14:38:54,096 [INFO] rhsmcertd-worker:32748 @container.py:170 - Copying: /etc/pki/entitlement/1439947531996000437-key.pem -> /etc/docker/certs.d/cdn.redhat.com/1439947531996000437.key 2015-09-29 14:38:54,096 [DEBUG] rhsmcertd-worker:32748 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com 2015-09-29 14:38:54,096 [INFO] rhsmcertd-worker:32748 @container.py:164 - Copying: /etc/pki/entitlement/1439947531996000437.pem -> /etc/docker/certs.d/access.redhat.com/1439947531996000437.cert 2015-09-29 14:38:54,097 [INFO] rhsmcertd-worker:32748 @container.py:170 - Copying: /etc/pki/entitlement/1439947531996000437-key.pem -> /etc/docker/certs.d/access.redhat.com/1439947531996000437.key Could you temporarily remove dontaudit rules and collect SELinux denials? # semodule -DB your scenario # ausearch -m avc -m user_avc -m selinux_err -i -ts today I'm sure there will be denials, but we have find out, which of them are important, because the rest will stay dontaudit-ed. Following command will add the dontaudit rules again: # semodule -B It appears that I had a broken install of selinux-policy-targeted as indicated by this response... [root@jsefler-7 ~]# semodule -DB libsemanage.semanage_exec_prog: Child process /sbin/load_policy did not exit cleanly. libsemanage.semanage_reload_policy: load_policy returned error code -1. libsemanage.semanage_exec_prog: Child process /sbin/load_policy did not exit cleanly. libsemanage.semanage_reload_policy: load_policy returned error code -1. semodule: Failed! To fix that, I re-installed selinux-policy-targeted as follows... mv /etc/selinux/targeted /etc/selinux/targeted.orig yum -y reinstall selinux-policy-targeted After re-installing selinux-policy-targeted, the policy appears fixed... [root@jsefler-7 ~]# rpm -q selinux-policy-targeted selinux-policy-targeted-3.13.1-53.el7.noarch [root@jsefler-7 ~]# semodule -DB [root@jsefler-7 ~]# semodule -B [root@jsefler-7 ~]# And most important... After re-testing the scenario in comment 12, the entitlement certs providing content of type "containerimage" now list in the hostname directories under /etc/docker/certs.d/ as expected by the rhsmcertd.service while getenforce is Enforcing. VERIFIED with packages... [root@jsefler-7 ~]# rpm -q docker docker-selinux selinux-policy selinux-policy-targeted subscription-manager-plugin-container docker-1.8.2-2.el7.x86_64 docker-selinux-1.8.2-2.el7.x86_64 selinux-policy-3.13.1-53.el7.noarch selinux-policy-targeted-3.13.1-53.el7.noarch subscription-manager-plugin-container-1.15.9-12.el7.x86_64 More Info... Once again I reproduced this bug on a second system. This time I did not see any evidence that the semodule was bad because when I ran semodule -DB and semodule -B, there was no failed response. Yet the test scenario in comment 12 continued to fail. Even with the instructions in comment 22, there were no AVC denials found. Repeating the fix in comment 23, I did this... mv /etc/selinux/targeted /etc/selinux/targeted.orig yum -y reinstall selinux-policy-targeted ...and then the scenario in comment 12 started working as expected. NEEDINFO to understand why re-installing selinux-policy-targeted solves this bug. |