1165692 – After auto-heal, entitlement certs having "containerImage" content type is not updated in hostname directories

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1165692 - After auto-heal, entitlement certs having "containerImage" content type is not updated in hostname directories

Summary: After auto-heal, entitlement certs having "containerImage" content type is no...

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	subscription-manager
Sub Component:
Version:	7.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	candlepin-bugs
QA Contact:	Shwetha Kallesh
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	rhsm-rhel72
TreeView+	depends on / blocked

Reported:	2014-11-19 13:52 UTC by Rehana
Modified:	2015-10-02 22:26 UTC (History)
CC List:	10 users (show)
Fixed In Version:	subscription-manager-1.15.5-1
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-09-30 16:12:59 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
rhsm.log for updation of certs via auto-heal (113.71 KB, text/plain) 2015-08-11 09:12 UTC, Shwetha Kallesh	no flags	Details
rhsm.log for updation of certs by attaching subscription manually (72.05 KB, text/plain) 2015-08-11 09:14 UTC, Shwetha Kallesh	no flags	Details
View All

Description Rehana 2014-11-19 13:52:57 UTC

Description of problem:
Observed that after auto-heal, the attached subscriptions having  "containerImage" content type is not updated in the hostname directories 

Version-Release number of selected component (if applicable):
# subscription-manager  version 
server type: Red Hat Subscription Management
subscription management server: 0.9.26.4-1
subscription management rules: 5.11
subscription-manager: 1.13.7-1.el7
python-rhsm: 1.13.6-1.el7


How reproducible:
3/3

Steps to Reproduce:
1.Register client(used atomic machine) to production server
2.Auto-heal the system
3.
subscription-manager  register 
Username: stage_atomic1
Password: 
The system has been registered with ID: 255a82ba-d82c-40dd-a792-a2e9f5be1204 
-bash-4.2# subscription-manager attach --auto

All installed products are covered by valid entitlements. No need to update subscriptions at this time.

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

Product Name: Red Hat Enterprise Linux Atomic Host Beta
Status:       Subscribed

-bash-4.2# subscription-manager  list --consumed 
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name: Red Hat Enterprise Linux Standard Support Business Partner NFR
Provides:          Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update Support
                   Red Hat Enterprise Linux Workstation
                   Red Hat Certificate System
                   Red Hat Enterprise Linux for IBM POWER - Extended Update Support
                   Red Hat Software Collections (for RHEL Client)
                   Red Hat Beta
                   Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER) - Extended Update Support
                   Red Hat Enterprise Linux Atomic Host Beta
                   Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                   Red Hat Software Collections Beta (for RHEL Client)
                   Red Hat Developer Toolset (for RHEL for IBM POWER)
                   Red Hat S-JIS Support (for RHEL Server)
                   Red Hat Hardware Certification Test Suite
                   Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                   Red Hat Enterprise Linux Server - Extended Update Support
                   Red Hat Enterprise Linux Server
                   Red Hat Enterprise Linux for IBM POWER
                   Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER)
                   Red Hat Software Collections (for RHEL Workstation)
                   Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                   Red Hat Container Images
                   Red Hat Enterprise Linux Scalable File System (for RHEL Workstation)
                   Red Hat Software Collections Beta (for RHEL Workstation)
                   Red Hat Developer Toolset (for RHEL Server EUS)
                   Red Hat Enterprise Linux Desktop
                   Red Hat Enterprise Linux for IBM System z - Extended Update Support
                   Red Hat EUCJP Support (for RHEL Server) - Extended Update Support
                   Red Hat Directory Server
                   MRG Realtime
                   Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                   Red Hat Developer Toolset (for RHEL Server)
                   Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                   Red Hat Software Test Suite 5 (for RHEL Server)
                   Red Hat Container Images Beta
                   Red Hat Enterprise Linux High Availability (for RHEL Server)
                   Red Hat Enterprise Linux for SAP Hana
                   Red Hat Developer Toolset (for RHEL Workstation)
                   Red Hat Enterprise Linux for SAP
                   Red Hat Enterprise Linux for IBM System z
                   Red Hat EUCJP Support (for RHEL Server)
                   Red Hat Enterprise Linux Atomic Host
                   Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                   Red Hat Software Collections (for RHEL Server)
                   Red Hat Software Collections Beta (for RHEL Server)
                   Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                   Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                   Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node)
                   Red Hat Enterprise Linux for Scientific Computing
                   Red Hat S-JIS Support (for RHEL Server) - Extended Update Support
                   Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
SKU:               RH3310120
Contract:          10589427
Account:           5500204
Serial:            5249961184270933632
Pool ID:           8a99f98148eb2ff70148eed8742c0f6d
Active:            True
Quantity Used:     1
Service Level:     STANDARD
Service Type:      L1-L3
Status Details:    
Subscription Type: Standard
Starts:            12/31/2013
Ends:              12/30/2014
System Type:       Virtual

-bash-4.2# rct cc /etc/pki/entitlement/5249961184270933632.pem |  grep "container"
	Type: containerimage
	Label: rhel-6-server-beta-containers
	URL: /content/beta/rhel/server/6/x86_64/containers
	Type: containerimage
	Label: rhel-6-server-containers
	URL: /content/dist/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-beta-containers
	URL: /content/beta/rhel/server/7/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-containers
	URL: /content/dist/rhel/server/7/7Server/x86_64/containers

# cat  /etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf
[main]
enabled = 1
registry_hostnames = registry.access.redhat.com,cdn.redhat.com

check hostname directories

# ls /etc/docker/certs.d/registry.access.redhat.com/
# ls /etc/docker/certs.d/cdn.redhat.com/
redhat-uep.crt


Actual results:
Hostname directories are not updated with the entitlement certs

Expected results:
entitlement certs having "containerImage" should list in the hostname directories

Additional info:

Comment 2 Adrian Likins 2015-07-09 20:17:15 UTC

This seems to work with 1.15.5-1

Comment 4 Shwetha Kallesh 2015-07-27 10:35:36 UTC

Failed QA


[root@dhcp35-98 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.9.26.11-1
subscription management rules: 5.12
subscription-manager: 1.15.9-1.el7
python-rhsm: 1.15.3-1.el7


[root@dhcp35-98 ~]# service rhsmcertd restart
Redirecting to /bin/systemctl restart  rhsmcertd.service
[root@dhcp35-98 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Employee SKU
Provides:            JBoss Enterprise Web Platform
                     Red Hat Enterprise Virtualization for IBM Power
                     Red Hat Enterprise Linux for Power, big endian - Extended Update Support
                     Red Hat Certificate System
                     MRG Management
                     Oracle Java (for RHEL Compute Node) - Extended Update Support
                     Red Hat OpenShift Enterprise Infrastructure Beta
                     Red Hat Enterprise Linux for Real Time
                     Red Hat OpenStack
                     Red Hat Hardware Certification Test Suite
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     Red Hat Certificate System with Advanced Access
                     Red Hat JBoss A-MQ Clients
                     Red Hat Enterprise Linux 7 for HPC Compute Node High Touch Beta
                     JBoss Enterprise Application Platform
                     Red Hat OpenShift Enterprise JBoss FUSE add-on
                     MRG Grid Execute
                     Red Hat Enterprise Linux Server
                     Oracle Java (for RHEL Workstation)
                     Red Hat Enterprise Linux for Power, big endian
                     Red Hat Enterprise Linux EUS Compute Node
                     Red Hat Ceph Storage MON
                     Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER)
                     Red Hat Software Collections (for RHEL Workstation)
                     Red Hat OpenShift Enterprise Application Node Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Workstation)
                     JBoss Enterprise Application Platform - ELS
                     Red Hat Enterprise Linux 7 for IBM POWER High Touch Beta
                     Red Hat OpenShift Enterprise JBoss A-MQ add-on
                     Red Hat Enterprise Linux EUS Compute Node High Performance Networking
                     Red Hat Gluster Storage Server for On-premise
                     Red Hat Enterprise Linux Atomic Host HTB
                     Red Hat Gluster Storage Nagios Server
                     Red Hat Ceph Storage Calamari
                     Red Hat OpenStack Beta
                     Red Hat Enterprise Linux for IBM z Systems - Extended Update Support
                     Red Hat Directory Server
                     MRG Realtime
                     Red Hat Enterprise MRG Messaging 3 for RHEL 7
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat OpenShift Enterprise Infrastructure
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Enterprise Linux EUS Compute Node Scalable File System
                     Red Hat Enterprise Linux for Power, little endian
                     Red Hat Enterprise Linux 7 for IBM z Systems High Touch Beta
                     JBoss Enterprise Web Server
                     Red Hat Developer Toolset (for RHEL Workstation)
                     Red Hat Enterprise Linux Server - AUS
                     Red Hat Enterprise Linux for SAP
                     Red Hat Enterprise Linux for IBM z Systems
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat Enterprise Linux 7 Desktop High Touch Beta
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Storage
                     Red Hat Software Collections (for RHEL Server)
                     Red Hat OpenShift Enterprise Application Node
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat OpenShift Enterprise
                     Red Hat OpenStack Beta Certification Test Suite
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
                     Kernel Derivative Works for HPC for Power Systems
                     Red Hat Enterprise Linux Workstation
                     Red Hat Gluster Storage Management Console (for RHEL Server)
                     Oracle Java (for RHEL Client)
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - AUS
                     Red Hat Enterprise MRG Messaging
                     Red Hat Beta
                     Red Hat Enterprise Linux Atomic Host Beta
                     MRG Grid
                     Red Hat Enterprise Linux 7 Workstation High Touch Beta
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat Software Collections Beta (for RHEL Client)
                     Red Hat S-JIS Support (for RHEL Server)
                     Red Hat Enterprise Virtualization
                     Red Hat Container Development Kit
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Enterprise Linux 7 Resilient Storage High Touch Beta
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - AUS
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 7 Server High Touch Beta
                     Red Hat Enterprise Linux 6 Server HTB
                     Red Hat Container Images
                     Red Hat Cloud Infrastructure
                     Red Hat Software Collections Beta (for RHEL Workstation)
                     Oracle Java (for RHEL Compute Node)
                     Red Hat CloudForms
                     Red Hat Developer Toolset (for RHEL Server EUS)
                     Red Hat Enterprise Linux Desktop
                     Red Hat Enterprise Linux 7 Load Balancer High Touch Beta
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - AUS
                     Red Hat Developer Toolset (for RHEL Server)
                     Red Hat Enterprise Linux Server - Extended Life Cycle Support
                     Red Hat Ceph Storage
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux Server for ARM Beta
                     Red Hat OpenShift Enterprise Client Tools Beta
                     Red Hat OpenShift Enterprise Client Tools
                     Red Hat Enterprise Linux for SAP Hana
                     Red Hat EUCJP Support (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 7 High Availability High Touch Beta
                     Oracle Java (for RHEL Server) - AUS
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - AUS
                     Red Hat Container Images HTB
                     Red Hat Enterprise Linux 6 Workstation HTB
                     Red Hat Enterprise Linux for Scientific Computing
                     Red Hat Enterprise Linux Server for ARM Development Preview
                     Kernel Derivative Works for Bluegene/Q
                     Red Hat OpenShift Enterprise JBoss EAP add-on Beta
                     Red Hat Ceph Storage OSD
                     Red Hat OpenShift Enterprise JBoss EAP add-on
SKU:                 ES0113909
Contract:            10169793
Account:             477931
Serial:              6967812494487310609
Pool ID:             8a85f9823e3d5e43013e3ddd4e9509c4
Provides Management: Yes
Active:              True
Quantity Used:       1
Service Level:       Self-Support
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              04/24/2013
Ends:                01/01/2022
System Type:         Virtual

[root@dhcp35-98 ~]# rct cc /etc/pki/entitlement/6967812494487310609.pem | grep "container"
	Type: containerimage
	Label: rhel-6-server-beta-containers
	URL: /content/beta/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-6-server-containers
	URL: /content/dist/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-6-server-htb-containers
	URL: /content/htb/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-beta-containers
	URL: /content/beta/rhel/server/7/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-containers
	URL: /content/dist/rhel/server/7/7Server/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-htb-containers
	URL: /content/htb/rhel/server/7/x86_64/containers
[root@dhcp35-98 ~]# cat  /etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf
[main]
enabled = 1
registry_hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
[root@dhcp35-98 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
[root@dhcp35-98 ~]# ls /etc/docker/certs.d/cdn.redhat.com/
redhat-entitlement-authority.crt
[root@dhcp35-98 ~]# ls /etc/docker/certs.d/access.redhat.com/

^^ no entitlements certs are listed in the hostname directories


--------------------------------------------------------------------------------

Now I remove the subscription attached above and manually attach the same pool 

[root@dhcp35-98 ~]# subscription-manager remove --all
1 subscription removed at the server.
1 local certificate has been deleted.
[root@dhcp35-98 ~]# subscription-manager attach --pool 8a85f9823e3d5e43013e3ddd4e9509c4
Successfully attached a subscription for: Employee SKU
[root@dhcp35-98 ~]# ls /etc/docker/certs.d/access.redhat.com/
3606093036906710716.cert  3606093036906710716.key
[root@dhcp35-98 ~]# ls /etc/docker/certs.d/cdn.redhat.com/
3606093036906710716.cert  3606093036906710716.key  redhat-entitlement-authority.crt
[root@dhcp35-98 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
3606093036906710716.cert  3606093036906710716.key

^^^ hostname directories are updated with ent certs

Comment 5 Adrian Likins 2015-08-07 16:30:41 UTC

any rhsm.log's from when rhsmcertd auto-healed?

Comment 6 Adrian Likins 2015-08-10 13:53:49 UTC

Having trouble reproducing the failure, attaching an employee sku from access.stage, and then letting rhsmd pick it up creates the container certificate links for me.

Logs from rhsmd runs where that fails would get me closer.

Comment 7 Shwetha Kallesh 2015-08-11 09:12:12 UTC

Created attachment 1061368 [details]
rhsm.log for updation of certs via auto-heal

Auto-heal: 

[root@dhcp35-140 ~]# subscription-manager register
Registering to: subscription.rhn.stage.redhat.com/subscription:443
Username: qa
Password: 
The system has been registered with ID: 86e8c677-f6c9-4a49-b2ad-c73d93d4c85a 
[root@dhcp35-140 ~]# service rhsmcertd restart
Redirecting to /bin/systemctl restart  rhsmcertd.service

[root@dhcp35-140 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Employee SKU
Provides:            JBoss Enterprise Web Platform
                     Oracle Java (for Middleware)
                     Red Hat Enterprise Virtualization for IBM Power
                     Red Hat Enterprise Linux for Power, big endian - Extended Update Support
                     Red Hat Certificate System
                     MRG Management
                     Oracle Java (for RHEL Compute Node) - Extended Update Support
                     Red Hat OpenShift Enterprise Infrastructure Beta
                     Red Hat Enterprise Linux for Real Time
                     Red Hat OpenStack
                     Red Hat Hardware Certification Test Suite
                     Red Hat Certificate System with Advanced Access
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     Red Hat JBoss A-MQ Clients
                     Red Hat Enterprise Linux 7 for HPC Compute Node High Touch Beta
                     JBoss Enterprise Application Platform
                     Red Hat OpenShift Enterprise JBoss FUSE add-on
                     MRG Grid Execute
                     Oracle Java (for RHEL Workstation)
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux for Power, big endian
                     Red Hat Enterprise Linux EUS Compute Node
                     Red Hat Ceph Storage MON
                     Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER)
                     Red Hat Software Collections (for RHEL Workstation)
                     Red Hat OpenShift Enterprise Application Node Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Workstation)
                     JBoss Enterprise Application Platform - ELS
                     Red Hat Enterprise Linux 7 for IBM POWER High Touch Beta
                     Red Hat OpenShift Enterprise JBoss A-MQ add-on
                     Red Hat Enterprise Linux EUS Compute Node High Performance Networking
                     Red Hat Gluster Storage Server for On-premise
                     Atomic Enterprise Platform Early Access
                     Red Hat Gluster Storage Nagios Server
                     Red Hat Enterprise Linux Atomic Host HTB
                     Red Hat Ceph Storage Calamari
                     Red Hat Enterprise Linux for IBM z Systems - Extended Update Support
                     Red Hat OpenStack Beta
                     Red Hat Directory Server
                     MRG Realtime
                     Red Hat Enterprise MRG Messaging 3 for RHEL 7
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat OpenShift Enterprise Infrastructure
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Enterprise Linux EUS Compute Node Scalable File System
                     Red Hat Enterprise Linux for Power, little endian
                     Red Hat Enterprise Linux 7 for IBM z Systems High Touch Beta
                     Red Hat Developer Toolset (for RHEL Workstation)
                     JBoss Enterprise Web Server
                     Red Hat Enterprise Linux Server - AUS
                     Red Hat Enterprise Linux for SAP
                     Red Hat Enterprise Linux for IBM z Systems
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat Storage
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 7 Desktop High Touch Beta
                     Red Hat Software Collections (for RHEL Server)
                     Red Hat OpenShift Enterprise Application Node
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat OpenShift Enterprise
                     Red Hat OpenStack Beta Certification Test Suite
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
                     Kernel Derivative Works for HPC for Power Systems
                     Red Hat Enterprise Linux Workstation
                     Red Hat Gluster Storage Management Console (for RHEL Server)
                     Oracle Java (for RHEL Client)
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - AUS
                     Red Hat Enterprise MRG Messaging
                     Red Hat Beta
                     MRG Grid
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Enterprise Linux 7 Workstation High Touch Beta
                     Red Hat Software Collections Beta (for RHEL Client)
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat S-JIS Support (for RHEL Server)
                     Red Hat Enterprise Virtualization
                     Red Hat Container Development Kit
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Enterprise Linux 7 Resilient Storage High Touch Beta
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - AUS
                     Red Hat Enterprise Linux 7 Server High Touch Beta
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 6 Server HTB
                     Red Hat Container Images
                     Red Hat Cloud Infrastructure
                     Red Hat Software Collections Beta (for RHEL Workstation)
                     Oracle Java (for RHEL Compute Node)
                     Red Hat CloudForms
                     Red Hat Developer Toolset (for RHEL Server EUS)
                     Red Hat Enterprise Linux Desktop
                     Red Hat Enterprise Linux 7 Load Balancer High Touch Beta
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     Red Hat Developer Toolset (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - AUS
                     Red Hat Enterprise Linux Server - Extended Life Cycle Support
                     Red Hat Ceph Storage
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux Server for ARM Beta
                     Red Hat OpenShift Enterprise Beta
                     Red Hat OpenShift Enterprise Client Tools Beta
                     Red Hat OpenShift Enterprise Client Tools
                     Red Hat Enterprise Linux for SAP Hana
                     Red Hat EUCJP Support (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 7 High Availability High Touch Beta
                     Oracle Java (for RHEL Server) - AUS
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - AUS
                     Red Hat Enterprise Linux 6 Workstation HTB
                     Red Hat Container Images HTB
                     Red Hat Enterprise Linux for Scientific Computing
                     Red Hat Enterprise Linux Server for ARM Development Preview
                     Kernel Derivative Works for Bluegene/Q
                     Red Hat OpenShift Enterprise JBoss EAP add-on Beta
                     Red Hat OpenShift Enterprise JBoss EAP add-on
                     Red Hat Ceph Storage OSD
SKU:                 ES0113909
Contract:            10169793
Account:             477931
Serial:              9184007015206335191
Pool ID:             8a85f9823e3d5e43013e3ddd4e9509c4
Provides Management: Yes
Active:              True
Quantity Used:       1
Service Level:       Self-Support
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              04/24/2013
Ends:                01/01/2022
System Type:         Virtual


[root@dhcp35-140 ~]# rct cc /etc/pki/entitlement/9184007015206335191.pem | grep "container"
	Type: containerimage
	Label: rhel-6-server-beta-containers
	URL: /content/beta/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-6-server-containers
	URL: /content/dist/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-6-server-htb-containers
	URL: /content/htb/rhel/server/6/6Server/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-aep-beta-containers
	URL: /content/beta/rhel/server/7/$basearch/aep/containers
	Type: containerimage
	Label: rhel-7-server-beta-containers
	URL: /content/beta/rhel/server/7/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-containers
	URL: /content/dist/rhel/server/7/7Server/x86_64/containers
	Type: containerimage
	Label: rhel-7-server-htb-containers
	URL: /content/htb/rhel/server/7/x86_64/containers
	Type: containerimage
	Label: rhel-server-7-ose-beta-containers
	URL: /content/beta/rhel/server/7/$basearch/ose/3/containers
[root@dhcp35-140 ~]# cat  /etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf
[main]
enabled = 1
registry_hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/cdn.redhat.com/
redhat-entitlement-authority.crt
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/access.redhat.com/

[root@dhcp35-140 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.9.51.4-1
subscription management rules: 5.15
subscription-manager: 1.15.9-3.el7
python-rhsm: 1.15.4-2.el7

Comment 8 Shwetha Kallesh 2015-08-11 09:14:38 UTC

Created attachment 1061370 [details]
rhsm.log for updation of certs by attaching subscription manually

Now remove the subscription attached by rhsmcertd process and attach the same subscription manually , certs are updated 


[root@dhcp35-140 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Employee SKU
Provides:            JBoss Enterprise Web Platform
                     Oracle Java (for Middleware)
                     Red Hat Enterprise Virtualization for IBM Power
                     Red Hat Enterprise Linux for Power, big endian - Extended Update Support
                     Red Hat Certificate System
                     MRG Management
                     Oracle Java (for RHEL Compute Node) - Extended Update Support
                     Red Hat OpenShift Enterprise Infrastructure Beta
                     Red Hat Enterprise Linux for Real Time
                     Red Hat OpenStack
                     Red Hat Hardware Certification Test Suite
                     Red Hat Certificate System with Advanced Access
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     Red Hat JBoss A-MQ Clients
                     Red Hat Enterprise Linux 7 for HPC Compute Node High Touch Beta
                     JBoss Enterprise Application Platform
                     Red Hat OpenShift Enterprise JBoss FUSE add-on
                     MRG Grid Execute
                     Oracle Java (for RHEL Workstation)
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux for Power, big endian
                     Red Hat Enterprise Linux EUS Compute Node
                     Red Hat Ceph Storage MON
                     Red Hat Enterprise Linux High Performance Networking (for RHEL for IBM POWER)
                     Red Hat Software Collections (for RHEL Workstation)
                     Red Hat OpenShift Enterprise Application Node Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Workstation)
                     JBoss Enterprise Application Platform - ELS
                     Red Hat Enterprise Linux 7 for IBM POWER High Touch Beta
                     Red Hat OpenShift Enterprise JBoss A-MQ add-on
                     Red Hat Enterprise Linux EUS Compute Node High Performance Networking
                     Red Hat Gluster Storage Server for On-premise
                     Atomic Enterprise Platform Early Access
                     Red Hat Gluster Storage Nagios Server
                     Red Hat Enterprise Linux Atomic Host HTB
                     Red Hat Ceph Storage Calamari
                     Red Hat Enterprise Linux for IBM z Systems - Extended Update Support
                     Red Hat OpenStack Beta
                     Red Hat Directory Server
                     MRG Realtime
                     Red Hat Enterprise MRG Messaging 3 for RHEL 7
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat OpenShift Enterprise Infrastructure
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Enterprise Linux EUS Compute Node Scalable File System
                     Red Hat Enterprise Linux for Power, little endian
                     Red Hat Enterprise Linux 7 for IBM z Systems High Touch Beta
                     Red Hat Developer Toolset (for RHEL Workstation)
                     JBoss Enterprise Web Server
                     Red Hat Enterprise Linux Server - AUS
                     Red Hat Enterprise Linux for SAP
                     Red Hat Enterprise Linux for IBM z Systems
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat Storage
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 7 Desktop High Touch Beta
                     Red Hat Software Collections (for RHEL Server)
                     Red Hat OpenShift Enterprise Application Node
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat OpenShift Enterprise
                     Red Hat OpenStack Beta Certification Test Suite
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
                     Kernel Derivative Works for HPC for Power Systems
                     Red Hat Enterprise Linux Workstation
                     Red Hat Gluster Storage Management Console (for RHEL Server)
                     Oracle Java (for RHEL Client)
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - AUS
                     Red Hat Enterprise MRG Messaging
                     Red Hat Beta
                     MRG Grid
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Enterprise Linux 7 Workstation High Touch Beta
                     Red Hat Software Collections Beta (for RHEL Client)
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat S-JIS Support (for RHEL Server)
                     Red Hat Enterprise Virtualization
                     Red Hat Container Development Kit
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Enterprise Linux 7 Resilient Storage High Touch Beta
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - AUS
                     Red Hat Enterprise Linux 7 Server High Touch Beta
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 6 Server HTB
                     Red Hat Container Images
                     Red Hat Cloud Infrastructure
                     Red Hat Software Collections Beta (for RHEL Workstation)
                     Oracle Java (for RHEL Compute Node)
                     Red Hat CloudForms
                     Red Hat Developer Toolset (for RHEL Server EUS)
                     Red Hat Enterprise Linux Desktop
                     Red Hat Enterprise Linux 7 Load Balancer High Touch Beta
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     Red Hat Developer Toolset (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - AUS
                     Red Hat Enterprise Linux Server - Extended Life Cycle Support
                     Red Hat Ceph Storage
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux Server for ARM Beta
                     Red Hat OpenShift Enterprise Beta
                     Red Hat OpenShift Enterprise Client Tools Beta
                     Red Hat OpenShift Enterprise Client Tools
                     Red Hat Enterprise Linux for SAP Hana
                     Red Hat EUCJP Support (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux 7 High Availability High Touch Beta
                     Oracle Java (for RHEL Server) - AUS
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Compute Node)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - AUS
                     Red Hat Enterprise Linux 6 Workstation HTB
                     Red Hat Container Images HTB
                     Red Hat Enterprise Linux for Scientific Computing
                     Red Hat Enterprise Linux Server for ARM Development Preview
                     Kernel Derivative Works for Bluegene/Q
                     Red Hat OpenShift Enterprise JBoss EAP add-on Beta
                     Red Hat OpenShift Enterprise JBoss EAP add-on
                     Red Hat Ceph Storage OSD
SKU:                 ES0113909
Contract:            10169793
Account:             477931
Serial:              9184007015206335191
Pool ID:             8a85f9823e3d5e43013e3ddd4e9509c4
Provides Management: Yes
Active:              True
Quantity Used:       1
Service Level:       Self-Support
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              04/24/2013
Ends:                01/01/2022
System Type:         Virtual

[root@dhcp35-140 ~]# subscription-manager remove --serial 9184007015206335191
Serial numbers successfully removed at the server:
   9184007015206335191
1 local certificate has been deleted.
[root@dhcp35-140 ~]# subscription-manager attach --pool 8a85f9823e3d5e43013e3ddd4e9509c4
Successfully attached a subscription for: Employee SKU
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
4664871445946251217.cert  4664871445946251217.key
[root@dhcp35-140 ~]# ls /etc/docker/certs.d/cdn.redhat.com/
4664871445946251217.cert  4664871445946251217.key  redhat-entitlement-authority.crt
[root@dhcp35-140 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.9.51.4-1
subscription management rules: 5.15
subscription-manager: 1.15.9-3.el7
python-rhsm: 1.15.4-2.el7

Comment 10 Barnaby Court 2015-09-10 14:04:09 UTC

Please reproduce and report back if there is a difference in the end state between an auto-heal and manual attach.

Comment 11 Adrian Likins 2015-09-10 14:38:56 UTC

The log in comment 7 (where it fails) shows:

2015-08-11 14:22:45,543 [DEBUG] rhsmcertd-worker:13430 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x254fc10>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x3595310>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x359a4d0>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x359a550>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x359bd10>]
2015-08-11 14:22:45,543 [DEBUG] rhsmcertd-worker:13430 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com
2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:141 - Exiting plugin
2015-08-11 14:22:45,543 [DEBUG] rhsmcertd-worker:13430 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com
2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-08-11 14:22:45,543 [WARNING] rhsmcertd-worker:13430 @container.py:141 - Exiting plugin
2015-08-11 14:22:45,544 [DEBUG] rhsmcertd-worker:13430 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com
2015-08-11 14:22:45,544 [WARNING] rhsmcertd-worker:13430 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-08-11 14:22:45,544 [WARNING] rhsmcertd-worker:13430 @container.py:141 - Exiting plugin

The code in src/subscription_manager/plugin/container.py in sync() logs those messages.
That code is checking that /etc/docker/certs.d exists, and not populating the container certs.

Those paths should be created by the 'subscription-manager-plugin-container' package, the 'rpm -ql':
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/rhsm/ca/redhat-entitlement-authority.pem
/etc/rhsm/pluginconf.d/container_content.ContainerContentPlugin.conf
/usr/share/rhsm-plugins/container_content.py
/usr/share/rhsm-plugins/container_content.pyc
/usr/share/rhsm-plugins/container_content.pyo
/usr/share/rhsm/subscription_manager/plugin/container.py
/usr/share/rhsm/subscription_manager/plugin/container.pyc
/usr/share/rhsm/subscription_manager/plugin/container.pyo


Do those paths (/etc/docker/certs.d), exist when rhsmd runs?
Is /etc/docker/certs.d a directory? (Verify that it is _not_ a symlink or a broken symlink)
Are they being removed (or symlinked) as part of testing? 


Note: The subman version referenced in the original report (comment 0) was before the /etc/docker/certs.d/ paths
were adding to subscription-manager-plugin-container, but the versions referenced in later comments do have it.
(Any version later than 1.13.18)

Comment 12 John Sefler 2015-09-10 17:42:42 UTC

I agree with comment 11 that the reason auto-heal fails to copy the container entitlements to the registry_hostnames directory is because the /etc/docker/certs.d/ directory appeared to not exist.  The mystery is why did it appear not to exist?  If the latest docker or subscription-manager-plugin-container is installed then the existence of /etc/docker/certs.d/ is true.

Unfortunately....
I can reproduce this bug despite the preexistence of /etc/docker/certs.d/ ....
I suspect an selinux denial is blocking this test...

[root@jsefler-7 ~]# rpm -q docker docker-selinux selinux-policy subscription-manager-plugin-container
docker-1.7.1-115.el7.x86_64
docker-selinux-1.7.1-115.el7.x86_64
selinux-policy-3.13.1-47.el7.noarch
subscription-manager-plugin-container-1.15.9-7.el7.x86_64

[root@jsefler-7 ~]# getenforce
Enforcing

[root@jsefler-7 ~]# rpm -q --whatprovides /etc/docker/certs.d/
docker-1.7.1-115.el7.x86_64
[root@jsefler-7 ~]# ls -l /etc/docker/
total 4
drwxr-xr-x. 5 root root  60 Sep 10 12:59 certs.d
-rw-------. 1 root root 281 Aug 18 18:42 key.json
[root@jsefler-7 ~]# ls -l /etc/docker/certs.d/
total 0
drwxr-xr-x. 2 root root 45 Sep 10 12:19 cdn.redhat.com
drwxr-xr-x. 2 root root 26 Sep 10 12:42 redhat.com
drwxr-xr-x. 2 root root 26 Sep 10 12:42 redhat.io

[root@jsefler-7 ~]# subscription-manager register --serverurl=subscription.rhn.stage.redhat.com:443/subscription --username=stage_auto_testuser1
Registering to: subscription.rhn.stage.redhat.com:443/subscription
Password: 
The system has been registered with ID: 091c9bf9-8d9b-4a52-bb5a-880843d114a8 

[root@jsefler-7 ~]# subscription-manager list --installed

+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Atomic Host
Product ID:     271
Version:        7
Arch:           x86_64
Status:         Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:         
Ends:           

Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.2 Beta
Arch:           x86_64
Status:         Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:         
Ends:           

[root@jsefler-7 ~]# subscription-manager auto-attach --show
Auto-attach preference: enabled

[root@jsefler-7 ~]# systemctl restart rhsmcertd.service 
[root@jsefler-7 ~]# sleep 180


[root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log

2015-09-10 13:16:39,663 [DEBUG] rhsmcertd-worker:4151 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin
2015-09-10 13:16:39,664 [DEBUG] rhsmcertd-worker:4151 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x2b81f50>
2015-09-10 13:16:39,664 [INFO] rhsmcertd-worker:4151 @container_content.py:43 - Updating container content.
2015-09-10 13:16:39,665 [INFO] rhsmcertd-worker:4151 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
2015-09-10 13:16:39,665 [DEBUG] rhsmcertd-worker:4151 @__init__.py:85 - Searching for content of type: containerimage
2015-09-10 13:16:39,665 [DEBUG] rhsmcertd-worker:4151 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x28e1bd0>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x28e1c10>]
2015-09-10 13:16:39,666 [DEBUG] rhsmcertd-worker:4151 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com
2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:141 - Exiting plugin
2015-09-10 13:16:39,666 [DEBUG] rhsmcertd-worker:4151 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com
2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-09-10 13:16:39,666 [WARNING] rhsmcertd-worker:4151 @container.py:141 - Exiting plugin
2015-09-10 13:16:39,667 [DEBUG] rhsmcertd-worker:4151 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com
2015-09-10 13:16:39,667 [WARNING] rhsmcertd-worker:4151 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-09-10 13:16:39,667 [WARNING] rhsmcertd-worker:4151 @container.py:141 - Exiting plugin

BANG!  For some unknown reason the  existence of /etc/docker/certs.d/ is not accessible to rhsmcertd-worker running as a service.

HOWEVER if I run rhsmcertd-worker manually, it works (Syncing container certificates is successful)...

[root@jsefler-7 ~]# /usr/libexec/rhsmcertd-worker --autoheal
Updating entitlement certificates & repositories
Installed Products
        status: 0
        updates: []
        exceptions: 
        
Total updates: 1
Found (local) serial# []
Expected (UEP) serial# [4866972821415011331]
Added (new)
  [sn:4866972821415011331 (Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Oracle Java (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Server - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Server,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Atomic Host,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat EUCJP Support (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Oracle Java (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Software Collections (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Beta,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Developer Toolset (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Atomic Host Beta,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Software Collections Beta (for RHEL Server),) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Container Images,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat Container Images Beta,) @ /etc/pki/entitlement/4866972821415011331.pem]
  [sn:4866972821415011331 (Red Hat S-JIS Support (for RHEL Server) - Extended Update Support,) @ /etc/pki/entitlement/4866972821415011331.pem]
Deleted (rogue):
  <NONE>
Total updates: 0
Found (local) serial# [4866972821415011331L]
Expected (UEP) serial# [4866972821415011331]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>

[root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log

2015-09-10 13:24:49,725 [DEBUG] rhsmcertd-worker:4423 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin
2015-09-10 13:24:49,726 [DEBUG] rhsmcertd-worker:4423 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x164d850>
2015-09-10 13:24:49,726 [INFO] rhsmcertd-worker:4423 @container_content.py:43 - Updating container content.
2015-09-10 13:24:49,726 [INFO] rhsmcertd-worker:4423 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
2015-09-10 13:24:49,727 [DEBUG] rhsmcertd-worker:4423 @__init__.py:85 - Searching for content of type: containerimage
2015-09-10 13:24:49,727 [DEBUG] rhsmcertd-worker:4423 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x13acc50>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x13acc90>]
2015-09-10 13:24:49,728 [DEBUG] rhsmcertd-worker:4423 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com
2015-09-10 13:24:49,728 [INFO] rhsmcertd-worker:4423 @container.py:164 - Copying: /etc/pki/entitlement/4866972821415011331.pem -> /etc/docker/certs.d/registry.access.redhat.com/4866972821415011331.cert
2015-09-10 13:24:49,729 [INFO] rhsmcertd-worker:4423 @container.py:170 - Copying: /etc/pki/entitlement/4866972821415011331-key.pem -> /etc/docker/certs.d/registry.access.redhat.com/4866972821415011331.key
2015-09-10 13:24:49,730 [DEBUG] rhsmcertd-worker:4423 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com
2015-09-10 13:24:49,730 [INFO] rhsmcertd-worker:4423 @container.py:164 - Copying: /etc/pki/entitlement/4866972821415011331.pem -> /etc/docker/certs.d/cdn.redhat.com/4866972821415011331.cert
2015-09-10 13:24:49,730 [INFO] rhsmcertd-worker:4423 @container.py:170 - Copying: /etc/pki/entitlement/4866972821415011331-key.pem -> /etc/docker/certs.d/cdn.redhat.com/4866972821415011331.key
2015-09-10 13:24:49,731 [DEBUG] rhsmcertd-worker:4423 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com
2015-09-10 13:24:49,732 [INFO] rhsmcertd-worker:4423 @container.py:164 - Copying: /etc/pki/entitlement/4866972821415011331.pem -> /etc/docker/certs.d/access.redhat.com/4866972821415011331.cert
2015-09-10 13:24:49,732 [INFO] rhsmcertd-worker:4423 @container.py:170 - Copying: /etc/pki/entitlement/4866972821415011331-key.pem -> /etc/docker/certs.d/access.redhat.com/4866972821415011331.key
2015-09-10 13:24:49,733 [DEBUG] rhsmcertd-worker:4423 @plugins.py:769 - Running update_content_hook in ostree_content.OstreeContentPlugin
2015-09-10 13:24:49,733 [DEBUG] rhsmcertd-worker:4423 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x164df90>


NOTICE above that manually executing rhsmcertd-worker will successfully syncing container certificates to /etc/docker/certs.d/ but allowing the rhsmcertd service to do it fails.

UNFORTUNATELY I did not find an denial in /var/log/audit/audit.log during the test.

Last attempt with Permissive...

[root@jsefler-7 ~]# subscription-manager unsubscribe --all
1 subscription removed at the server.
1 local certificate has been deleted.
[root@jsefler-7 ~]# setenforce 0
[root@jsefler-7 ~]# getenforce 
Permissive
[root@jsefler-7 ~]# systemctl restart rhsmcertd.service 
[root@jsefler-7 ~]# sleep 180
[root@jsefler-7 ~]# ls /etc/docker/certs.d/registry.access.redhat.com/
254917606347016976.cert  254917606347016976.key
[root@jsefler-7 ~]# ls /etc/docker/certs.d/access.redhat.com/
254917606347016976.cert  254917606347016976.key
[root@jsefler-7 ~]# subscription-manager list --consumed | grep Serial
Serial:              254917606347016976


YUP, turning off selinux definitely helped avoid the failure.

Comment 13 Adrian Likins 2015-09-11 13:18:48 UTC

(In reply to John Sefler from comment #12)
> I agree with comment 11 that the reason auto-heal fails to copy the
> container entitlements to the registry_hostnames directory is because the
> /etc/docker/certs.d/ directory appeared to not exist.  The mystery is why
> did it appear not to exist?  If the latest docker or
> subscription-manager-plugin-container is installed then the existence of
> /etc/docker/certs.d/ is true.
> 
> Unfortunately....
> I can reproduce this bug despite the preexistence of /etc/docker/certs.d/
> ....
> I suspect an selinux denial is blocking this test...
> 

Ah. Interesting.

Got any logs with the AVC denials?

Comment 14 John Sefler 2015-09-11 13:42:20 UTC

UNFORTUNATELY I did not find any denial in /var/log/audit/audit.log during the test.

Comment 15 Adrian Likins 2015-09-23 13:47:03 UTC

(for reference)
/etc/docker/certs.d is system_u:object_r:cert_t:s0


/etc/pki/consumer unconfined_u:object_r:cert_t:s0
/etc/pki/entitlement unconfined_u:object_r:cert_t:s0

/usr/bin/rhsmcertd is system_u:object_r:rhsmcertd_exec_t:s0
/usr/libexec/rhsmcertd-worker system_u:object_r:bin_t:s0
(So I believe that means that running rhsmcertd-worker via rhsmcertd will have
different context that runing rhsmcertd-worker directly, so that doesn't contradict a selinux cause)

Comment 16 Adrian Likins 2015-09-24 20:00:14 UTC

could this be related to https://bugzilla.redhat.com/show_bug.cgi?id=1262812

Comment 17 John Sefler 2015-09-24 20:05:41 UTC

Please retest with selinux-policy-3.13.1-51.el7.noarch as verified in bug 1262812 to determine if this fixes our bug.

Comment 18 John Sefler 2015-09-25 21:20:24 UTC

Retesting with versions...

[root@jsefler-7 ~]# rpm -q docker docker-selinux selinux-policy subscription-manager-plugin-container
docker-1.8.2-2.el7.x86_64
docker-selinux-1.8.2-2.el7.x86_64
selinux-policy-3.13.1-52.el7.noarch
subscription-manager-plugin-container-1.15.9-11.el7.x86_64

Unfortunately, I continue to see the same behavior as outlined in comment 12.

The fix for Bug 1262812 did not fix this bug too.

Comment 19 John Sefler 2015-09-25 22:01:47 UTC

Seeking NEEDINFO from the selinux-policy experts for help resolving this bug that occurs when selinux is enforcing and does not occur when selinux is permissive.  Comment 15 is our current clue for why the rhsmcertd service fails to recognize the existence of directory /etc/docker/certs.d/ for writing files.

Comment 20 Milos Malik 2015-09-29 08:23:51 UTC

Do any SELinux denials appear in enforcing mode?

# ausearch -m avc -m user_avc -m selinux_err -i -ts today

Comment 21 John Sefler 2015-09-29 18:46:38 UTC

Unfortuinately, NO denials are logged when setenforce Enforcing...
[root@jsefler-7 ~]# ausearch -m avc -m user_avc -m selinux_err -i -ts today
<no matches>


Here is what is written to audit.log at the time of failure...
[root@jsefler-7 ~]# tail -f /var/log/audit/audit.log
type=USER_ACCT msg=audit(1443551281.197:462903): pid=32478 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(1443551281.197:462904): pid=32478 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(1443551281.200:462905): pid=32478 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=25045 res=1
type=USER_START msg=audit(1443551281.234:462906): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(1443551281.235:462907): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(1443551281.477:462908): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1443551281.485:462909): pid=32478 uid=0 auid=0 ses=25045 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grant
Here what is written to audit.log at the time of failure...ors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'


Here is what is written to rhsm.log at the time of failure (while Enforcing)...
[root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log
2015-09-29 14:28:32,195 [DEBUG] rhsmcertd-worker:32507 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin
2015-09-29 14:28:32,195 [DEBUG] rhsmcertd-worker:32507 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x2832f10>
2015-09-29 14:28:32,196 [INFO] rhsmcertd-worker:32507 @container_content.py:43 - Updating container content.
2015-09-29 14:28:32,196 [INFO] rhsmcertd-worker:32507 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
2015-09-29 14:28:32,196 [DEBUG] rhsmcertd-worker:32507 @__init__.py:85 - Searching for content of type: containerimage
2015-09-29 14:28:32,197 [DEBUG] rhsmcertd-worker:32507 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832090>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832d10>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832d90>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x2832dd0>]
2015-09-29 14:28:32,198 [DEBUG] rhsmcertd-worker:32507 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com
2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:141 - Exiting plugin
2015-09-29 14:28:32,198 [DEBUG] rhsmcertd-worker:32507 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com
2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:141 - Exiting plugin
2015-09-29 14:28:32,198 [DEBUG] rhsmcertd-worker:32507 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com
2015-09-29 14:28:32,198 [WARNING] rhsmcertd-worker:32507 @container.py:140 - Container cert directory does not exist: /etc/docker/certs.d/
2015-09-29 14:28:32,199 [WARNING] rhsmcertd-worker:32507 @container.py:141 - Exiting plugin



And when I setenforce Permissive, notice that there are no more warnings from rhsmcertd-worker...
(This is the behavior we want when selinux is Enforcing)
[root@jsefler-7 ~]# tail -f /var/log/rhsm/rhsm.log
2015-09-29 14:38:54,090 [DEBUG] rhsmcertd-worker:32748 @plugins.py:769 - Running update_content_hook in container_content.ContainerContentPlugin
2015-09-29 14:38:54,090 [DEBUG] rhsmcertd-worker:32748 @base_action_client.py:85 - running lib: <subscription_manager.content_action_client.ContentPluginActionInvoker object at 0x14a2290>
2015-09-29 14:38:54,091 [INFO] rhsmcertd-worker:32748 @container_content.py:43 - Updating container content.
2015-09-29 14:38:54,092 [INFO] rhsmcertd-worker:32748 @container_content.py:45 - registry hostnames = registry.access.redhat.com,cdn.redhat.com,access.redhat.com
2015-09-29 14:38:54,092 [DEBUG] rhsmcertd-worker:32748 @__init__.py:85 - Searching for content of type: containerimage
2015-09-29 14:38:54,093 [DEBUG] rhsmcertd-worker:32748 @container.py:53 - Got content_sets: [<subscription_manager.model.ent_cert.EntitlementCertContent object at 0x11f4a10>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x1219b50>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x1219bd0>, <subscription_manager.model.ent_cert.EntitlementCertContent object at 0x1219c10>]
2015-09-29 14:38:54,093 [DEBUG] rhsmcertd-worker:32748 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/registry.access.redhat.com
2015-09-29 14:38:54,094 [INFO] rhsmcertd-worker:32748 @container.py:164 - Copying: /etc/pki/entitlement/1439947531996000437.pem -> /etc/docker/certs.d/registry.access.redhat.com/1439947531996000437.cert
2015-09-29 14:38:54,094 [INFO] rhsmcertd-worker:32748 @container.py:170 - Copying: /etc/pki/entitlement/1439947531996000437-key.pem -> /etc/docker/certs.d/registry.access.redhat.com/1439947531996000437.key
2015-09-29 14:38:54,095 [DEBUG] rhsmcertd-worker:32748 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/cdn.redhat.com
2015-09-29 14:38:54,095 [INFO] rhsmcertd-worker:32748 @container.py:164 - Copying: /etc/pki/entitlement/1439947531996000437.pem -> /etc/docker/certs.d/cdn.redhat.com/1439947531996000437.cert
2015-09-29 14:38:54,096 [INFO] rhsmcertd-worker:32748 @container.py:170 - Copying: /etc/pki/entitlement/1439947531996000437-key.pem -> /etc/docker/certs.d/cdn.redhat.com/1439947531996000437.key
2015-09-29 14:38:54,096 [DEBUG] rhsmcertd-worker:32748 @container.py:138 - Syncing container certificates to /etc/docker/certs.d/access.redhat.com
2015-09-29 14:38:54,096 [INFO] rhsmcertd-worker:32748 @container.py:164 - Copying: /etc/pki/entitlement/1439947531996000437.pem -> /etc/docker/certs.d/access.redhat.com/1439947531996000437.cert
2015-09-29 14:38:54,097 [INFO] rhsmcertd-worker:32748 @container.py:170 - Copying: /etc/pki/entitlement/1439947531996000437-key.pem -> /etc/docker/certs.d/access.redhat.com/1439947531996000437.key

Comment 22 Milos Malik 2015-09-30 06:25:29 UTC

Could you temporarily remove dontaudit rules and collect SELinux denials?

# semodule -DB
your scenario
# ausearch -m avc -m user_avc -m selinux_err -i -ts today

I'm sure there will be denials, but we have find out, which of them are important, because the rest will stay dontaudit-ed. Following command will add the dontaudit rules again:

# semodule -B

Comment 23 John Sefler 2015-09-30 16:12:08 UTC

It appears that I had a broken install of selinux-policy-targeted as indicated by this response...

[root@jsefler-7 ~]# semodule -DB
libsemanage.semanage_exec_prog: Child process /sbin/load_policy did not exit cleanly.
libsemanage.semanage_reload_policy: load_policy returned error code -1.
libsemanage.semanage_exec_prog: Child process /sbin/load_policy did not exit cleanly.
libsemanage.semanage_reload_policy: load_policy returned error code -1.
semodule:  Failed!

To fix that, I re-installed selinux-policy-targeted as follows...
  mv /etc/selinux/targeted /etc/selinux/targeted.orig
  yum -y reinstall selinux-policy-targeted

After re-installing selinux-policy-targeted, the policy appears fixed...
[root@jsefler-7 ~]# rpm -q selinux-policy-targeted
selinux-policy-targeted-3.13.1-53.el7.noarch
[root@jsefler-7 ~]# semodule -DB
[root@jsefler-7 ~]# semodule -B
[root@jsefler-7 ~]# 


And most important...
After re-testing the scenario in comment 12, the entitlement certs providing content of type "containerimage" now list in the hostname directories under /etc/docker/certs.d/ as expected by the rhsmcertd.service while getenforce is Enforcing.


VERIFIED with packages...
[root@jsefler-7 ~]# rpm -q docker docker-selinux selinux-policy selinux-policy-targeted subscription-manager-plugin-container
docker-1.8.2-2.el7.x86_64
docker-selinux-1.8.2-2.el7.x86_64
selinux-policy-3.13.1-53.el7.noarch
selinux-policy-targeted-3.13.1-53.el7.noarch
subscription-manager-plugin-container-1.15.9-12.el7.x86_64

Comment 24 John Sefler 2015-10-02 22:26:26 UTC

More Info...
Once again I reproduced this bug on a second system.  This time I did not see any evidence that the semodule was bad because when I ran semodule -DB and semodule -B, there was no failed response.  Yet the test scenario in comment 12 continued to fail.  Even with the instructions in comment 22, there were no AVC denials found.

Repeating the fix in comment 23, I did this...
  mv /etc/selinux/targeted /etc/selinux/targeted.orig
  yum -y reinstall selinux-policy-targeted

...and then the scenario in comment 12 started working as expected.

NEEDINFO to understand why re-installing selinux-policy-targeted solves this bug.

Note You need to log in before you can comment on or make changes to this bug.