1797719 – ipa trust-add fetch domain failure when selinux enforcing

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1797719 - ipa trust-add fetch domain failure when selinux enforcing

Summary: ipa trust-add fetch domain failure when selinux enforcing

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	freeipa
Sub Component:
Version:	32
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	---
Assignee:	IPA Maintainers
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-02-03 18:03 UTC by Scott Poore
Modified:	2021-02-24 20:44 UTC (History)
CC List:	21 users (show)
Fixed In Version:	freeipa-4.9.2-2.fc33 freeipa-4.9.2-2.fc32
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-02-24 20:40:39 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Scott Poore 2020-02-03 18:03:45 UTC

Description of problem:

when I run trust-add with SELinux in enforcing mode, I get an error:

[root@fedora-0 ~]# echo Secret123| ipa trust-add adcs.test --admin Administrator --password
ipa: ERROR: error on server 'fedora-0.example.test': Fetching domains from trusted forest failed. See details in the error_log

From error_log:

[Mon Feb 03 11:44:42.544594 2020] [wsgi:error] [pid 6831:tid 140492134995712] [remote 192.168.122.100:47994] ipalib.errors.RemoteRetrieveError: CIFS server communication error: code "3221225485", message "An invalid parameter was passed to a service or function." (both may be "None")

But, I don't see an AVC.

[root@fedora-0 httpd]# ausearch -m avc -ts recent


When I disable dontaudit, I see:

# ausearch -m avc -ts recent
----
time->Mon Feb  3 11:52:25 2020
type=AVC msg=audit(1580748745.535:428): avc:  denied  { noatsecure } for  pid=7604 comm="oddjobd" scontext=system_u:system_r:oddjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:ipa_helper_t:s0 tclass=process permissive=0
----
time->Mon Feb  3 11:52:25 2020
type=AVC msg=audit(1580748745.536:429): avc:  denied  { rlimitinh } for  pid=7604 comm="com.redhat.idm." scontext=system_u:system_r:oddjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:ipa_helper_t:s0 tclass=process permissive=0
----
time->Mon Feb  3 11:52:25 2020
type=AVC msg=audit(1580748745.536:430): avc:  denied  { siginh } for  pid=7604 comm="com.redhat.idm." scontext=system_u:system_r:oddjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:ipa_helper_t:s0 tclass=process permissive=0

Version-Release number of selected component (if applicable):

# rpm -q freeipa-server krb5-server samba selinux-policy
freeipa-server-4.8.4-6.fc32.x86_64
krb5-server-1.18-0.beta2.1.fc32.x86_64
samba-4.12.0-0.1.rc1.fc32.x86_64
selinux-policy-3.14.5-21.fc32.noarch


How reproducible:

Unknown but, I can consistently reproduce by deleting/adding trust when SELinux in enforcing mode.

Steps to Reproduce:
1. ipa-server-install
2. add dns forwarder and conditional forwarder on AD server if necessary
3. kinit admin
4. ipa trust-add ad.domain --admin Administrator --password  

Actual results:

Errors as above.  If I put SELinux in Permissive mode, it works.

Expected results:
Works in Enforcing mode

Additional info:

Comment 1 Zdenek Pytela 2020-02-04 07:31:02 UTC

Hi Scott,

The AVC denials are the common dontaudited ones so likely not related. Could you also check journal for USER_AVC entries?
Did this started to happen with some particular version of ipa, kerberos, samba, or other packages?

Comment 2 Scott Poore 2020-02-04 14:39:18 UTC

I see this:

[root@fedora-0 ~]# journalctl -xel --since 09:27:48 |grep AVC

Feb 04 09:27:49 fedora-0.example.test audit[1676]: AVC avc:  denied  { noatsecure } for  pid=1676 comm="oddjobd" scontext=system_u:system_r:oddjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:ipa_helper_t:s0 tclass=process permissive=0
Feb 04 09:27:49 fedora-0.example.test audit[1676]: AVC avc:  denied  { rlimitinh } for  pid=1676 comm="com.redhat.idm." scontext=system_u:system_r:oddjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:ipa_helper_t:s0 tclass=process permissive=0
Feb 04 09:27:49 fedora-0.example.test audit[1676]: AVC avc:  denied  { siginh } for  pid=1676 comm="com.redhat.idm." scontext=system_u:system_r:oddjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:ipa_helper_t:s0 tclass=process permissive=0
Feb 04 09:27:51 fedora-0.example.test audit[1679]: AVC avc:  denied  { read } for  pid=1679 comm="ipa-getkeytab" name="config" dev="dm-0" ino=397052 scontext=system_u:system_r:ipa_helper_t:s0 tcontext=system_u:object_r:selinux_config_t:s0 tclass=file permissive=0

The last one there being more interesting.

[root@fedora-0 ~]# find / -inum 397052
/etc/selinux/config

But, I'm not seeing that every time.  In fact, the first time I ran it today was the only time I saw it.  I just reran trust-del/trust-add about 10 times and didn't see it.

Is there some other way to check logs that would help?  or set higher log level?

Comment 3 Zdenek Pytela 2020-02-07 14:30:31 UTC

Scott,

The fourth AVC is also a common dontaudited one. I am afraid we cannot get more out of audit. Is it possible to make the ipa command more verbose instead?

In SELinux permissive, there are no AVC reported either?

Comment 4 Ben Cotton 2020-02-11 16:38:16 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 5 Scott Poore 2020-02-12 17:26:12 UTC

Zdenek,

I don't think I can see any USER_AVC entries in the journal.  All I see are the same AVCs I listed above.  Is there another way I could search for journal USER_AVCs or other selinux messages?   

When I run it in permissive mode, I see the same AVCs also and no more.  Anything else I can check?

As for more verbose/debug output from ipa command:

[root@fedora-0 ~]# SINCE=$(date +%H:%M:%S); ipa trust-del adcs.test; sleep 1; echo Secret123|ipa trust-add adcs.test --admin Administrator --password
ipa: ERROR: adcs.test: trust not found
ipa: ERROR: error on server 'fedora-0.example.test': Fetching domains from trusted forest failed. See details in the error_log
[root@fedora-0 ~]# SINCE=$(date +%H:%M:%S); ipa trust-del adcs.test; sleep 1; echo Secret123|ipa -vvv trust-add adcs.test --admin Administrator --password
-------------------------
Deleted trust "adcs.test"
-------------------------
ipa: INFO: Request: {
    "id": 0,
    "method": "trust_add/1",
    "params": [
        [
            "adcs.test"
        ],
        {
            "realm_admin": "Administrator",
            "realm_passwd": "Secret123",
            "version": "2.235"
        }
    ]
}
send: b'POST /ipa/session/json HTTP/1.1\r\nHost: fedora-0.example.test\r\nAccept-Encoding: gzip\r\nAccept-Language: en-us\r\nReferer: https://fedora-0.example.test/ipa/xml\r\nCookie: ipa_session=MagBearerToken=%2bJeNk3TLstzkWoJ4Y5aoeOVhe8KSGSZ%2fiI%2bYTkPhfo3jzsqByeHnCnpAmAPhcANpkwsKLf5gBAzmDvk5zsLfPSZ%2bHEZbK7z7bZ0itgFrr1qiYin1CmXCCDxnd0TuDosQFHYY%2bdctuspONFMmyAoOdXIZeyXHF09hrk07WvkLIwU1SOgSaGXJ51jTEjOFPvXZ;\r\nAccept-Encoding: gzip\r\nUser-Agent: Python-xmlrpc/3.8\r\nContent-Type: application/json\r\nContent-Length: 248\r\n\r\n'
send: b'{\n    "id": 0,\n    "method": "trust_add/1",\n    "params": [\n        [\n            "adcs.test"\n        ],\n        {\n            "realm_admin": "Administrator",\n            "realm_passwd": "Secret123",\n            "version": "2.235"\n        }\n    ]\n}'
reply: 'HTTP/1.1 200 Success\r\n'
header: Date: Wed, 12 Feb 2020 17:08:58 GMT
header: Server: Apache/2.4.41 (Fedora) OpenSSL/1.1.1d mod_wsgi/4.6.8 Python/3.8 mod_auth_gssapi/1.6.1
header: X-Frame-Options: DENY
header: Content-Security-Policy: frame-ancestors 'none'
header: Cache-Control: no-cache, private
header: Vary: Accept-Encoding
header: Content-Encoding: gzip
header: Transfer-Encoding: chunked
header: Content-Type: application/json; charset=utf-8
body: b'{"result": null, "error": {"code": 906, "message": "error on server \'fedora-0.example.test\': Fetching domains from trusted forest failed. See details in the error_log", "data": {"server": "fedora-0.example.test", "error": "Fetching domains from trusted forest failed. See details in the error_log"}, "name": "ServerCommandError"}, "id": 0, "principal": "admin", "version": "4.8.4"}'
ipa: INFO: Response: {
    "error": {
        "code": 906,
        "data": {
            "error": "Fetching domains from trusted forest failed. See details in the error_log",
            "server": "fedora-0.example.test"
        },
        "message": "error on server 'fedora-0.example.test': Fetching domains from trusted forest failed. See details in the error_log",
        "name": "ServerCommandError"
    },
    "id": 0,
    "principal": "admin",
    "result": null,
    "version": "4.8.4"
}
ipa: ERROR: error on server 'fedora-0.example.test': Fetching domains from trusted forest failed. See details in the error_log

And for info if needed later (httpd/error_log):



[Wed Feb 12 12:09:02.571411 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] ipa: ERROR: Helper fetch_domains was called for forest adcs.test, return code is 1
[Wed Feb 12 12:09:02.572235 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] ipa: ERROR: Standard output from the helper:
[Wed Feb 12 12:09:02.572257 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] ---
[Wed Feb 12 12:09:02.572268 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] 
[Wed Feb 12 12:09:02.572332 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] ipa: ERROR: Error output from the helper:
[Wed Feb 12 12:09:02.572345 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] Traceback (most recent call last):
[Wed Feb 12 12:09:02.572356 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]   File "/usr/lib/python3.8/site-packages/ipaserver/dcerpc.py", line 847, in __gen_lsa_connection
[Wed Feb 12 12:09:02.572367 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]     result = lsa.lsarpc(binding, self.parm, self.creds)
[Wed Feb 12 12:09:02.572377 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] samba.NTSTATUSError: (3221225485, 'An invalid parameter was passed to a service or function.')
[Wed Feb 12 12:09:02.572388 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] 
[Wed Feb 12 12:09:02.572398 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] During handling of the above exception, another exception occurred:
[Wed Feb 12 12:09:02.572409 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] 
[Wed Feb 12 12:09:02.572437 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] Traceback (most recent call last):
[Wed Feb 12 12:09:02.572448 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]   File "/usr/libexec/ipa/oddjob/com.redhat.idm.trust-fetch-domains", line 313, in <module>
[Wed Feb 12 12:09:02.572458 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]     domains = dcerpc.fetch_domains(
[Wed Feb 12 12:09:02.572468 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]   File "/usr/lib/python3.8/site-packages/ipaserver/dcerpc.py", line 1519, in fetch_domains
[Wed Feb 12 12:09:02.572478 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]     domains = communicate(td)
[Wed Feb 12 12:09:02.572489 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]   File "/usr/lib/python3.8/site-packages/ipaserver/dcerpc.py", line 1479, in communicate
[Wed Feb 12 12:09:02.572499 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]     td.init_lsa_pipe(td.info['dc'])
[Wed Feb 12 12:09:02.572509 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]   File "/usr/lib/python3.8/site-packages/ipaserver/dcerpc.py", line 871, in init_lsa_pipe
[Wed Feb 12 12:09:02.572520 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]     self._pipe = self.__gen_lsa_connection(binding)
[Wed Feb 12 12:09:02.572549 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]   File "/usr/lib/python3.8/site-packages/ipaserver/dcerpc.py", line 850, in __gen_lsa_connection
[Wed Feb 12 12:09:02.572575 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422]     raise assess_dcerpc_error(e)
[Wed Feb 12 12:09:02.572586 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] ipalib.errors.RemoteRetrieveError: CIFS server communication error: code "3221225485", message "An invalid parameter was passed to a service or function." (both may be "None")
[Wed Feb 12 12:09:02.572596 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] --
[Wed Feb 12 12:09:02.572607 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] 
[Wed Feb 12 12:09:02.572852 2020] [wsgi:error] [pid 937:tid 139775378171648] [remote 192.168.122.100:33422] ipa: INFO: [jsonserver_session] admin: trust_add/1('adcs.test', realm_admin='Administrator', realm_passwd='********', version='2.235'): ServerCommandError

Comment 6 Zdenek Pytela 2020-02-13 13:54:20 UTC

Scott,

This is a list of steps to do:

==================
# remove syscall auditing suppression
auditctl -d never,task
# enable full path auditing
auditctl -w /etc/shadow -p w
# rebuild the policy with temporarily removing dontaudit rules
semodule -DB
# possibly make the system permissive or not, you can get different results
setenforce 0
==================

and then after reproducing check both the audit log and journal.

# changing back to the previous state
setenforce 1
semodule -B
auditctl -W /etc/shadow -p w


Off the ipa logs, is it possible to isolate the library/system call which fails?

Comment 7 Tom Stultiens 2020-05-15 14:00:41 UTC

Please let me know if additional information is required. I have the same issue on a fresh install of freeipa on Fedora 32. ipa trust-fetch-domains fails after adding trust with ipa trust-add. After setting selinux to permissive, it works immediately without issues.  Please let me know where I can find the logs that are of interest to you. 

With SELinux set to permissive, no AVC's appear when running ipa trust-fetch-domains. Interestingly enough, I cannot get the problem to occur after enabling enforcing mode in SELinux again.

[root@nis01 toms]# setenforce 1

[root@nis01 toms]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      32
[root@nis01 toms]# ipa trust-fetch-domains
Realm name: sdktest.local
----------------------------------------------------------------------------------------
List of trust domains successfully refreshed. Use trustdomain-find command to list them.
----------------------------------------------------------------------------------------
----------------------------
Number of entries returned 0
----------------------------

[root@nis01 toms]# ipa trust-del sdktest.local
-----------------------------
Deleted trust "sdktest.local"
-----------------------------
[root@nis01 toms]# ipa trust
ipa: ERROR: unknown command 'trust'
[root@nis01 toms]# ipa trust-find
----------------
0 trusts matched
----------------
----------------------------
Number of entries returned 0
----------------------------

[root@nis01 toms]# ipa trust-add sdktest.local
Active Directory domain administrator: Administrator
Active Directory domain administrator's password:
------------------------------------------------------
Added Active Directory trust for realm "sdktest.local"
------------------------------------------------------
  Realm name: sdktest.local
  Domain NetBIOS name: SDKTEST
  Domain Security Identifier: S-1-5-21-1506927830-4214915980-971779718
  Trust direction: Trusting forest
  Trust type: Active Directory domain
  Trust status: Established and verified
[root@nis01 toms]# ipa trust-fetch-domains sdktest.local
----------------------------------------------------------------------------------------
List of trust domains successfully refreshed. Use trustdomain-find command to list them.
----------------------------------------------------------------------------------------
----------------------------
Number of entries returned 0
----------------------------

To double check, I ended up deleting the trust again, setting selinux to enforcing in the config, rebooting, and adding the trust again. I am unable to reproduce the issue after succesfully adding the trust once. I have not tried this with alternate domains.

Comment 9 Zdenek Pytela 2020-09-29 10:59:12 UTC

François,

Will this bz be resolved in freeipa-selinux? If yes, the component should change.

Comment 10 François Cami 2020-10-15 16:56:56 UTC

Zdenek, I've changed the component for now and will let you know soon.

Comment 12 Florence Blanc-Renaud 2021-01-10 10:06:08 UTC

Fixed upstream:
master:
    add58fb selinux: modify policy to allow one-way trust

ipa-4-8:
https://pagure.io/freeipa/c/40b69f2739590a3c954b76e88e557a277129819f

ipa-4-9:
https://pagure.io/freeipa/c/952b6bdcceda9f460e17075404084f1f3ddb5eaa

Comment 13 Fedora Update System 2021-02-15 21:01:52 UTC

FEDORA-2021-9ead482997 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-9ead482997

Comment 14 Fedora Update System 2021-02-15 21:01:53 UTC

FEDORA-2021-a53fdfcaeb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-a53fdfcaeb

Comment 15 Fedora Update System 2021-02-16 01:15:08 UTC

FEDORA-2021-a53fdfcaeb has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-a53fdfcaeb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-a53fdfcaeb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2021-02-16 01:18:45 UTC

FEDORA-2021-9ead482997 has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-9ead482997`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-9ead482997

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-02-24 20:40:39 UTC

FEDORA-2021-a53fdfcaeb has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 18 Fedora Update System 2021-02-24 20:44:59 UTC

FEDORA-2021-9ead482997 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.

abokovoy
dwalsh
fcami
fdc
frenaud
ftrivino
grepl.miroslav
ipa-maint
jcholast
jhrozek
lvrabec
mhjacks
plautrba
pvoborni
rcritten
rharwood
ssorce
toms
twoerner
vmojzis
zpytela