Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2170957 - parsec service fails to start on Fedora IoT 37 upgraded from F36 IoT
Summary: parsec service fails to start on Fedora IoT 37 upgraded from F36 IoT
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: parsec
Version: 40
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Peter Robinson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException AcceptedBlocker
Depends On:
Blocks: BetaFreezeException, F40BetaFreezeException F40FinalBlocker, FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2023-02-17 20:05 UTC by Micah Abbott
Modified: 2024-03-16 23:08 UTC (History)
10 users (show)

Fixed In Version: parsec-1.3.0-1.fc39 parsec-1.3.0-1.fc40
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-03-16 23:08:51 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Micah Abbott 2023-02-17 20:05:26 UTC 
The parsec service fails to start on Fedora IoT 37 (37.20230217.0):

```
# systemctl status parsec
× parsec.service - Parsec Service
     Loaded: loaded (/usr/lib/systemd/system/parsec.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-02-17 19:37:57 UTC; 3min 45s ago
       Docs: https://parallaxsecond.github.io/parsec-book/parsec_service/install_parsec_linux.html
    Process: 744 ExecStart=/usr/libexec/parsec -c /etc/parsec/config.toml (code=exited, status=1/FAILURE)
   Main PID: 744 (code=exited, status=1/FAILURE)
        CPU: 3ms

Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Scheduled restart job, restart counter is at 5.
Feb 17 19:37:57 localhost.localdomain systemd[1]: Stopped parsec.service - Parsec Service.
Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Start request repeated too quickly.
Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:57 localhost.localdomain systemd[1]: Failed to start parsec.service - Parsec Service.
```

Version-Release number of selected component (if applicable):

parsec-1.1.0-1.fc37.x86_64


Looking at the journal, there are a lot of `Error: Operation not permitted (os error 1)`

```
# journalctl -b -u parsec
Feb 17 19:37:56 localhost systemd[1]: Starting parsec.service - Parsec Service...
Feb 17 19:37:56 localhost parsec[706]: Error: Operation not permitted (os error 1)
Feb 17 19:37:56 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Feb 17 19:37:56 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:56 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Feb 17 19:37:56 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 1.
Feb 17 19:37:56 localhost systemd[1]: Stopped parsec.service - Parsec Service.
Feb 17 19:37:56 localhost systemd[1]: Starting parsec.service - Parsec Service...
Feb 17 19:37:56 localhost parsec[730]: Error: Operation not permitted (os error 1)
Feb 17 19:37:56 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Feb 17 19:37:56 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:56 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 2.
Feb 17 19:37:57 localhost systemd[1]: Stopped parsec.service - Parsec Service.
Feb 17 19:37:57 localhost systemd[1]: Starting parsec.service - Parsec Service...
Feb 17 19:37:57 localhost parsec[731]: Error: Operation not permitted (os error 1)
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:57 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 3.
Feb 17 19:37:57 localhost systemd[1]: Stopped parsec.service - Parsec Service.
Feb 17 19:37:57 localhost systemd[1]: Starting parsec.service - Parsec Service...
Feb 17 19:37:57 localhost parsec[732]: Error: Operation not permitted (os error 1)
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:57 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 4.
Feb 17 19:37:57 localhost systemd[1]: Stopped parsec.service - Parsec Service.
Feb 17 19:37:57 localhost systemd[1]: Starting parsec.service - Parsec Service...
Feb 17 19:37:57 localhost parsec[744]: Error: Operation not permitted (os error 1)
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Feb 17 19:37:57 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:57 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Scheduled restart job, restart counter is at 5.
Feb 17 19:37:57 localhost.localdomain systemd[1]: Stopped parsec.service - Parsec Service.
Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Start request repeated too quickly.
Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Failed with result 'exit-code'.
Feb 17 19:37:57 localhost.localdomain systemd[1]: Failed to start parsec.service - Parsec Service.
```

The config file

```
# cat /etc/parsec/config.toml 
# See a more complete example in /usr/shar/doc/parsec/config.toml
[core_settings]

[listener]
listener_type = "DomainSocket"

timeout = 200 # in milliseconds

[authenticator]
auth_type = "Direct"

[[key_manager]]
name = "on-disk-manager"
manager_type = "OnDisk"

[[provider]]
provider_type = "MbedCrypto"
key_info_manager = "on-disk-manager"

# [[provider]]
# provider_type = "Tpm"
# key_info_manager = "on-disk-manager"
# tcti = "device:/dev/tpmrm0"
# owner_hierarchy_auth = ""
```


I installed `strace` on my system and ran the `parsec` binary through that; I think the failure is happening when the binary tries to `chmod` /var/lib/parsec/mappings

```
# sudo -u parsec strace /usr/libexec/parsec -c /etc/parsec/config.toml
...
openat(AT_FDCWD, "/etc/parsec/config.toml", O_RDONLY|O_CLOEXEC) = 3
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address)
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=479, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "# See a more complete example in"..., 479) = 479
read(3, "", 32)                         = 0
close(3)                                = 0
getuid()                                = 983
ioctl(2, TCGETS, {c_iflag=ICRNL|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
mkdir("/var/lib/parsec/mappings", 0777) = -1 EEXIST (File exists)
statx(AT_FDCWD, "/var/lib/parsec/mappings", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/lib/parsec/mappings", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(3, 0x563f196fcf60 /* 2 entries */, 32768) = 48
getdents64(3, 0x563f196fcf60 /* 0 entries */, 32768) = 0
close(3)                                = 0
chmod("/var/lib/parsec/mappings", 0700) = -1 EPERM (Operation not permitted)
write(2, "Error: ", 7Error: )                  = 7
write(2, "Operation not permitted", 23Operation not permitted) = 23
write(2, " (os error ", 11 (os error )             = 11
write(2, "1", 11)                        = 1
write(2, ")", 1))                        = 1
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7f0d27cd7000, 12288)           = 0
exit_group(1)                           = ?
+++ exited with 1 +++
```

Looking at that directory, it looks like it is owned by the `dnsmasq` user and write access is only for that user:

```
# ls -laRZ /var/lib/parsec/
/var/lib/parsec/:
total 12
drwxr-x---.  3 parsec  parsec         system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 .
drwxr-xr-x. 27 root    root           system_u:object_r:var_lib_t:s0 4096 Feb 15 23:26 ..
drwxr-xr-x.  2 dnsmasq parsec-clients system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 mappings

/var/lib/parsec/mappings:
total 8
drwxr-xr-x. 2 dnsmasq parsec-clients system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 .
drwxr-x---. 3 parsec  parsec         system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 ..
```

Since the directory is being created when the service/binary runs, it looks like in my case the directory was created as part of a previous deployment.  This particular system started as an F36 IoT install, so perhaps the owner on that directory got mangled somehow?  I'd need to test a fresh F36 IoT install.
Comment 1 Simon de Vlieger 2023-02-17 20:27:34 UTC 
I can reproduce this with a fresh Fedora IoT 37 install using the installation ISO on x86_64. More interestingly, the initial deployment from the ISO (...) does start parsec correctly. After doing `rpm-ostree upgrade` and a reboot parsec fails to start with the same error.

It works in commit 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0 and doesn't work in commit 8dde1800d7d3aeaf71e0737c9551e550114a20dd73046ba2e3b4d8aec3873f2e.

In commit 8dde I have the same permissions on /var/lib/parsec/mappings as mentioned by Micah, in commit 2334 the permissions are as follows:

```
[root@localhost ~]# ls -laRZ /var/lib/parsec/
/var/lib/parsec/:
total 12
drwxr-x---.  3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 .
drwxr-xr-x. 26 root   root   system_u:object_r:var_lib_t:s0 4096 Feb 17 21:18 ..
drwx------.  2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 mappings

/var/lib/parsec/mappings:
total 8
drwx------. 2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 .
drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 ..
```


It seems that betweens these two commits a relevant change might be that dnsmasq changed version from 2.87-1.fc37 to 2.88-1.fc37?
Comment 2 Micah Abbott 2023-02-17 20:28:18 UTC 
I did a fresh install of F36 IoT and observed the following:

```
[root@fedora ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
                   Version: 36.20220618.0 (2022-06-18T10:37:36Z)
                    Commit: 329c785ac61ca4d6ccad4c7d65049a49fdbe682242dc3e49b0a764c10b89fed2
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
[root@fedora ~]# rpm -q parsec
parsec-1.0.0-1.fc36.x86_64
[root@fedora ~]# ls -laRZ /var/lib/parsec/
/var/lib/parsec/:
total 12
drwxr-x---.  3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 .
drwxr-xr-x. 25 root   root   system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 ..
drwxr-xr-x.  2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 mappings

/var/lib/parsec/mappings:
total 8
drwxr-xr-x. 2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 .
drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 ..
[root@fedora ~]# id parsec
uid=982(parsec) gid=982(parsec) groups=982(parsec),981(parsec-clients)
```

And looking at the UIDs on the F37 IoT system:

```
[root@localhost ~]# id parsec
uid=983(parsec) gid=983(parsec) groups=983(parsec),982(parsec-clients)
[root@localhost ~]# id dnsmasq
uid=982(dnsmasq) gid=981(dnsmasq) groups=981(dnsmasq)
```


So it looks like the UID mapping changed between Fedora releases?
Comment 3 Simon de Vlieger 2023-02-17 20:33:55 UTC 
And here's how the uids/gids changed:

```
# working (commit 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0)
[root@localhost ~]# id dnsmasq
uid=984(dnsmasq) gid=984(dnsmasq) groups=984(dnsmasq)
[root@localhost ~]# id parsec
uid=982(parsec) gid=982(parsec) groups=982(parsec),981(parsec-clients)

# non-working (commit 8dde1800d7d3aeaf71e0737c9551e550114a20dd73046ba2e3b4d8aec3873f2e)
[root@localhost ~]# id dnsmasq
uid=982(dnsmasq) gid=981(dnsmasq) groups=981(dnsmasq)
[root@localhost ~]# id parsec
uid=983(parsec) gid=983(parsec) groups=983(parsec),982(parsec-clients)
```
Comment 4 Micah Abbott 2023-02-17 21:08:27 UTC 
I suspect this may be due to how both packages handle user creation.

The parsec package uses `useradd` to create users

https://src.fedoraproject.org/rpms/parsec/blob/rawhide/f/parsec.spec#_75-81

The dnsmasq package uses systemd sysusers

https://src.fedoraproject.org/rpms/dnsmasq/blob/rawhide/f/dnsmasq.spec#_177-179


I'm out of my depth here, but I wonder if `parsec` should adopt the sysusers approach.

Regardless, I think it might benefit from some changes that can detect if the /var/lib/parsec/mappings folder is owned by a different UID and chown it to the current UID for the `parsec` user
Comment 5 Ricardo Gerardi 2023-03-11 15:47:08 UTC 
I had the same issue on a fresh install of Fedora IoT 37 aarch64 on Raspberry Pi 4.

Issue persisted after updating to the latest version.
Comment 6 Arne 2023-09-25 15:59:24 UTC 
Still present on latest Fedora 38 aarch64 on Pi 4 (fresh install with all updates)
Comment 7 Aoife Moloney 2023-11-23 01:16:48 UTC 
This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.
Comment 8 Micah Abbott 2023-11-27 21:23:33 UTC 
This is still an issue in Fedora 39
Comment 9 Jan Pazdziora 2024-01-04 21:25:43 UTC 
I hit the same problem on fresh Fedora 39 IoT installation, rpm-ostree upgrade'd to

# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
                  Version: 39.20231231.0 (2023-12-31T16:28:21Z)
                   Commit: 9ccb803a38143ddb2803255c205d9cc73367f852d383dc517a6e6d9f8ffb70ea
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C

  fedora-iot:fedora/stable/aarch64/iot
                  Version: 39.20231103.1 (2023-11-03T18:17:43Z)
                   Commit: cc8d419f72d84ac24d0a95e235c0cdf72844f73d9d6f42a41fcddf23dfb34f7d
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

The ownership gets messed up to

# ls -laRZ /var/lib/parsec/
/var/lib/parsec/:
total 12
drwxr-x---.  3 parsec parsec         system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 .
drwxr-xr-x. 25 root   root           system_u:object_r:var_lib_t:s0 4096 Jan  4 20:46 ..
drwx------.  2 clevis parsec-clients system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 mappings

/var/lib/parsec/mappings:
total 8
drwx------. 2 clevis parsec-clients system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 .
drwxr-x---. 3 parsec parsec         system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 ..
Comment 10 Sebastian 2024-01-06 07:47:08 UTC 
I've been digging into this a little. My first attempt was to add a

%attr(-,parsec,parsec) %ghost %config %{_localstatedir}/lib/parsec/mappings/

to the package spec. Unsure if that's the correct approach, but I was just going to test that out.

However, rebuilding parsec is blocked by a couple dependency updates that are breaking cargo/mockbuild. Some of those are fixable with more Cargofile patches, the remaining one that's still blocking me is https://bugzilla.redhat.com/show_bug.cgi?id=2229923 which I think has to be patched first to get the actual parsec build running again.
Comment 11 Peter Robinson 2024-01-24 13:06:48 UTC 
> %attr(-,parsec,parsec) %ghost %config %{_localstatedir}/lib/parsec/mappings/
> 
> to the package spec. Unsure if that's the correct approach, but I was just
> going to test that out.

I think we need the following:

%attr(0750,parsec,parsec) %dir %{_localstatedir}/lib/parsec/mappings/

Will have an update to test shortly.
Comment 12 Fedora Update System 2024-03-10 21:41:19 UTC 
FEDORA-2024-a223050257 (parsec-1.3.0-1.fc40 and rust-cryptoki-0.6.2-1.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-a223050257
Comment 13 Fedora Update System 2024-03-10 22:05:27 UTC 
FEDORA-2024-6cf8aec019 (parsec-1.3.0-1.fc39, rust-cryptoki-0.6.2-1.fc39, and 6 more) has been submitted as an update to Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-6cf8aec019
Comment 14 Fedora Blocker Bugs Application 2024-03-11 00:13:47 UTC 
Proposed as a Blocker for 40-beta by Fedora user pbrobinson using the blocker tracking app because:

 Fedora IoT default service
Comment 15 Fedora Update System 2024-03-11 01:40:01 UTC 
FEDORA-2024-6cf8aec019 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-6cf8aec019`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-6cf8aec019

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 16 Adam Williamson 2024-03-12 01:31:18 UTC 
Discussed at today's blocker review meeting: https://meetbot-raw.fedoraproject.org/blocker-review_matrix_fedoraproject-org/2024-03-11/f40-blocker-review.2024-03-11-16.00.html . This was rejected as a Beta blocker but accepted as a Final blocker, because it violates the "all system services must start" criterion after upgrade, which is a Final criterion. Also accepted as a Beta FE so we can fix upgrades ASAP.
Comment 17 Fedora Update System 2024-03-12 02:18:13 UTC 
FEDORA-2024-a223050257 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-a223050257`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-a223050257

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 18 Fedora Update System 2024-03-12 03:01:28 UTC 
FEDORA-2024-6cf8aec019 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-6cf8aec019`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-6cf8aec019

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 19 František Zatloukal 2024-03-12 13:20:22 UTC 
https://bodhi.fedoraproject.org/updates/FEDORA-2024-a223050257 fixes this issue.
Comment 20 Fedora Update System 2024-03-13 01:24:48 UTC 
FEDORA-2024-6cf8aec019 (parsec-1.3.0-1.fc39, parsec-tool-0.4.0-6.fc39, and 8 more) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 21 Peter Robinson 2024-03-16 19:13:05 UTC 
Reopening as we need it for F-40 freeze exception and it was closed for F39
Comment 22 Fedora Update System 2024-03-16 23:08:51 UTC 
FEDORA-2024-a223050257 (parsec-1.3.0-1.fc40, parsec-tool-0.4.0-6.fc40, and 2 more) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.