Created attachment 1718948 [details]
retrace server failure screenshot

Created attachment 1718949 [details]
retrace server details screenshot

Per bug 1882319 comment 11 I'm suggesting this as a Final blocker, it might violate:
"All applications that can be launched using the standard graphical mechanism after a default installation of Fedora Workstation on the x86_64 architecture must start successfully and withstand a basic functionality test. "
https://fedoraproject.org/wiki/Fedora_33_Final_Release_Criteria#Default_application_functionality

(In reply to Kamil Páral from comment #0)
> Retrace job failed
> Retrace failed. Try again later and if the problem persists report this
> issue please.

I see exactly the same error when I try to report a crash from F32. So this seems to be a server-related problem.

Discussed during the 2020-10-05 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as the app is likely working (the bug is likely in the retrace server itself), and local traceback generation does work so we aren't stuck without being able to report crashes. Accepted as a freeze exception issue just in case a fix is needed to the package, obviously we would want that in for release.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-10-05/f33-blocker-review.2020-10-05-16.00.txt

The underlying problem should be fixed now and I have successfully verified the solution myself, via retracing a few synthetic crashes.

If the same problem persists, please feel free to reopen the issue with more details.

Problem persists: I tried to report a gnome-shell crash today, and the retrace step failed, so I had to retrace locally. What more details do you want?

(Reporting to Bugzilla failed too, because "the generated backtrace has low informational value," which is an ABRT bug because I can get a good backtrace in gdb. But that is a different bug.)

I tried sending SIGABRT to gnome-calculator and htop, and both generated the backtrace remotely just fine, but both were marked as "low information value" and none of them could be reported.

Been trying to do remote-retracing on fc32 for a while without success, and saw this item on fc33's common bugs page. Btw, there is a typo on the fc33 common bugs page: it says 1895154 instead of 1885154

(In reply to Hin-Tak Leung from comment #9)
> Btw, there is a typo on the fc33 common bugs page: it says 1895154 instead of 1885154

Thanks, fixed

After bug 1893549 has been fixed, the retrace now always fails here too. Hitting "Repeat" does not help so far:


--- Running report_uReport ---
('report_uReport' completed successfully)

--- Skipping collect_GConf ---
No matching actions found for this event.

--- Skipping collect_vimrc_system ---
No matching actions found for this event.

--- Skipping collect_vimrc_user ---
No matching actions found for this event.

--- Skipping collect_xsession_errors ---
No matching actions found for this event.

--- Running analyze_CCpp ---
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES'
Querying server settings
Preparing an archive to upload
Uploading 5.0 MiB
Upload successful
Retrace job started
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.

[2020-11-03 17:39:16] [I] Analyzing crash data

Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO'
('analyze_CCpp' exited with 1)

Could you please provide a bit more detail on your crash, Christian?
Which packages did the crash occur in?
Could you upload the coredump (provided it's not sensitive).

Just today I saw a crash of xdg-desktop-portal. When I used retrace server, I was told it has a low information value. When I reopened the report and instead performed a local tracing, I could report a new bug just fine - bug 1894965. There is something broken on the retrace server, it doesn't seem to generate the backtrace correctly.

This time the retrace job finished successfully, so I guess it's an intermittent problem then, going on for quite(!) a while though:


$ sleep 1234 &
$ pkill -SEGV sleep
$ fg
bash: fg: job has terminated
[1]+  Segmentation fault      (core dumped) sleep 1234

$ abrt list
Id           8347495  
Component    coreutils  
Count        1  
Time         2020-11-05 18:35:14  
User id      1000 (christian)  
Reported to    
ABRT Server  https://retrace.fedoraproject.org/faf/reports/bthash/0fa8854a6e813ce5de3ac04e7c7db9318dfcb787


$ abrt report 8347495
Reporting problem 8347495

no actions matching this problem found for event 'collect_GConf'
no actions matching this problem found for event 'collect_vimrc_system'
no actions matching this problem found for event 'collect_vimrc_user'
no actions matching this problem found for event 'collect_xsession_errors'
('report_uReport' completed successfully)
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y
Querying server settings
Preparing an archive to upload
Uploading 16.3 KiB
Upload successful
Retrace job started
Analyzing crash data
Generating backtrace
......
Retrace job finished successfully
Looking for similar problems in bugzilla
Bugzilla User name:
[....]



===== And, sure enough, the 2nd test shortly after didn't even get to the retrace part:



$ ksh
$ kill -SEGV $$                                          
### 688908 Function backtrace:
1   (null) + 94719070134799
2   (null) + 140434281270352
3   kill + 11
4   (null) + 94719070216393
5   (null) + 94719070215054
6   (null) + 94719070314513
7   (null) + 94719070031220
8   (null) + 94719069877289
9   (null) + 94719070660120
10  __libc_start_main + 242
11  (null) + 94719069859246
Aborted (core dumped)



$ abrt list
Id           0b84ec4  
Component    ksh  
Count        1  
Time         2020-11-05 18:48:14  
User id      1000 (christian)  
Reported to    
ABRT Server  https://retrace.fedoraproject.org/faf/reports/bthash/a509aef686e612d2166f70fa256d1eab0b5440a9


$ abrt report 0b84ec4
Reporting problem 0b84ec4

no actions matching this problem found for event 'collect_GConf'
no actions matching this problem found for event 'collect_vimrc_system'
no actions matching this problem found for event 'collect_vimrc_user'
no actions matching this problem found for event 'collect_xsession_errors'
('report_uReport' completed successfully)
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y
Querying server settings
A server-side error occurred on 'https://retrace.fedoraproject.org'
Unexpected HTTP response from server: 403
Package NVR contains illegal characters
Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). [y/N/f] n
('analyze_CCpp' exited with 1)



Is this maybe a variation of bug 1893549, still?

It seems we had some trouble pulling the right packages for Fedora 33 crashes
in the background. The problem should be fixed now.

Feel free reopen again if you stumble on the same problem again.

This just happened again:


$ abrt list
Id           d36ed72  
Component    libreoffice  
Count        1  
Time         2020-11-21 02:48:40  
User id      1000 (christian)  
Reported to    
ABRT Server  https://retrace.fedoraproject.org/faf/reports/bthash/5b5dd0a9e3f7209ca9c835194768a9d1af029c7e



$ abrt report d36ed72
Reporting problem d36ed72

no actions matching this problem found for event 'collect_GConf'
no actions matching this problem found for event 'collect_vimrc_system'
no actions matching this problem found for event 'collect_vimrc_user'
no actions matching this problem found for event 'collect_xsession_errors'
('report_uReport' completed successfully)
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y
Querying server settings
A server-side error occurred on 'https://retrace.fedoraproject.org'
Unexpected HTTP response from server: 403
Package NVR contains illegal characters
Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). [y/N/f] 


Can anybody re-open this? Or is this being tracked upstream?

  > Handle crashes in modules
  > https://github.com/abrt/retrace-server/issues/400

Yours seems to be a different bug, Christian. It might be related to the upstream one,
though I can't say that with certainty right now. This issue is tracking the bug which
results in the "Retrace job failed" message.

Could you please specify what version of the libreoffice package you have installed
and which version of Fedora you're on?

This was a crash in libreoffice-core-7.0.3.1-1.fc33.x86_64 running on Fedora 33. But year, I'll continue the "Package NVR contains illegal characters" thing over in bug 1900982. Thanks.

Happened again today on a freshly installed Fedora 33 installation. gnome-shell crashed (yikes!) and the "Generating backtrace" took ~4 hours to complete, and then finally failed:




--- Running report_uReport ---
('report_uReport' completed successfully)

--- Skipping collect_GConf ---
No matching actions found for this event.

--- Skipping collect_vimrc_system ---
No matching actions found for this event.

--- Skipping collect_vimrc_user ---
No matching actions found for this event.

--- Skipping collect_xsession_errors ---
No matching actions found for this event.

--- Running analyze_CCpp ---
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES'
Querying server settings
Preparing an archive to upload
You are going to upload 19.1 MiB. Continue? 'YES'
Uploading 19.1 MiB
Upload successful
Retrace job started
Analyzing crash data
................................................................................
......................................
Generating backtrace
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
.......An error occurred while connecting to 'https://retrace.fedoraproject.org'
Could not connect: Connection refused
Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'YES'
Analyzing coredump 'coredump'
Cleaning cache...
Cache cleaning has finished
Coredump references 200 debuginfo files, 400 of them are not installed
Initializing package manager
Setting up repositories
Looking for needed packages in repositories

Argh, and while the local processing appeared to be successful, it's the retrace server again:

=============================================================================
[....]
.......An error occurred while connecting to 'https://retrace.fedoraproject.org'
Could not connect: Connection refused
Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'YES'
Analyzing coredump 'coredump'
Cleaning cache...
Cache cleaning has finished
Coredump references 200 debuginfo files, 400 of them are not installed
Initializing package manager
Setting up repositories
Looking for needed packages in repositories
Can't find packages for 200 debuginfo files
Packages to download: 179
Downloading 1234.30Mb, installed size: 5281.56Mb. Continue? 'YES'
Downloading (1 of 179) gnome-shell-debuginfo-3.38.2-1.fc33.x86_64.rpm: 100%
[....]
Generating backtrace
Backtrace is generated and saved, 225233 bytes

--- Running analyze_BodhiUpdates ---
Looking for similar problems in bugzilla
Duplicate bugzilla bug '#1842589' was found
Searching for updates
No updates for this package found

--- Running report_Bugzilla ---
Logging into Bugzilla at https://bugzilla.redhat.com
Checking for duplicates
Creating a new bug
New bug id: 1905110
Adding External URL to bug 1905110
Adding attachments to bug 1905110
Logging out
Status: NEW https://bugzilla.redhat.com/show_bug.cgi?id=1905110

--- Running post_report ---
Failed to upload uReport to the server 'https://retrace.fedoraproject.org/faf' with curl: Failed to connect to retrace.fedoraproject.org port 443: Connection refused
Error: curl_easy_perform: Couldn't connect to server
('post_report' exited with 1)
=============================================================================


So, while I'm surely repeating myself, but: isn't this a real problem? People not being able to report bugs properly, for such a long time?

Thanks for the detailed report, Christian.

We're now tracking the first issue in bug 1902312.
The latter issue was caused by a brief maintenance reboot. The server should be up and running now.

Unfortunately this just happened again. It's not too often that I have to use abrt to report bugs, but when I do they always seem to fail. That wasn't the case a few years back, when abrt was working pretty reliably and bug reporting was easy (from a user's perspective). So, this happened just now:


===================================================================
--- Running report_uReport ---
('report_uReport' completed successfully)

--- Skipping collect_GConf ---
No matching actions found for this event.

--- Skipping collect_vimrc_system ---
No matching actions found for this event.

--- Skipping collect_vimrc_user ---
No matching actions found for this event.

--- Skipping collect_xsession_errors ---
No matching actions found for this event.

--- Running analyze_CCpp ---
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES'
Querying server settings
Preparing an archive to upload
Uploading 7.8 MiB
Upload successful
Retrace job started
Analyzing crash data
................................................................................
................................................................................
..
Generating backtrace
................................................................................
..................
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.
[2021-02-06 16:44:45] [I] Analyzing crash data

Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO'
('analyze_CCpp' exited with 1)
===================================================================


Maybe bug 1902312 would be more fitting, I couldn't decide. Or maybe I should just open a new bug, please let me know.

Hello,

This keeps happening to me while trying to report this crash:

https://retrace.fedoraproject.org/faf/reports/27174/

abrt retrace 241df89                                                                Ma 09 feb 2021 06:21:34 +0000
Upload core dump and perform remote retracing? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. Local retracing requires downloading potentially large amount of debuginfo data [y/N] y
Remote retracing
Querying server settings
Preparing an archive to upload
You are going to upload 48,9 MiB. Continue? [y/N] y
Uploading 48,9 MiB
Upload successful
Retrace job started
Analyzing crash data
................................................................................
......................................................................
Generating backtrace
................................................................................
......................
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.
[2021-02-09 06:22:16] [I] Analyzing crash data

And again just now:



$ abrt report cfba423
Reporting problem cfba423

no actions matching this problem found for event 'collect_GConf'
no actions matching this problem found for event 'collect_vimrc_system'
no actions matching this problem found for event 'collect_vimrc_user'
no actions matching this problem found for event 'collect_xsession_errors'
('report_uReport' completed successfully)
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y
Querying server settings
Preparing an archive to upload
You are going to upload 25.5 MiB. Continue? [y/N] y
Uploading 25.5 MiB
Upload successful
Retrace job started
Analyzing crash data
................................................................................
..........................................................................
Generating backtrace
................................................................................
.....................
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.
[2021-02-09 20:32:55] [I] Analyzing crash data

And again:

--- Running report_uReport ---
('report_uReport' completed successfully)

--- Skipping collect_GConf ---
No matching actions found for this event.

--- Skipping collect_vimrc_system ---
No matching actions found for this event.

--- Skipping collect_vimrc_user ---
No matching actions found for this event.

--- Running collect_xsession_errors ---
No /home/christian/.cache/gdm/session.log

--- Running analyze_CCpp ---
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES'
Querying server settings
Preparing an archive to upload
Uploading 107.3 KiB
Upload successful
Retrace job started
Analyzing crash data
.................
Generating backtrace
................................................................................
...
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.
[2021-02-20 16:56:11] [I] Analyzing crash data

Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO'
('analyze_CCpp' exited with 1)


---- Can this report be re-opened?

Still happens (this is for geoclue, but it happens for every crash I try/have tried to report recently)

--- Running report_uReport ---
('report_uReport' completed successfully)

--- Skipping collect_GConf ---
No matching actions found for this event.

--- Skipping collect_vimrc_system ---
No matching actions found for this event.

--- Skipping collect_vimrc_user ---
No matching actions found for this event.

--- Skipping collect_xsession_errors ---
No matching actions found for this event.

--- Running analyze_CCpp ---
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES'
Querying server settings
Preparing an archive to upload
Uploading 1.4 MiB
Upload successful
Retrace job started
Analyzing crash data
.............................................
Generating backtrace
................................................................................
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.
[2021-03-02 10:53:30] [I] Analyzing crash data

Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO'
('analyze_CCpp' exited with 1)

I have the same issue on Fedora 34, but in my case it fails in the middle of "Analyzing crash data"

Still not working, I wasn't successful with a single remote retracing. Reopen.

This is almost certainly not the same problem any more. All you can really tell from "Retrace job failed" is "we sent the dump to the retrace server and it failed somehow trying to analyze it". There's obviously a pretty broad range of possibilities for how that could have happened.

If you're running into persistent failures at that point, it's probably best to file an issue at https://github.com/abrt/retrace-server/issues rather than filing or reopening a Bugzilla bug, as the issue is likely on the retrace server side somewhere.

FWIW, I've been following this issue the whole time, and I'm not convinced this was ever really fixed. I'm not sure if I've seen the retrace server ever *not* fail to process a backtrace in the past year or two.

There is no way to distinguish why processing failed client side, so IMO it's reasonable to lump all possible occurrences of these failures into one issue report. Creating multiple reports for "it never works" doesn't seem useful....

It worked for a bit the last time it was closed; I am back here because, well, f34 is out, things break, and I am trying to file and found the retrace server is not responding... Maybe the redhat folks can re-visit the retrace server a bit before the half-yearly releases to give it a bit of love and care?

Multiple downstream bug reports would not be any use to anyone, but what would probably be more useful is reports of "it is not working for me right now with (details about the crash)" to https://github.com/abrt/retrace-server/issues .

We have a pull request in the pipeline in the upstream that should fix most of these (and related) bugs: https://github.com/abrt/retrace-server/pull/418

I'm hopeful we'll be able to merge and deploy it on retrace.fedoraproject.org early next week.

Still failing:

--- Running analyze_CCpp ---
Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES'
Querying server settings
Preparing an archive to upload
You are going to upload 67.0 MiB. Continue? 'YES'
Uploading 67.0 MiB
Upload successful
Retrace job #249863983 started
Analyzing crash data
.........................................................................
Retrace job failed
Retrace failed. Try again later and if the problem persists report this issue please.

We have deployed an upgrade for Retrace Server yesterday which should fix most of these problems.

At the moment, retracing Fedora 33 and 34 on x86-64 should should proceed with little problems.

(Note that very big coredumps might still take some time and perhaps fail anyway. We'll be looking into that as a separate problem.)

FWIW, I just tried to retrace a Rawhide crash, it failed. Seemed to take a bit longer to fail than it was before, but still failed.

Thanks for the heads up, Rawhide indeed seems broken on retrace.fp.org. I'm looking into it right now.

Retracing should work for Rawhide as well now.

There was a bug in how we were pulling Rawhide components and RPMs.

Thanks again for the heads up, Adam.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days