Thanks for a bug report. What is the account type you see this behaviour on, please? Like On This Computer/, EWS, IMAP, ... I do not have Fedora 26 installed yet, but I have Rawhide which I just updated and I see successful changes of read/unread counts on read/unread state change in the message list on both On This Computer/Inbox and IMAPx/Inbox folders.

Do you see any pending activities in the status bar? Maybe evolution is busy with something and even the UI shows the changes are done they are not necessarily on the server yet, or the UI part is waiting for a confirmation from the other side.

A backtrace of evolution may show whether anything is pending in the background, though if it's in some queue then it's not there. We can try at least, I think. You can get the backtrace with command like this:
   $ gdb --batch --ex "t a a bt" -pid=`pidof evolution` &>bt.txt
Please check the bt.txt for any private information, like passwords, email address, server addresses,... I usually search for "pass" at least (quotes for clarity only).

Ideally have installed up to date (the same version as their binary packages counterparts) versions of the debuginfo packages for evolution and evolution-ata-server itself (no need for their dependencies, only these two).

(In reply to Milan Crha from comment #1)
> Thanks for a bug report. What is the account type you see this behaviour on,
> please? Like On This Computer/, EWS, IMAP, ... I do not have Fedora 26
> installed yet, but I have Rawhide which I just updated and I see successful
> changes of read/unread counts on read/unread state change in the message
> list on both On This Computer/Inbox and IMAPx/Inbox folders.

Seems to be having with IMAPx. I tested it on:
- Gmail setup with IMAPx using GOA
- Hotmail.co.uk set up with IMAPx using GOA
- Other hosting account using IMAPx also using GOA

It didn't happen when I tried it on my university email that uses EWS (also set up using GOA)

> 
> Do you see any pending activities in the status bar? Maybe evolution is busy
> with something and even the UI shows the changes are done they are not
> necessarily on the server yet, or the UI part is waiting for a confirmation
> from the other side.

Not always. Please see the attached screen recording.

> 
> A backtrace of evolution may show whether anything is pending in the
> background, though if it's in some queue then it's not there. We can try at
> least, I think. You can get the backtrace with command like this:
>    $ gdb --batch --ex "t a a bt" -pid=`pidof evolution` &>bt.txt
> Please check the bt.txt for any private information, like passwords, email
> address, server addresses,... I usually search for "pass" at least (quotes
> for clarity only).
> 
> Ideally have installed up to date (the same version as their binary packages
> counterparts) versions of the debuginfo packages for evolution and
> evolution-ata-server itself (no need for their dependencies, only these two).

I'll do this and get back to you.

Thanks,
Ankur

Created attachment 1270554 [details]
Screencast showing bug

Thanks for the update. I see that neither the Window title, nor the label above folder list changes its unread count (they both still show "1 unread" regardless how many times you click on the icon in the message list), it's only the folder list changing the count. I also noticed that the icon in the message list doesn't change, but it should, as soon as the state is propagated to the server.

Could you enable only one affected account in the Preferences, then run evolution with IMAPx debugging on, to see whether it tried to save the state change, please? The command is:

   $ CAMEL_DEBUG=imapx:io evolution

Wait a bit, until the output on the terminal stops coming, then mark somehow the current line and reproduce the problem. The newly added output should show something like:
   [imapx:A] I/O: 'A00020 UID STORE 4 -FLAGS.SILENT (\SEEN)'
   [imapx:A] I/O: 'A00020 OK Completed'
   [imapx:A] I/O: 'A00021 NOOP'
   [imapx:A] I/O: 'A00021 OK Completed'
   [imapx:A] I/O: 'A00022 UID STORE 4 +FLAGS.SILENT (\SEEN)'
   [imapx:A] I/O: 'A00022 OK Completed'
which is not shown immediately, but after few seconds since the click on the icon in the message list. The UI updates immediately though, at least here.

Created attachment 1271861 [details]
Output of "gdb --batch --ex "t a a bt" -pid=`pidof evolution` &>bt.txt" when replicating bug

I disabled all accounts but one and while I repeatedly clicked on a message and the unread count kept increasing, I got the bt.

> Could you enable only one affected account in the Preferences, then run
> evolution with IMAPx debugging on, to see whether it tried to save the state
> change, please? The command is:
> 
>    $ CAMEL_DEBUG=imapx:io evolution
> 
> Wait a bit, until the output on the terminal stops coming, then mark somehow
> the current line and reproduce the problem. The newly added output should
> show something like:
>    [imapx:A] I/O: 'A00020 UID STORE 4 -FLAGS.SILENT (\SEEN)'
>    [imapx:A] I/O: 'A00020 OK Completed'
>    [imapx:A] I/O: 'A00021 NOOP'
>    [imapx:A] I/O: 'A00021 OK Completed'
>    [imapx:A] I/O: 'A00022 UID STORE 4 +FLAGS.SILENT (\SEEN)'
>    [imapx:A] I/O: 'A00022 OK Completed'
> which is not shown immediately, but after few seconds since the click on the
> icon in the message list. The UI updates immediately though, at least here.

So, when the bug doesn't occur, these messages do turn up, but when the bug occurs, it stops at "idling" and then unless I check another folder or another e-mail, no other updates occur - it won't even mark an unread message as read if I simply select it. If the bug has occurred, though, even after I switch to another e-mail in the same folder, the unread counts will remain wrong howmany ever times I click it:

[imapx:A] I/O: ')'
[imapx:A] I/O: 'A00492 OK Success'
[imapx:A] I/O: 'A00493 SELECT INBOX'
[imapx:A] I/O: '* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Label1 $Labelimportant $Labellater $Labelpersonal $Labeltodo $Labelwork $NotPhishing $Phishing $has_cal JUNK NOTJUNK receipt-handled)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $Label1 $Labelimportant $Labellater $Labelpersonal $Labeltodo $Labelwork $NotPhishing $Phishing $has_cal JUNK NOTJUNK receipt-handled \*)] Flags permitted.
* OK [UIDVALIDITY 2] UIDs valid.
* 1914 EXISTS
* 0 RECENT
* OK [UIDNEXT 431873] Predicted next UID.
* OK [HIGHESTMODSEQ 39001104]
A00493 OK [READ-WRITE] INBOX selected. (Success)'
[imapx:A] I/O: 'A00494 IDLE'
[imapx:A] I/O: '+ idling'


It seems to get stuck on "idling" behaviour for this particular folder.

It's erratic though. I've been trying to come up with steps that would reproduce it all the time but I haven't been able to yet. Sometimes it'll happen, sometimes it won't.

I've removed all my local evolution files (.config, .local, .cache) and am setting my accounts up again - just in case.

(In reply to Ankur Sinha (FranciscoD) from comment #7)
> I've removed all my local evolution files (.config, .local, .cache) and am
> setting my accounts up again - just in case.

This didn't seem to help - I just ran into it again :/

Thanks for the update. Your backtrace shows basically the same. There is one IMAPx thread in IDLE call, then one thread from EWS also waiting for change notifications from the server. Otherwise Evolution is just in a state of "waiting for orders", according to the backtrace.

What do you do to reproduce the issue? I know you said there are no exact steps for the reproducer, but I guess you do particular things before you manage to reproduce it, like selecting messages which were not downloaded yet, marking other messages as read/unread, and so on. Maybe opening folder properties?

If I got it right, then a change to other folder and back makes things work properly again, until the issue happens again?

I think I'll create a test build for you, which will print some useful debugging information on the console, to see what it does in the background. Maybe there's some race condition in time when the IDLE should be stopped, but it isn't.

I tinkered a bit more, and it only happens in folders that have *not* received any new messages, i.e., when I click on a folder, and the folder is updated, it does not receive any new messages. The folder may or may not have unread messages in it before it was selected, but that does not seem to affect the bug.

I double checked this - I clicked on a folder and it had a few new messages and the bug wouldn't show - the folder count would update correctly. Then I immediately clicked on another folder and came back to the previous one - this time there were no new messages and sure enough, the bug occurred.

Then:

- toggling the "read/unread" status of the message does not
* update the window title
* change the "read/unread" icon
* correctly update the folder unread message count (it always increases when the message should be marked as unread, but does not decrease when the message should be marked as read)

- clicking on a different folder and then returning to the previous "buggy" one resets the unread count to pre-toggling values, but not the view - so the view and folder count are incoherent now (one of them is wrong)

- "mark all messages as unread" does not work either
- "refresh" folder doesn't do anything either.

- In an affected folder, deleting messages also seems to get stuck - the messages are struck-through, but the view isn't updated and the status bar does not show any processes happening. If one clicks on a different folder and then returns to the folder, the view is updated, and so is the folder unread count, when the status bar says "refreshing folder ..."
* In this situation, "refresh folder" does work, though - it removes the deleted (struck-through) message from the view and correctly updates the folder count.


I've now seen this issue on my second work machine too - completely different hardware, completely up to date F26 (but also upgraded from F25), so at least it doesn't point to being a one machine issue.

Thank you for suggesting the test build - I'll be more than happy to help get more debuginfo that would help solve the issue.

Cheers!

I have such folders too, but still not able to reproduce this. Here [1] is a test build of evolution-data-server, which adds tons of prints on the evolution console. It will be removed within a week or so, I think. Ideally like before, have active only one IMAP account, then run evolution from a terminal (this time without IMAPx logging) and wait until the output stops coming. "Mark the line in the output" and then select one of the folders and reproduce the issue. I would try several times (click the unread icon several times). An immediate change doesn't do anything on the console, it's done after "store_changes_interval", which is printed as one of the top lines of the log after starting evolution, like here:
> * 0x2123560 camel_offline_settings_set_store_changes_interval: 0 ~> 3
Unfortunately, it's printed for disabled accounts too.

Anyway, after this timeout (in seconds) the just made changes are saved to the server, which is supposed to produce new output on the terminal.

It's possible that these prints will not uncover any issue and I'd go deeper and I will create another debug build with other prints, thus please bare with me. Interestingly, I noticed one memory leak (caused by missing unref), which I'm going to fix shortly, after some more testing. It's part of this debug build.

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=19076872

(In reply to Milan Crha from comment #11)
> Interestingly, I noticed one memory leak (caused by missing unref)

Yeah, it's valid memory leak. Fixed for 3.25.1+ and 3.24.2+ with:
https://git.gnome.org/browse/evolution-data-server/commit/?h=gnome-3-24&id=778b08f78cd

Hi Milan,

1. So I installed the new build:

[asinha@ankur  ~]$ rpm -q evolution-data-server
evolution-data-server-3.24.1-1.1.fc26.x86_64

2. Then, to have a clean environment, I created a brand new user, switched to this user, and set up Google in GOA for this user. So, only one account here.

3. Then, I ran evolution from the terminal

4. Then, I opened the account settings for this new e-mail account and:
4.a. set my name
4.b. in Receiving options, I have the following ticked:
4.b.i. Check in all folders
4.b.ii Check in subscribed folders
4.b.iii Quick resync
4.b.iv Server change notifications

5. Then, I clicked "Receive all" to update all my folders

6. Then, I viewed each folder to make sure that the folders were up to date. I just started from inbox and selected each folder using the down arrow key

7. I changed the preview to "vertical"
---

This was the setup. Then I went ahead to reproduce the issue:

1. I clicked on a folder that had unread e-mails - Bodhi (Attachment: evo-bodhi.txt)

a - an unread e-mail was selected, and even when I let it remain selected, it was not marked as "seen"

b. I clicked the "read/unread" button multiple times and the folder count went wrong again.

c. I then changed folders by clicking on the "Bugzilla" folder. The folder unread count for the "Bodhi" folder remained wrong when I did this. 

d. Then I returned to the "Bodhi" folder, and the folder count corrected itself.
--

2. I clicked on the "CommOps" folder which also had unread e-mails in it. (Attachment - evo-commops.txt) and saw the same behaviour on clicking the "read/unread" icon.

3. Similarly for the "Journals" folder.


In all these tests, I waited after clicking the "read/unread" icon, but the output didn't change even after a while. It just stayed as it was until I changed folders.

I hope this helps somewhat. Please let me know if there are more steps you'd like me to take to get more debug info.

Cheers!

Created attachment 1272906 [details]
Folder "journals"

Created attachment 1272907 [details]
Folder "bodhi"

Created attachment 1272908 [details]
Folder "commops"

Thanks for the update. Just by an accident, I've been able to reproduce it today. It was rather special, from my point of view, but still possible, thus definitely needs to be fixed. In my case, the account could not be connected after start, which resulted in an error above message list and no folders shown in the folder tree on the left, only the account itself had been there. Once I fixed the account login I could expand the account node in the folder tree and it showed folders, but when I selected one of those folders and played with read/unread there, then it behaved exactly the same as you described. I have similar debug prints as you.

I made changes, this time in evolution, which fixed it for me, but I'd like to ask you to test it before I commit, thus I have a confirmation for the change. The test build of the evolution package is here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=19101893

Hi Milan,

Unfortunately, it didn't fix it for me :(

evolution-3.24.1-1.1.fc26.x86_64

Do you think I should maybe try to install F26 afresh on a VM and see how that goes? The one thing in common between these two systems is that they were both upgrades - maybe there's a remnant of the upgrade that's causing the issue?

Hrm, I hoped it'll work. Here [1] is a new test build. It's the same as the previous one, expect I added debug prints on various places. It's pretty chatty, thus the less accounts you've enabled the better. While reproducing, please state which folder you've reproduced this with, then it will be possible to search the log and check what happened to that particular folder.

The debug prints expose the account name and all your folders in those accounts, thus if you'd not want to share this in public, then feel free to send the log only to my bugzilla email, just reference this bug report in the Subject, otherwise I might overlook it in my spam folder.

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=19200490

Hi Milan,

I've sent you the logs via e-mail. The subject of the e-mail is:

"Bug 1440068 - Folders do not show correct unread message count - debug logs"

Thanks again,
Ankur

Hi Milan,

Just a quick note - the bug just happened with another account that is EWS based and not imap, so this may not be an imap specific issue :(

Thanks,
Ankur

(In reply to Ankur Sinha (FranciscoD) from comment #20)
> I've sent you the logs via e-mail.

Thanks, it had been received (twice) and I think I see it. I bet you use real Trash and Junk folders for both accounts, which is set automatically these days, after the first connection to the server. It can be seen in the Defaults tab of the account Properties. I do not do that, which explains why I cannot reproduce it.

I'll commit changes from comment #17 separately, then I'll look on the cause of this bug report. I think that something similar is filled in GNOME bugzilla too, something older, not so recent.

(In reply to Milan Crha from comment #22)
> (In reply to Ankur Sinha (FranciscoD) from comment #20)
> > I've sent you the logs via e-mail.
> 
> Thanks, it had been received (twice) and I think I see it. I bet you use
> real Trash and Junk folders for both accounts, which is set automatically
> these days, after the first connection to the server. It can be seen in the
> Defaults tab of the account Properties. I do not do that, which explains why
> I cannot reproduce it.
> 

Yes - I just checked settings and it does use real folders for both Trash and Junk.


> I'll commit changes from comment #17 separately, then I'll look on the cause
> of this bug report. I think that something similar is filled in GNOME
> bugzilla too, something older, not so recent.

Great. Please do let me know if you need me to test more builds. Always happy to do so.

(In reply to Ankur Sinha (FranciscoD) from comment #23)
> Great. Please do let me know if you need me to test more builds. Always
> happy to do so.

Okay, thanks. I committed that initial part for evolution 3.25.2+ and 3.24.2+.

A quick test with a new user and Gmail account configured through GNOME Online Accounts with real Trash and Junk folders did not reproduce the issue here, unfortunately. I left the account as it was preconfigured, but I also tried with some changes in account settings, but still nothing.

Could you give me your settings from
   ~/.config/evolution/sources/xxx.source
file, the one where [Imapx Backend] section exists, with anything private censored, please? There can be also some settings in the evolution Preferences, though I guess you didn't change anything when you created the new user, did you?

I tried with steps from above, but it still doesn't reproduce it here. I guess what happened here is that there are two instances of that offending folder in memory. The MailFolderCache knew about two before the folder change notification broke. The thing is that there should be always only one instance of the folder, while there are two at the moment. It seems at that to me at least. Having non-real Trash or Junk folder makes the CamelStore hold a cache of existing folders and return exactly one instance, but when both are set as real folders the cache is basically emptied with the last unref of the folder. It can be there's missing an unref for some ref, the log doesn't show any such thing.

I'd like to reproduce it locally though. Maybe, if you could create a backup of evolution settings from the new user (File->Backup Evolution Data...), with which you can reproduce the issue, then I can restore it here, change the email address there (and GOA account ID) and see whether it'll reproduce the issue here as well, with exactly the same evolution settings. Send it privately to me, if you'll do, please.

I'm sorry to ask, but could you summarize the steps in some very high detail for the reproducer, please? I feel shamed to ask it, but I think I miss some little detail for some reason (some issue on my side). By the way, I run Fedora 26 Alpha in a virtual machine with a fresh user. I believe that the difference between real hardware and a virtual machine is nothing crucial.

Here are two other test builds. They do not do anything more than before, just are more chatty on console. Please install them and reproduce the issue like before. The captured log will help me to understand a bit more what it does in the background. The questions from the previous comment still apply. Thanks in advance.

https://koji.fedoraproject.org/koji/taskinfo?taskID=19393860
https://koji.fedoraproject.org/koji/taskinfo?taskID=19393914

Hi Milan,

I've downloaded the builds and am trying to test them. I wanted to try it out on a fresh account and a fresh install on VM too, just to be sure it isn't something to do with the upgrade. At the moment, GOA can't set up Google accounts, though - I'm waiting for the fix now. 

I'll have the info as soon as possible. 

https://bugzilla.redhat.com/show_bug.cgi?id=1448651

Thanks,

Hi Milan,

I just e-mailed you the logs and a screencast too. 

Cheers,
Ankur

Thanks. Your email contained steps with which I've been able to reproduce the misbehaviour and I even found the cause, it was a memory leak of the message info (a structure which described a message). It's a bit complicated, thus I'll rather describe it into the upstream bug report [1] and then I'll make a new Fedora 26 release with that fix included.

[1] https://bugzilla.gnome.org/show_bug.cgi?id=782096

evolution-data-server-3.24.2-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a15cebdb95

evolution-data-server-3.24.2-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a15cebdb95

evolution-data-server-3.24.2-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.