Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1221165 - [RFC] What is the future of par2cmdline?
Summary: [RFC] What is the future of par2cmdline?
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: par2cmdline
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Erik van Pienbroek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-13 12:05 UTC by Tadej Janež
Modified: 2017-01-10 13:47 UTC (History)
3 users (show)

Fixed In Version: par2cmdline-0.6.14-1.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-30 02:20:47 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Tadej Janež 2015-05-13 12:05:04 UTC
Hi!

I'm packaging bup [0], which requires par2 and I've tried to research what's the latest state around par2cmdline.

The current packaged version of par2cmdline is outdated and can leave files in worse condition than before running par2: https://bugzilla.redhat.com/show_bug.cgi?id=1197560.
Also, it fails to build from source since F21 mass rebuild: https://bugzilla.redhat.com/show_bug.cgi?id=1105926.

The chuchusoft fork [1] of par2cmdline that is currently packaged has been updated two times lately (version 20141125 and version 20150503) [2]. There is a bug report requesting this update: https://bugzilla.redhat.com/show_bug.cgi?id=1198492, but it hasn't been acted upon yet.

In August 2014, a reporter asked [3] if it would make sense to switch to a more active BlackIkeEagle fork [4] (which has been later renamed to just Parchive [5]). The package maintainer, Erik van Pienbroek, said he refuses to switch to this fork due to lack of TBB (Intel Threading Building Blocks) support implemented by the chuchusoft fork [6].

In March 2015, the author of the BlackIkeEagle fork advised [7] to check out jkansanen fork [8], which implements multi-threaded (OpenMP) version of par2cmdline. Additionally, he said:
"I will eventually ask if I can integrate his solution or see if I find another solution suitable for everyone.

The solution must be supporting threads and be as widely as possible usable so maybe intel tbb and openmp are not perfect solutions for this but usable if you want it now."

So, my question is, what is the future of par2cmdline?

Thanks and regards,
Tadej


[0] http://bup.github.io/
[1] http://www.chuchusoft.com/par2_tbb/
[2] http://www.chuchusoft.com/par2_tbb/download.html
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1133401
[4] https://github.com/BlackIkeEagle/par2cmdline
[5] https://github.com/Parchive/par2cmdline
[6] https://bugzilla.redhat.com/show_bug.cgi?id=1133401#c1
[7] https://github.com/Parchive/par2cmdline/issues/27#issuecomment-76959794
[8] https://github.com/jkansanen/par2cmdline-mt

Comment 1 Tadej Janež 2015-05-13 12:10:46 UTC
FWIW, Debian uses the BlackIkeEagle (Parchive) fork:
https://tracker.debian.org/pkg/par2cmdline

Comment 2 Erik van Pienbroek 2015-05-15 18:00:44 UTC
I agree that the situation with par2 is pretty messy these days, but I don't know what the future is of par2cmdline (as I'm only the downstream Fedora packager).. I was hoping that the BlackIkeEagle fork would have threading support by now, but apparently this takes longer than expected.

In order to decide what path to go more testing is needed of the various possible solutions. I'm tempted to start off with the 0.6.12 release (https://github.com/Parchive/par2cmdline/releases/tag/v0.6.12) and backport the OpenMP pieces onto this: https://github.com/Parchive/par2cmdline/compare/3038e09fcee00f4dd628bcb040ebd43741ebadfb...jkansanen:master

If this works reliable we can put it up for testing in Fedora to give it more attention. By doing this we can help upstream with testing these changes and increasing the chance that these changes will be accepted upstream more quickly.

Comment 3 Erik van Pienbroek 2015-05-15 20:17:41 UTC
I've prepared a test build containing OpenMP support @ http://koji.fedoraproject.org/koji/taskinfo?taskID=9755424
Could you test if this works for you?

I've done some initial testing myself and things look good on it. It is using all my available CPU cores during repair. This package was also able to repair some files where the old par2-tbb failed.

I've also done some time tests (to verify and repair some broken files) with the various versions and here are the results:
par2-tbb:
real	2m13.139s
user	14m36.899s
sys	0m6.576s

par2cmdline 0.6.12:
real	5m29.718s
user	5m27.731s
sys	0m1.223s

par2cmdline 0.6.12 with OpenMP support:
real	3m34.538s
user	23m47.680s
sys	0m2.383s

So with OpenMP changes the performance improves nicely but it's not on par yet with par2-tbb. But on the other hand the new par2cmdline is able to repair files which par2-tbb wasn't able to so that's a plus.

Comment 4 Tadej Janež 2015-05-20 08:44:27 UTC
(In reply to Erik van Pienbroek from comment #2)
> I agree that the situation with par2 is pretty messy these days, but I don't
> know what the future is of par2cmdline (as I'm only the downstream Fedora
> packager).

Sure, I understand.

> In order to decide what path to go more testing is needed of the various
> possible solutions. I'm tempted to start off with the 0.6.12 release
> (https://github.com/Parchive/par2cmdline/releases/tag/v0.6.12) and backport
> the OpenMP pieces onto this:
> https://github.com/Parchive/par2cmdline/compare/
> 3038e09fcee00f4dd628bcb040ebd43741ebadfb...jkansanen:master

Seems like a good plan.

> If this works reliable we can put it up for testing in Fedora to give it
> more attention. By doing this we can help upstream with testing these
> changes and increasing the chance that these changes will be accepted
> upstream more quickly.

Agreed.

(In reply to Erik van Pienbroek from comment #3)
> I've prepared a test build containing OpenMP support @
> http://koji.fedoraproject.org/koji/taskinfo?taskID=9755424
> Could you test if this works for you?

Thanks! I'll test this and report back.
 
> I've done some initial testing myself and things look good on it. It is
> using all my available CPU cores during repair. This package was also able
> to repair some files where the old par2-tbb failed.
> 
> I've also done some time tests (to verify and repair some broken files) with
> the various versions and here are the results:
> par2-tbb:
> real	2m13.139s
> user	14m36.899s
> sys	0m6.576s
> 
> par2cmdline 0.6.12:
> real	5m29.718s
> user	5m27.731s
> sys	0m1.223s
> 
> par2cmdline 0.6.12 with OpenMP support:
> real	3m34.538s
> user	23m47.680s
> sys	0m2.383s
> 
> So with OpenMP changes the performance improves nicely but it's not on par
> yet with par2-tbb. But on the other hand the new par2cmdline is able to
> repair files which par2-tbb wasn't able to so that's a plus.

Great. I would go for ability to repair first and performance later.

BTW, why are 'user' times so different for the different runs above?

Comment 5 Erik van Pienbroek 2015-05-20 09:53:10 UTC
(In reply to Tadej Janež from comment #4)
> BTW, why are 'user' times so different for the different runs above?
My guess would be that the par2-tbb and openmp versions both use multiple CPU cores but I'm not sure about that

Comment 6 Robin Laing 2015-05-22 18:47:05 UTC
I think it will be nice to see a consolidation of forks to fix and improve all projects.  This will start to clean up the mess.  I am following the parchive-devel mail list on this.

Comment 7 Jan Kurik 2015-07-15 14:10:04 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Comment 8 Fedora Update System 2015-10-18 22:47:25 UTC
par2cmdline-0.6.14-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-d36d127ea7

Comment 9 Fedora Update System 2015-10-19 16:25:23 UTC
par2cmdline-0.6.14-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update par2cmdline'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-d36d127ea7

Comment 10 Robin Laing 2015-10-20 01:43:19 UTC
I had just downloaded the Github version to repair an archive that the F22 version of par2 wouldn't fix.  Yes it is slower not being multithreaded but it is better since it actually fixes the archive instead of leaving a major mess.

Thank you for moving ahead.

Comment 11 Fedora Update System 2015-10-28 21:25:28 UTC
nntpgrab-0.7.2-12.fc23 par2cmdline-0.6.14-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-d36d127ea7

Comment 12 Fedora Update System 2015-11-01 06:59:49 UTC
nntpgrab-0.7.2-12.fc23, par2cmdline-0.6.14-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nntpgrab par2cmdline'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-d36d127ea7

Comment 13 Fedora Update System 2015-11-30 02:20:32 UTC
nntpgrab-0.7.2-12.fc23, par2cmdline-0.6.14-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Reik Red 2016-09-26 03:51:33 UTC
Okay, so I now have par2cmdline-0.6.14-1.fc23, which per the above I have deduced (hopefully correctly) should be safe to use with "bup fsck -g". But there is no new version of bup in fc23 that allows par2 to be run, and, AFAICT, no override flag.

What should I do? Download the latest bup src and compile it myself? Or is there an fc 24 version where par2 has been enabled, that might dnf install in fc23?

Comment 15 Tadej Janež 2016-09-26 07:16:57 UTC
Hi, Reik!

(In reply to Reik Red from comment #14)
> Okay, so I now have par2cmdline-0.6.14-1.fc23, which per the above I have
> deduced (hopefully correctly) should be safe to use with "bup fsck -g". But
> there is no new version of bup in fc23 that allows par2 to be run, and,
> AFAICT, no override flag.
> 
> What should I do? Download the latest bup src and compile it myself? Or is
> there an fc 24 version where par2 has been enabled, that might dnf install
> in fc23?

I'm the maintainer of bup in Fedora. I haven't yet updated bup to version 0.28 (tracking bug is #1323522). After the update, I'll test par2 support and if successful, reenable par2cmdline support in bup afterwards.

Comment 16 Reik Red 2016-09-26 18:34:25 UTC
(In reply to Tadej Janež from comment #15)

I decided to recompile bup and install to .usr/local to get a solution that works right away. It was pretty simple given that bup is mostly python anyway. Seems to work. People who are in a a pinch getting "bup fsck -g" to work in Fedora 23 or 24 may want to try this.

% /bin/bup --version
unknown-d612d9a
% /usr/local/bin/bup --version
0.28.1-23-g461b5b4

How to compile, assuming you already have the necessary pytho stuff, otherwise refer to https://github.com/bup/bup

mkdir ~/src
cd ~/src
git clone https://github.com/bup/bup
cd bup
./configure |& tee log.configure
make | & tee log.make
make install -n
# install goes to /usr/local/
sudo make install | & tee log.make.install
make test |& tee log.make.test

make test has 2 errors related to xattr.set, but that appears to be just some selinux weirdness.

Comment 17 Tadej Janež 2017-01-10 13:47:53 UTC
(In reply to Tadej Janež from comment #15)
> 
> I'm the maintainer of bup in Fedora. I haven't yet updated bup to version
> 0.28 (tracking bug is #1323522). After the update, I'll test par2 support
> and if successful, reenable par2cmdline support in bup afterwards.

For the record, bup 0.28.1 with par2 support is available on Fedora 24+:
https://bodhi.fedoraproject.org/updates/FEDORA-2016-1bf266af1e
https://bodhi.fedoraproject.org/updates/FEDORA-2016-addd1684e1


Note You need to log in before you can comment on or make changes to this bug.