Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1886567 - libdnf 0.54.2 / dnf 4.4.0 break ARM disk image builds (and hence Rawhide composes)
Summary: libdnf 0.54.2 / dnf 4.4.0 break ARM disk image builds (and hence Rawhide comp...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: livecd-tools
Version: rawhide
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Neal Gompa
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: F34BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2020-10-08 18:59 UTC by Adam Williamson
Modified: 2020-10-19 21:41 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-19 21:41:52 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2020-10-08 18:59:49 UTC
Today's Rawhide and Branched composes both failed on the ARM minimal disk image task failing, with this traceback:

DEBUG util.py:634:  Traceback (most recent call last):
DEBUG util.py:634:    File "/usr/bin/appliance-creator", line 193, in <module>
DEBUG util.py:634:      sys.exit(main())
DEBUG util.py:634:    File "/usr/bin/appliance-creator", line 154, in main
DEBUG util.py:634:      creator.install()
DEBUG util.py:634:    File "/usr/lib/python3.9/site-packages/imgcreate/creator.py", line 722, in install
DEBUG util.py:634:      self.__apply_selections(dbo)
DEBUG util.py:634:    File "/usr/lib/python3.9/site-packages/imgcreate/creator.py", line 625, in __apply_selections
DEBUG util.py:634:      dbo.selectGroup('core', excludedPkgs)
DEBUG util.py:634:    File "/usr/lib/python3.9/site-packages/imgcreate/dnfinst.py", line 132, in selectGroup
DEBUG util.py:634:      self.group_install(grp.id, package_types, exclude=exclude)
DEBUG util.py:634:    File "/usr/lib/python3.9/site-packages/dnf/base.py", line 1624, in group_install
DEBUG util.py:634:      pkg_types = libdnf.transaction.listToCompsPackageType(pkg_types)
DEBUG util.py:634:    File "/usr/lib/python3.9/site-packages/libdnf/transaction.py", line 424, in listToCompsPackageType
DEBUG util.py:634:      return _transaction.listToCompsPackageType(types)
DEBUG util.py:634:  TypeError: in method 'listToCompsPackageType', argument 1 of type 'std::vector< std::string,std::allocator< std::string > > const &'

both composes ran with libdnf-0.54.2 and dnf-4.4.0 . This seems to be the only significant difference from yesterday's successful tasks. The F33 compose only ran with the updated packages because a buildroot override was active at the time (it's since been expired), so we've just re-run the F33 compose and it should be okay, but this will break Rawhide composes until it's fixed or the packages are untagged (mboddu is going to untag them for now and re-run the compose).

Is this a dnf/libdnf bug, or does imgcreate need to change something to work right with the new version?

Comment 1 Adam Williamson 2020-10-08 19:11:31 UTC
From the imgcreate source, I believe excludedPkgs is a list.

Comment 2 Adam Williamson 2020-10-08 19:21:21 UTC
oh, it seems we're actually concerned about package_types / pkg_types here. imgcreate passes it as a set:

        package_types = {'mandatory', 'default'}
        if include == GROUP_REQUIRED:
            package_types.remove('default')
        elif include == GROUP_ALL:
            package_types.add('optional')

looking at dnf source, I can see cases where it's passed as a tuple (in several of the tests and in dnf/cli/commands/install.py via env_group_install) and as a list (in some of the examples, at least). I don't see any cases of it being passed as a set. The API docs state "`pkg_types` is a sequence of strings...". Python 3 docs state "There are three basic sequence types: lists, tuples, and range objects." - https://docs.python.org/3/library/stdtypes.html - so by implication a set is not considered a "sequence", so I guess we can say livecd-creator (where imgcreate comes from) is kinda out of bounds here and should convert the package types to a tuple or list before passing them. Though since it worked before it might be nice if it could be made to keep working, I guess.

Comment 3 Adam Williamson 2020-10-08 19:30:02 UTC
Looks like the upstream change that triggered this was probably https://github.com/rpm-software-management/dnf/commit/4d991f6154cbf888d304f7accb0a7024951e6b68 . I'm having trouble figuring where the hell 'listToCompsPackageType' actually comes from though. That string doesn't exist in the libdnf source. Must be weird Python binding stuff?

Comment 4 Adam Williamson 2020-10-08 19:45:56 UTC
Sigh, I'm looking on the wrong libdnf branch. Not 'master' but 'dnf-4-master' is right. Still, that doesn't tell us a lot more, just confirms that listToCompsPackageType is expecting a vector of strings and presumably not getting one. I tried to figure out what C++ type SWIG converts a Python set of strings to, but didn't get very far because, well, it's complicated and hard to understand. But I'm guessing the answer is "not a vector of strings".

Comment 5 Adam Williamson 2020-10-08 20:04:21 UTC
OK, yup, I think I'm right.

>>> libdnf.transaction.listToCompsPackageType({"mandatory", "default"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/site-packages/libdnf/transaction.py", line 424, in listToCompsPackageType
    return _transaction.listToCompsPackageType(types)
TypeError: in method 'listToCompsPackageType', argument 1 of type 'std::vector< std::string,std::allocator< std::string > > const &'
>>> libdnf.transaction.listToCompsPackageType(tuple({"mandatory", "default"}))
6
>>> 

I'll send a PR for livecd-tools and backport it.

Comment 6 Adam Williamson 2020-10-08 20:45:23 UTC
https://koji.fedoraproject.org/koji/taskinfo?taskID=53027536 should fix this for Rawhide. I also built it for F33 - https://koji.fedoraproject.org/koji/taskinfo?taskID=53027564 - and that build should go into the F33 update for dnf 4.4.0 assuming one is submitted.

Comment 7 Adam Williamson 2020-10-08 21:14:14 UTC
For dnf folks: I checked the dnf code back to 2013 and it looks to me like it always worked with any kind of iterable, because it just used `if X in pkg_types:` style logic. So even though the wording in the API docs kinda does suggest only sequences are allowed, it might be a good idea if possible to have the libdnf code also handle sets and maybe any other iterable types we can think of, just in case there's other code lurking out there which does something similar. I checked the things I could think of (anaconda and dnfdaemon) and they both look okay, but there may well be other things that use this codepath.

Comment 8 Adam Williamson 2020-10-08 21:15:47 UTC
oh, I suppose you could also try to handle this in dnf by just changing:

            types = libdnf.transaction.listToCompsPackageType(types)

to:

            types = libdnf.transaction.listToCompsPackageType(list(types))

or tuple(types).

Comment 9 Adam Williamson 2020-10-19 21:41:52 UTC
This has been fixed for a while.


Note You need to log in before you can comment on or make changes to this bug.