Errors from the log file - ovirt-hosted-engine-setup-ansible-create_storage_domain-2019025212943-xqti5r.log

<snip>
2019-01-25 21:30:16,161+0530 DEBUG ansible on_any args TASK: oVirt.hosted-engine-setup : Add glusterfs storage domain kwargs is_conditional:False 
2019-01-25 21:30:19,134+0530 DEBUG var changed: host "localhost" var "otopi_storage_domain_details_gluster" type "<type 'dict'>" value: "{
    "changed": false, 
    "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_ovirt_storage_domain_payload_RxcsV3/__main__.py\", line 682, in main\n    ret = storage_domains_module.create()\n  File \"/tmp/ansible_ovirt_storage_domain_payload_RxcsV3/ansible_ovirt_storage_domain_payload.zip/ansible/module_utils/ovirt.py\", line 587, in create\n    **kwargs\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 25077, in add\n    return self._internal_add(storage_domain, headers, query, wait)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n    return future.wait() if wait else future\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n    return self._code(response)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n    self._check_fault(response)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n    self._raise_error(response, body)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n    raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[Storage format is unsupported]\". HTTP response code is 400.\n", 
    "failed": true, 
    "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage format is unsupported]\". HTTP response code is 400."
}"
</snip>

We are using ovirt_storage_domain ansible module with glusterfs parameter.

https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/tasks/create_storage_domain.yml#L63

Ondra, any hint?

Created attachment 1523592 [details]
ovirt-hosted-engine-setup-ansible-create_storage_domain

Do you also have engine.log?

(In reply to Simone Tiraboschi from comment #4)
> Do you also have engine.log?

I have reprovisioned systems for RHV 4.2.8-1 testing. I have hit the problem with another set of systems.
Same issue persists. I have captured the engine.log as well.

Below is the snip from engine.log
<snip>
2019-01-30 07:23:42,382+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [646ed358] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM rhsqa-grafton8-nic2.lab.eng.blr.redhat.com command CreateStorageDomainVDS failed: Domain version `5` is unsupported by this version of VDSM: ''
2019-01-30 07:23:42,382+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [646ed358] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return value 'StatusOnlyReturn [status=Status [code=394, message=Domain version `5` is unsupported by this version of VDSM: '']]'
2019-01-30 07:23:42,382+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [646ed358] HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com
2019-01-30 07:23:42,382+05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [646ed358] Command 'CreateStorageDomainVDSCommand(HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com, CreateStorageDomainVDSCommandParameters:{hostId='a8e8fdba-6353-48e6-8d17-68b01432d66e', storageDomain='StorageDomainStatic:{name='hosted_storage', id='97b88734-885c-4f62-be61-5ff87fee10de'}', args='rhsqa-grafton8.lab.eng.blr.redhat.com:/engine'})' execution failed: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Domain version `5` is unsupported by this version of VDSM: ''
</snip>

Created attachment 1524829 [details]
engine.log

Resetting needinfo on Ondra

(In reply to SATHEESARAN from comment #5)
> (In reply to Simone Tiraboschi from comment #4)
> > Do you also have engine.log?
> 
> I have reprovisioned systems for RHV 4.2.8-1 testing. I have hit the problem
> with another set of systems.
> Same issue persists. I have captured the engine.log as well.
> 
> Below is the snip from engine.log
> <snip>
> 2019-01-30 07:23:42,382+05 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-1) [646ed358] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802),
> VDSM rhsqa-grafton8-nic2.lab.eng.blr.redhat.com command
> CreateStorageDomainVDS failed: Domain version `5` is unsupported by this
> version of VDSM: ''
> 2019-01-30 07:23:42,382+05 INFO 
> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand]
> (default task-1) [646ed358] Command
> 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand'
> return value 'StatusOnlyReturn [status=Status [code=394, message=Domain
> version `5` is unsupported by this version of VDSM: '']]'

It seams that you mixed up a newer version of the engine with an older vdsm or something like that.
Nir?

I see on that env:

vdsm.x86_64                                   4.30.8-2.el7ev
rhvm-appliance.x86_64                         2:4.3-20190124.1.el7

(In reply to Simone Tiraboschi from comment #8)
> 
> It seams that you mixed up a newer version of the engine with an older vdsm
> or something like that.
> Nir?

Hi Simone, 

I have used the latest RHVH 4.3 & latest RHVH 4.3 appliance. 
There are no moving parts here.

I think that we need a respin with https://gerrit.ovirt.org/#/c/97262/

We definitively need also https://gerrit.ovirt.org/#/c/97408/

A new respin is on the way

(In reply to Simone Tiraboschi from comment #15)
> We definitively need also https://gerrit.ovirt.org/#/c/97408/
> 
> A new respin is on the way

When can we expect this change in the build ?

Same error for me using NFS

[ INFO ] TASK [oVirt.hosted-engine-setup : Fetch Datacenter ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Fetch Datacenter name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Add NFS storage domain]
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage format is unsupported]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage format is unsupported]\". HTTP response code is 400."}

Created attachment 1525462 [details]
failed deploy - NFS

No moving parts here either. This is a fresh Centos 7.6 VM, yum updated, deployed 4.3-pre just now.

fix included in ovirt-engine-4.3.0.4 , appliance ovirt-engine-appliance-4.3-20190131.1.el7.x86_64

build published as rc5

Verified with the latest RHVM appliance and RHVH, but no errors are seen for RHHI-V deployment

Components used:
RHVH-4.3-20190201.0-RHVH-x86_64-dvd1.iso
rhvm-appliance-4.3-20190201.0.el7.x86_64.rpm

This bugzilla is included in oVirt 4.3.0 release, published on February 4th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

I think this BUG has appeared again in newest version on Hosted Engine appliance.

I have multiple times tried to move Hosted Engine to another oVirt node. I'm starting HE deployment with command: 
# hosted-engine --deploy --restore-from-file=/root/HE_BCK_2019.05.28.bck

At storage selection I setup NFS share from company's NAS storage. Each try has failed:

          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: nfs
          Please specify the nfs version you would like to use (auto, v3, v4, v4_1)[auto]: v4
          Please specify the full shared storage connection path to use (example: host:/path): mynfsserver:/he_nas
          If needed, specify additional mount options for the connection to the hosted-engine storagedomain (example: rsize=32768,wsize=32768) []:
[ INFO  ] Creating Storage Domain
[ INFO  ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Force facts gathering]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Check local VM dir stat]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Enforce local VM dir existence]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Fetch host facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Fetch cluster ID]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Fetch cluster facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Fetch Datacenter facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Fetch Datacenter ID]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Fetch Datacenter name]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Add NFS storage domain]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Add glusterfs storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Add iSCSI storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Add Fibre Channel storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Get storage domain details]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Find the appliance OVF]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Parse OVF]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Get required size]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove unsuitable storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Check storage domain free space]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Activate storage domain]
[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Storage format is unsupported]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage format is unsupported]\". HTTP response code is 400."}
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]:

		  
In Hosted Engine VM in /var/log/ovirt-engine/engine.log the error logs are:
2019-05-28 19:25:13,357+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command AttachStorageomainVDS failed: Domain version `5` is unsupported by this version of VDSM: ''
2019-05-28 19:25:13,357+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandarameters:{storagePoolId='4669438a-3f99-11e9-aef4-00163e3912a0', ignoreFailoverLimit='false', storageDomainId='51ef1a02-271e-4e27-96e8-dfa1886917c7'})' execution failed: IRSGenericException: IRSErrorException: Domain version `5` is nsupported by this version of VDSM: ''
2019-05-28 19:25:13,358+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] FINISH, AttachStorageDomainVDSCommand, return: , log id: 580e7243
2019-05-28 19:25:13,358+02 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] Command 'org.ovirt.engine.core.bll.storage.domain.AttachStorageDomanToPoolCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Domain version `5` is unsupported by this version of VDSM: '' (Failed ith error UnsupportedDomainVersion and code 394)
2019-05-28 19:25:13,367+02 INFO  [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] Command [id=0e8d4f68-9e59-4944-a3d1-89b591ff811c]: Compensating NEW_ENTITY_ID of org.ovirt.engin.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='4669438a-3f99-11e9-aef4-00163e3912a0', storageId='51ef1a02-271e-4e27-96e8-dfa1886917c7'}.
2019-05-28 19:25:13,397+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attah Storage Domain hosted_storage to Data Center *******. (User: admin@internal-authz)
2019-05-28 19:25:13,407+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [cba462c2-1d73-4978-89fd-f6db40162601] Lock freed to object 'EngineLock:{exclusiveLocks='[51ef1a02-271e-4e7-96e8-dfa1886917c7=STORAGE]', sharedLocks=''}'
2019-05-28 19:25:13,407+02 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Storage format is unsupported]

Host which started installation have the newest available versions of packages:
# rpm -qa | grep vdsm-4
vdsm-4.30.13-1.el7.x86_64
# rpm -qa | grep ovirt-engine
ovirt-engine-appliance-4.3-20190416.1.el7.x86_64
python-ovirt-engine-sdk4-4.3.1-2.el7.x86_64
# rpm -qa | grep ovirt-hosted
ovirt-hosted-engine-ha-2.3.1-1.el7.noarch
ovirt-hosted-engine-setup-2.3.7-1.el7.noarch

I think this is reapperance of BUG https://bugzilla.redhat.com/show_bug.cgi?id=1669606
Please check code for regression.

I know this bug is closed but I have seen the same behavior on RHVH-4.4-20201117.0.
The workaround I have found was to use a different name for the hosted engine NFS share (hosted_storage).
After that, I could install and see 2 storage domains:
-hosted_storage as unattached
-engine_storage (name I picked up) as active