Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1218882 - liotarget: "fd_do_rw() write returned -22", iSCSI not functioning
Summary: liotarget: "fd_do_rw() write returned -22", iSCSI not functioning
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-06 06:44 UTC by Tomasz Torcz
Modified: 2016-05-17 07:23 UTC (History)
10 users (show)

Fixed In Version: kernel-4.0.5-300.fc22
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-20 23:58:34 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
saveconfig.json (3.57 KB, text/plain)
2015-05-06 06:47 UTC, Tomasz Torcz
no flags Details

Description Tomasz Torcz 2015-05-06 06:44:15 UTC
Description of problem:
Since some time iSCSI target is not functioning. After some initial I/O, target throws "fd_do_rw() write returned -22" and initiator throws IO error.
Detailed configuration steps follow.

TARGET (iqn.2003-01.org.linux-iscsi.seurat.x8664, 192.168.22.1) :
Rootfs is btrfs (rw,relatime,seclabel,compress=zlib,ssd,space_cache,autodefrag)
1. truncate -s 800M /iscsi.bs0
2. targetcli:
  cd backstores/fileio
  create file_or_dev=/iscsi.bs0 name=fbs0
  cd /iscsi
  create
  cd iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7/tpg1 \
  cd luns
  create /backstores/fileio/fbs0
  cd /iscsi/iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7/tpg1/acls
  create iqn.2003-01.org.linux-iscsi.ginjo.x8664:sn.37648a3d9cbe

  ls /
o- / ............................................................................ [...]
  o- backstores ................................................................. [...]
  | o- block ..................................................... [Storage Objects: 0]
  | o- fileio .................................................... [Storage Objects: 1]
  | | o- fbs0 ............................ [/iscsi.bs0 (800.0MiB) write-back activated]
  | o- pscsi ..................................................... [Storage Objects: 0]
  | o- ramdisk ................................................... [Storage Objects: 0]
  | o- user ...................................................... [Storage Objects: 0]
  o- iscsi ............................................................... [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7 ............. [TPGs: 1]
  |   o- tpg1 .................................................. [no-gen-acls, no-auth]
  |     o- acls ............................................................. [ACLs: 1]
  |     | o- iqn.2003-01.org.linux-iscsi.ginjo.x8664:sn.37648a3d9cbe . [Mapped LUNs: 1]
  |     |   o- mapped_lun0 .................................... [lun0 fileio/fbs0 (rw)]
  |     o- luns ............................................................. [LUNs: 1]
  |     | o- lun0 .......................................... [fileio/fbs0 (/iscsi.bs0)]
  |     o- portals ....................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ........................................................ [OK]




INITIATOR (iqn.2003-01.org.linux-iscsi.ginjo.x8664:sn.37648a3d9cbe, 192.168.22.3) :
1. iscsiadm -m discovery -t st -p 192.168.22.1
   192.168.22.1:3260,1 iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7
2. iscsiadm -m node -l
[May 6 08:05] scsi host4: iSCSI Initiator over TCP/IP
[  +0.041362] scsi 4:0:0:0: Direct-Access     LIO-ORG  fbs0             4.0  PQ: 0 ANSI: 5
[  +0.028042] sd 4:0:0:0: Attached scsi generic sg2 type 0
[  +0.014333] sd 4:0:0:0: [sdb] 1638400 512-byte logical blocks: (838 MB/800 MiB)
[  +0.010611] sd 4:0:0:0: [sdb] Write Protect is off
[  +0.003731] sd 4:0:0:0: [sdb] Mode Sense: 43 00 10 08
[  +0.013365] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.083011] sd 4:0:0:0: [sdb] Attached SCSI disk

3. mkfs.btrfs -M -L seurat-iscsi /dev/sdb
4. btrfstune -x /dev/sdb

5. mount /dev/sdb /mnt/tmp -o autodefrag,compress=lzo
[May 6 08:06] BTRFS: device label seurat-iscsi devid 1 transid 3 /dev/sdb
[ +22.736061] BTRFS info (device sdb): disk space caching is enabled
[  +0.003400] BTRFS: has skinny extents
[  +0.010614] BTRFS: creating UUID tree
[May 6 08:07] BTRFS info (device sdb): enabling auto defrag
[  +0.005219] BTRFS info (device sdb): setting 8 feature flag
[  +0.003252] BTRFS info (device sdb): disk space caching is enabled
(there are two mounts in the log snippet above)


NOW START THE I/O. Soon:

INITIATOR:
[May 6 08:08] fd_do_rw() write returned -22
[  +0.107407] fd_do_rw() write returned -22


TARGET:
[May 6 08:08] sd 4:0:0:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  +0.004159] sd 4:0:0:0: [sdb] Sense Key : Not Ready [current] 
[  +0.001032] sd 4:0:0:0: [sdb] Add. Sense: Logical unit communication failure
[  +0.003389] sd 4:0:0:0: [sdb] CDB: Write(10) 2a 00 00 01 6b 20 00 2b e0 00
[  +0.003223] blk_update_request: I/O error, dev sdb, sector 92960
[  +0.001409] BTRFS: bdev /dev/sdb errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[  +0.002002] BTRFS: bdev /dev/sdb errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[  +0.000759] BTRFS: bdev /dev/sdb errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[  +0.001236] BTRFS: bdev /dev/sdb errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
[  +0.000801] BTRFS: bdev /dev/sdb errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
[  +0.000644] BTRFS: bdev /dev/sdb errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
[  +0.088333] sd 4:0:0:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  +0.000003] sd 4:0:0:0: [sdb] Sense Key : Not Ready [current] 
[  +0.000003] sd 4:0:0:0: [sdb] Add. Sense: Logical unit communication failure
[  +0.000003] sd 4:0:0:0: [sdb] CDB: Write(10) 2a 00 00 00 a0 00 00 3b 20 00
[  +0.000003] blk_update_request: I/O error, dev sdb, sector 40960
[  +0.000006] BTRFS: bdev /dev/sdb errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[  +0.000195] BTRFS: bdev /dev/sdb errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[  +0.000085] BTRFS: bdev /dev/sdb errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
[  +0.000082] BTRFS: bdev /dev/sdb errs: wr 10, rd 0, flush 0, corrupt 0, gen 0

Application exits with I/O errors, btrfs sometimes switches to read-only.



Version-Release number of selected component (if applicable):
TARGET:
kernel-core-4.0.1-300.fc22.x86_64
targetcli-2.1.fb40-1.fc21.noarch

INITIATOR:
kernel-core-4.1.0-0.rc1.git1.1.fc23.x86_64
iscsi-initiator-utils-6.2.0.873-25.gitc9d830b.fc22.x86_64

How reproducible:
I've reproduced it twice on different hosts/VMs. In both cases initiator/target are directly connected: a) target on physical host, initiator is a VM hosted on this host - KVM is used; b) target and initiator are VMs hosted on the same machine, VirtualBox is used.

Comment 1 Tomasz Torcz 2015-05-06 06:47:10 UTC
Created attachment 1022488 [details]
saveconfig.json

Comment 2 Oleg Drokin 2015-06-01 22:08:33 UTC
I think I met the same problem.

So,e digging around and I decided to give a try to this patch: https://git.kernel.org/cgit/linux/kernel/git/gregkh/staging.git/commit/?h=staging-testing&id=6b9a44d0939ca6235c72b811bd55b462d6a0a553 

With this in-place my problems seems to be all gone now, or so it seems.

If you wnat to give it a try, but don't want to build the RPM yourself, here is the one I built: http://crimea.edu/~green/kernel/

Comment 3 Oleg Drokin 2015-06-01 22:14:32 UTC
I just want to add that the patch is to the target, not to initiator. so the patched rpm is that of the patched current F22 kernel.

Comment 4 Fedora Update System 2015-06-09 12:57:28 UTC
kernel-4.0.5-300.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/kernel-4.0.5-300.fc22

Comment 5 Fedora Update System 2015-06-09 12:59:32 UTC
kernel-4.0.5-200.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/kernel-4.0.5-200.fc21

Comment 6 Fedora Update System 2015-06-10 19:09:05 UTC
Package kernel-4.0.5-200.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-4.0.5-200.fc21'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-9704/kernel-4.0.5-200.fc21
then log in and leave karma (feedback).

Comment 7 Fedora Update System 2015-06-20 23:58:34 UTC
kernel-4.0.5-200.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 8 Fedora Update System 2015-06-21 00:31:28 UTC
kernel-4.0.5-300.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.