I found out that after reboot the storage controller, the loop0 - loop7 got problems, but loop8 - loop255 I think are ok.

10:41:45 ERROR [SystemUtil:Thread-16] com.eucalyptus.util.ExecutionException: //
/usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop5 error: loop: can't get inf
o on device /dev/loop5: No such device or address

10:41:45 ERROR [SystemUtil:Thread-16] com.eucalyptus.util.ExecutionException: //
/usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop2 error: loop: can't get inf
o on device /dev/loop2: No such device or address

10:41:45 ERROR [SystemUtil:Thread-16] com.eucalyptus.util.ExecutionException: //
/usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop0 error: loop: can't get inf
o on device /dev/loop0: No such device or address

10:41:45 ERROR [ISCSIManager:Thread-16] com.eucalyptus.util.EucalyptusCloudExcep
tion: tgtadm: invalid request

10:41:46 ERROR [ISCSIManager:Thread-16] com.eucalyptus.util.EucalyptusCloudExcep
tion: tgtadm: invalid request

Seems those old volumes originally mount with loop0-7 lost some files. Do I need to manually losetup -d loop devices everytime I need to restart?

Sorry. I'm using CEntOS 5.5 and install eucalyptus 2.0 by package. But the iSCSI is default value after I install, so I didn't change.

The storage is now okay, I deleted all volumes, and create new, they can be attached now. I think some part of the volumes were broken. But still, I lost data. I'll keep trying to figure this out.

Thanks for your reply.

Hi There

We are having exactly the same problem as described above with a clean install of Eucalyptus 2.0

Our setup is as follows:

A single cluster with

* Cloud Controller, SC, Walrus and CC on the same machine
* 4 Node controllers

Eucalyptus is fully functioning, and we can create, attach and snapshot volumes correctly. However if we reboot cleanly every machine in the cloud and bring it back up again cleanly, any Volumes which we created before the reboot cannot be attached to any instances. We are still able to create and attach new volumes and the old volumes do appear in HybridFox as volumes but any attempt to attach or snapshot them fails.

I have tested both with ISCSI enabled and disabled and this has no effect.

The errors we get back are very generic. On the Cloud Controller in cloud-error.log we see:

22:07:30 ERROR [QueuedEventCallback:New I/O client worker #2-26] Caught exception in asynchronous response handler.
com.eucalyptus.util.EucalyptusClusterException: [AttachVolumeResponseType attachedVolume=AttachedVolume null null null null null Fri Sep 03 22:07:30 BST 2010 correlationId=null userId=admin effectiveUserId=null _return=false statusMessage=ERROR]
at com.eucalyptus.cluster.callback.QueuedEventCallback.messageReceived(QueuedEventCallback.java:209)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:134)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:134)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:134)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:134)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
at org.jboss.n

And on the Node controller in nc.log we get:

[Thu Sep 2 22:08:19 2010][003979][EUCAINFO ] doAttachVolume() invoked (id=i-37EB065F vol=vol-5F4F0657 remote=//,192.168.123.125,iqn.2009-06.com.eucalyptus.oxilcloudservice:store8,R9NCWVLP7kQ8n27kus6uMeaqk4K+HHN6XX2JpjfFQ5ruEMu0wIlQPZEQxkR1vDVohzgJEXJU2CqxfnQ6CadpPIHT74iTU0POzhjPw9ICv1OIZBoe6Ubpvbl/IA7MTr8JDiYMsy842zatmbJBD5EYdPESM5yca8HzOk7UkQnPeylFUKZItBN2thP3g3gx+6QE8g9ub5W+dOdPgbTj82AO7o8YT7nwosxUv9t25pvUGgO3/ZPiH8maDkiq8yYjPqaNVxJXYYdrwns2JhWLlPsBbMrBQ5MyGsMp89VanbPhuRXgOyi5H+zWNjAnVfKn5qJCl09CBO/rvZuy7KUCzhcTsw== local=/dev/sdc)
[Thu Sep 2 22:08:19 2010][003979][EUCAINFO ] connect_iscsi_target invoked (dev_string=//,192.168.123.125,iqn.2009-06.com.eucalyptus.oxilcloudservice:store8,R9NCWVLP7kQ8n27kus6uMeaqk4K+HHN6XX2JpjfFQ5ruEMu0wIlQPZEQxkR1vDVohzgJEXJU2CqxfnQ6CadpPIHT74iTU0POzhjPw9ICv1OIZBoe6Ubpvbl/IA7MTr8JDiYMsy842zatmbJBD5EYdPESM5yca8HzOk7UkQnPeylFUKZItBN2thP3g3gx+6QE8g9ub5W+dOdPgbTj82AO7o8YT7nwosxUv9t25pvUGgO3/ZPiH8maDkiq8yYjPqaNVxJXYYdrwns2JhWLlPsBbMrBQ5MyGsMp89VanbPhuRXgOyi5H+zWNjAnVfKn5qJCl09CBO/rvZuy7KUCzhcTsw==)
[Thu Sep 2 22:08:19 2010][003979][EUCADEBUG ] system_output(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/connect_iscsitarget.pl //,192.168.123.125,iqn.2009-06.com.eucalyptus.oxilcloudservice:store8,R9NCWVLP7kQ8n27kus6uMeaqk4K+HHN6XX2JpjfFQ5ruEMu0wIlQPZEQxkR1vDVohzgJEXJU2CqxfnQ6CadpPIHT74iTU0POzhjPw9ICv1OIZBoe6Ubpvbl/IA7MTr8JDiYMsy842zatmbJBD5EYdPESM5yca8HzOk7UkQnPeylFUKZItBN2thP3g3gx+6QE8g9ub5W+dOdPgbTj82AO7o8YT7nwosxUv9t25pvUGgO3/ZPiH8maDkiq8yYjPqaNVxJXYYdrwns2JhWLlPsBbMrBQ5MyGsMp89VanbPhuRXgOyi5H+zWNjAnVfKn5qJCl09CBO/rvZuy7KUCzhcTsw==]
[Thu Sep 2 22:08:24 2010][003979][EUCAERROR ] ERROR: connect_iscsi_target failed
[Thu Sep 2 22:08:24 2010][003979][EUCAERROR ] AttachVolume(): failed to connect to iscsi target
[Thu Sep 2 22:08:24 2010][003979][EUCAERROR ] ERROR: doAttachVolume() failed error=1

Can anybody shed any light on this problem? Obviously it is a huge showstopper if we cannot retrieve all of our persistent data on a power outage or planned shutdown !!!!!

Thanks a lot

Mark

CEntOS 5.5
Eucalyptus 2.0.0 Community (Xen) via RPMs
One CLC/CC/SC
Four NC

In the cloud error log we get on startup

With ISCSI disabled:

11:31:22 ERROR [SystemUtil:main] java.io.IOException: Cannot run program "bttrack": java.io.IOException: error=13, Permission denied
11:31:24 ERROR [ISCSIManager:Thread-15] Unable to unbind tid: 8
11:31:26 ERROR [OverlayManager:Thread-15] com.eucalyptus.util.EucalyptusCloudException: Could not export AoE device /dev/vg-cSWzAw../lv-chAKhA.. StorageInfo.getStorageInfo().getStorageInterface(): eth0 pid: 5174 returnValue:
11:31:27 ERROR [OverlayManager:Thread-15] com.eucalyptus.util.EucalyptusCloudException: Could not export AoE device /dev/vg-GYqVEw../lv-t9gbXQ.. StorageInfo.getStorageInfo().getStorageInterface(): eth0 pid: 5215 returnValue:

and with ISCSI enabled:

11:33:45 ERROR [SystemUtil:main] java.io.IOException: Cannot run program "bttrack": java.io.IOException: error=13, Permission denied
11:33:47 ERROR [SystemUtil:Thread-21] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap tgtadm --lld iscsi --op show --mode target --tid 11 error: tgtadm: can't find the target

11:33:49 ERROR [ISCSIManager:Thread-21] com.eucalyptus.util.EucalyptusCloudException: tgtadm: invalid request

11:33:51 ERROR [ISCSIManager:Thread-21] com.eucalyptus.util.EucalyptusCloudException: tgtadm: invalid request

11:33:51 ERROR [ISCSIManager:Thread-21] com.eucalyptus.util.EucalyptusCloudException: tgtadm: invalid request

Dear Neil,

We have a Sun v40z as CLC, CC, SC and three Sun v20z as NCs. They are older machines (multiple single core Opterons thus the requirement to use Xen).

After configuring the initial 'cloud' following the steps in the Eucalyptus Administratrors Guide (2.0), we are using Hybridfox to make instances, block storages, etc., and then connect them.

If you attach the block storage to an instance ant then pull the plug (which did happen thus this particular thread) the block storage is orphaned. The instances, quite expectedly, have died and new instances cannot attach to the old storage.

We have done bare metal rebuilds to remove artifacts of our tinkering and have easily reproduced the issue by a) simply halting all the servers in turn CLC to last NC; b) rebooting just the CLC/CC/SC server (leaving the NCs on); and, c) pulling the plug at the wall (already mentioned).

Best wishes,
Philip

Dear Neil,

My colleague actually pulled the plug out of the CLC, CC, SC box. We later simulated it by cutting the power at the PDU for all of the boxes at once.

I have bought a VT capable box so I am going to try UEC now...

Thank you for trying,
Philip

Hi guys, I think that I am having the same problem here. I have a CLC-SC-CC-Walrus box and a NC box. They are configured for multi-clustering and the CLC machine has 2 NIC's. Both boxes were based on UEC 10.04 LTS, but I was having some stability problems concerning the SC.
So I decided to upgrade both boxes to a 10.10 Maverick and Eucalyptus 2.0, also to test how the upgrade goes. Unfortunately, the upgrade did not go so well. The NC went fine, but the CLC machine had serios IRQ management problems, which made the box extremely slow at startup. Only by restarting Eucalyptus (which could take 1 hour) then the machine became responsive again, including Eucalyptus itself. Really weird.
Yesterday I upgraded the kernel with the most recent release (must be from this week) though and it seems to have solved this IRQ problem at least. So now the box is responding properly again. I can start VI's again now, I can login to the instance, I can create new volumes and attach them.. but:

- Previous volumes cannot be attached. I get the following errors:

# tail -f /var/log/eucalyptus/cc.log | grep ERROR
[Thu Nov 25 15:18:32 2010][010580][EUCAERROR ] ERROR: AttachVolume returned an error
[Thu Nov 25 15:18:32 2010] ERROR: doAttachVolume() returned FAIL


# tail -f /var/log/eucalyptus/cloud-output.log | grep ERROR
15:18:32 DEBUG 112 VolumeAttachCallback | com.eucalyptus.util.EucalyptusClusterException: [AttachVolumeResponseType attachedVolume=AttachedVolume null null null null null Thu Nov 25 15:18:32 CET 2010 correlationId=null userId=testUser effectiveUserId=null _return=false statusMessage=ERROR]
15:18:32 ERROR 271 QueuedEventCallback | Caught exception in asynchronous response handler.


# tail -f /var/log/eucalyptus/nc.log | grep ERR
[Thu Nov 25 15:18:27 2010][002259][EUCAERROR ] ERROR: connect_iscsi_target failed
[Thu Nov 25 15:18:27 2010][002259][EUCAERROR ] AttachVolume(): failed to connect to iscsi target
[Thu Nov 25 15:18:27 2010][002259][EUCAERROR ] ERROR: doAttachVolume() failed error=1


- If I create a new volume, it gets created, but I get the following error:

# tail -f /var/log/eucalyptus/cloud-output.log | grep ERROR
14:42:17 ERROR 94 SystemUtil | com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap tgtadm --lld iscsi --op show --mode target --tid 53 error: tgtadm: can't find the target


- If I try to attach the above volume, then I don't get any errors at all, the volume gets attached according to the CC, but connectivity to the instance is completely disrupted. I cannot login anymore, unless I reboot the instance, but then I cannot see the attached volume. Hybridfox still show the volume as attached though and if I try to detach it, then I get the following errors:

# tail -f /var/log/eucalyptus/cc.log | grep ERROR
[Thu Nov 25 15:22:57 2010][012079][EUCAERROR ] ERROR: DetachVolume returned an error
[Thu Nov 25 15:22:57 2010] ERROR: doDetachVolume() returned FAIL


# tail -f /var/log/eucalyptus/cloud-output.log | grep ERROR
15:22:57 DEBUG 99 VolumeDetachCallback | com.eucalyptus.util.EucalyptusClusterException: [DetachVolumeResponseType detachedVolume=AttachedVolume null null null null null Thu Nov 25 15:22:57 CET 2010 correlationId=null userId=testUser effectiveUserId=null _return=false statusMessage=ERROR]
15:22:57 ERROR 271 QueuedEventCallback | Caught exception in asynchronous response handler.


# tail -f /var/log/eucalyptus/nc.log | grep ERR
[Thu Nov 25 15:21:53 2010][002259][EUCAERROR ] libvirt: internal error '//,PUB.LIC.ADD.RESS,iqn.2009-06.com.eucalyptus.Cluster-CC01:store37,eJlnlcel1kdY++GE8l9AkOQsRPyWAcbVLl2q4F28e27maX1P5VsldPCFDogD2j9vrnP/h4SKkYMhrDCPp9yhZpoqrE7SolZgiKSDiJJTPwW22rl95TjQ440Q6qXN9HxptqPoezTtahUaM+fKT87dmkP0nmLNUMCHHTeg5Jaye3xUzJNHS3Vm3c11BWV0iDgNj2kLkiP8ZjQwlnJ7W/7aU72FbxwTEPhQXKWr3jpwDksaB2950gvTF2JgY9UjvQF2d/wIXiDOUANIBFYi138I87XLUttf6afccwW42+KYsRANc1z870wk9HYPqBfNK2kk2bHA+RDNM5RN0sFovUf4YQ==' does not exist (code=1)
[Thu Nov 25 15:21:53 2010][002259][EUCAERROR ] virDomainAttachDevice() failed (err=-1) XML={driver name='phy'/}{source dev='//,PUB.LIC.ADD.RESS,iqn.2009-06.com.eucalyptus.Cluster-CC01:store37,eJlnlcel1kdY++GE8l9AkOQsRPyWAcbVLl2q4F28e27maX1P5VsldPCFDogD2j9vrnP/h4SKkYMhrDCPp9yhZpoqrE7SolZgiKSDiJJTPwW22rl95TjQ440Q6qXN9HxptqPoezTtahUaM+fKT87dmkP0nmLNUMCHHTeg5Jaye3xUzJNHS3Vm3c11BWV0iDgNj2kLkiP8ZjQwlnJ7W/7aU72FbxwTEPhQXKWr3jpwDksaB2950gvTF2JgY9UjvQF2d/wIXiDOUANIBFYi138I87XLUttf6afccwW42+KYsRANc1z870wk9HYPqBfNK2kk2bHA+RDNM5RN0sFovUf4YQ=='/}{target dev='vdb'/}
[Thu Nov 25 15:22:52 2010][002259][EUCAERROR ] libvirt: operation failed: disk vdb not found (code=9)
[Thu Nov 25 15:22:52 2010][002259][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML= {disk type='block'}
[Thu Nov 25 15:22:52 2010][002259][EUCAERROR ] ERROR: doDetachVolume() failed error=1


I don't know if the Maverick upgrade was supposed to work smoothly, but it didn't happen in my case. I hope the above will help to understand more about this important issue and shed some light on the AoE/SCSI world.
Looking forward to solve the problem for my self as well, but I suspect already that I will have to reinstall everything from scratch. Any advice would be highly appreciated.
Thank you very much for your help!

Best regards!
TritoLux

Sorry if I bump this post so early again. I am trying to disable iSCSI and use AoE, but it seems that whatever I do with the config, iSCSI just stays active.
I set DISABLE_ISCSI="Y" in CLC-SC-CC and on the NC, I also tried to disable all VIRTIO options and of course I tried to cleanrestart all controllers several times and rebooted all servers several times as well. When I try to attach a volume now, the CC tells me that everything is fine, no errors reported and the volumes get attached, both old and new volumes, but no new devices/voumes appear on the instance.
aoe-stats and aoe-discover on the node return empty results, but when I attach a volume, then the NC reports the following line:

# tail -f /var/log/syslog | grep iscsi
Dec 2 02:43:53 dC-CC01-NC01 iscsid: connection3:0 is operational now

It seems that iSCSI did not get disabled at all.
Am I missing anything? Any idea how to use pure AoE again?

Kind regards,
TritoLux

Thanks dseven for your answer. Yes I did try to create new volumes as well of course. But the volumes that get created now do not appear in the instance (I checked both sd* and vd*), even though I receive no error messages or warnings on the cc.log or nc.log. The only difference is that, after several clean restarting and server rebooting, I finally get no iscsi connection created (on the NC), but aoe daoesn't get created either, as aoe-stat is still empty. According to the CC, a new volume gets properly attached, but it seems that the NC doesn't know about it.
I still get this error on the CC though:

# tail -f /var/log/eucalyptus/cloud-error.log
11:30:22 ERROR [SystemUtil:pool-8-thread-1] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap tgtadm --lld iscsi --op show --mode target --tid 4 error: tgtadm: can't find the target

I also increased the number of loop devices to 255 just to be sure.
At the moment the situation that I am experiencing is the following:

iSCSI: seems to work fine, but the CC-SC box doesn't have to be rebooted, otherwise all existing volumes cannot be attached anymore.
AoE: existing volumes cannot get attached. New volumes get attached but they do not appear within the instance.

Basically, I can't use EBS at the moment.
Any further help would be appreciated.
Best regards,

TritoLux

TritoLux says...
I still get this error on the CC though:

# tail -f /var/log/eucalyptus/cloud-error.log
11:30:22 ERROR [SystemUtil:pool-8-thread-1] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap tgtadm --lld iscsi --op show --mode target --tid 4 error: tgtadm: can't find the target


That is not an error, from what I can tell. The SC checks if the TID is available before it attempts to grab/allocate it. The "error" is really the SC checking the status, but I agree it should be reported more "nicely."

Did you check nc.log for errors? If the NC thinks that the volume is attached (as indicated in nc.log), then the issue is in your image (do you have acpiphp loaded for instance?) Rebooting/turning services on and off is unlikely to help you get to the bottom of this...it will only cause more confusion :)

Neil, thanks for adding up to your comment. You can find more logs about my problem in my post from Thu, 11/25/2010 - 16:47. I am only using images from the UEC Store actually. What I noticed though is that if I start an instance then it never comes up correctly the first time, and I always have to reboot it before I can use it properly.
I have been installing clouds for 9 months by now, with different configs and different host OS. I always follow the instructions you provide us with step by step. But this is the first 2.0 cloud I install and I am having serious EBS problems. In what controller box should I check if acpiphp is loaded? I had to clean reload services due to config changes, which were necessary for testing. I hope you will help me to shed some light on my issue, please.
Regards, TritoLux

p.s. If you are referring to the NC log reported in my post from Thu, 12/02/2010 - 04:24, then that log shows that NC responds to an iSCSI request, while I had disabled iSCSI completely from all configs. However, if I restart the network interfaces, then I see that iSCSI is still active. After a couple of reboots of the node, then I see that the logs do not report any iSCSI message, but AoE is also not active as I said already above. That's why no volume gets ever attached. I would like to use EBS with either AoE os iSCSI, whichever works first and well enough for a production environment.

Hello there,

We set up a test environment on 2 64bits servers, with a simple CLC-CC-SC-Walrus and separate NC, in order to see if CentOS 5.5/Xen/Euca2.01 were behaving better than UEC/Kvm/Euca2.0 regarding this EBS issue. We left every config with default settings, apart from the network mode, as we tried both MANAGED and MANAGED-NOVLAN.
Unfortunately, the problem is still there, clearly reproduced, with pretty much the same error messages.
The only difference is that on CentOS I occasionally can attach previously created volumes, even after rebooting the machines, but all pre-existing partitions are gone anyway (together with their data of course). According to me, this is an easily reproducible problem affecting Eucalyptus since iSCSI was introduced. Something that should not be present on stable releases, considering that we could consistently reproduce this major bug on fresh Ubuntu, Debian and CentOS installations.
As this is a serious "show stopper" and it seems impossible to get back to AoE unless starting an installation from 1.6.2, I think that this is one of those problems that should be treated with the highest priority.
Any further comment from the Eucalyptus team would be surely appreciated.
Thank you very much for your time and attention.

Best regards,
TritoLux

Hi Neil,

thank you for your response. I am sorry if my posts were not clear enough or came across as complaints only.
My main objective was to report what I thought was a serious problem with as many details as possible, so that Eucalyptus can be improved even more.
I am happy that a bug regarding this iSCSI issue has been already opened and it is being addressed for the next release.

However, falling back to AoE is not always working, as you can clearly read from my post Sat, 12/04/2010 - 03:41, where I am only reporting error logs from our UEC systems after disabling iSCSI. I don't know what is causing them either. Maybe those logs will help you to better understand what is going on.
Without having neither iSCSI nor AoE at work, you may safely agree on the fact that not being able to use EBS at all can be a serious show stopper for most of the users, wouldn't you?

Fortunately, we just found out that falling back to AoE works fine on CentOS for some reason, as opposed to UEC or Debian for instance. We haven't tested extensively yet though. At least, on CentOS we can finally use EBS with Euca2.x for the first time without loosing any newly created AoE volumes after reboot, even though the original settings were based on iSCSI.
Still, we are getting thousands of weird warnings concerning AoE on the syslog, even though they are not preventing the SC to work fine so far. This is what we get on the NC (many times at any SC operation):

Dec 7 17:40:21 testCloud-NC01 kernel: BUG: warning at lib/kref.c:32/kref_get() (Not tainted)
Dec 7 17:40:21 testCloud-NC01 kernel:
Dec 7 17:40:21 testCloud-NC01 kernel: Call Trace:
Dec 7 17:40:21 testCloud-NC01 kernel: [] kref_get+0x38/0x3d
Dec 7 17:40:21 testCloud-NC01 kernel: [] kobject_get+0x12/0x17
Dec 7 17:40:21 testCloud-NC01 kernel: [] blk_get_queue+0x1f/0x26
Dec 7 17:40:21 testCloud-NC01 kernel: [] :blkbk:dispatch_rw_block_io+0x4db/0x5a2
Dec 7 17:40:21 testCloud-NC01 kernel: [] __next_cpu+0x19/0x28
Dec 7 17:40:21 testCloud-NC01 kernel: [] find_busiest_group+0x1db/0x44a
Dec 7 17:40:21 testCloud-NC01 kernel: [] dequeue_task+0x18/0x37
Dec 7 17:40:21 testCloud-NC01 kernel: [] deactivate_task+0x28/0x5f
Dec 7 17:40:21 testCloud-NC01 kernel: [] monotonic_clock+0x35/0x7b
Dec 7 17:40:21 testCloud-NC01 kernel: [] thread_return+0x6c/0x113
Dec 7 17:40:21 testCloud-NC01 kernel: [] kobject_cleanup+0x39/0x7e
Dec 7 17:40:21 testCloud-NC01 kernel: [] :blkbk:blkif_schedule+0x36e/0x456
Dec 7 17:40:21 testCloud-NC01 kernel: [] :blkbk:blkif_schedule+0x0/0x456
Dec 7 17:40:21 testCloud-NC01 kernel: [] keventd_create_kthread+0x0/0xc4
Dec 7 17:40:21 testCloud-NC01 kernel: [] kthread+0xfe/0x132
Dec 7 17:40:21 testCloud-NC01 kernel: [] child_rip+0xa/0x12
Dec 7 17:40:21 testCloud-NC01 kernel: [] keventd_create_kthread+0x0/0xc4
Dec 7 17:40:21 testCloud-NC01 kernel: [] kthread+0x0/0x132
Dec 7 17:40:21 testCloud-NC01 kernel: [] child_rip+0x0/0x12

And this is what I found on the net regarding this bug:

It seems this error is caused by the aoe module (v22) from linux kernel 2.6.18 in combination with 32 bit domU on 64 bit dom0. Using the aoe devices directly doesn't cause this error message. With the newest aoe module (v63) this error message is supposed to be gone.

Hope this helps and that the problem is clearer now.

Best regards,

TritoLux

Sorry TritoLux, but it does not appear that the fix is in 2.0.2

If you're building eucalyptus from source, this fix is very easy to apply, but if you're relying on pre-built binaries, I guess you'll have to wait a bit longer...

Hi everyone, I am having the exact same problem discussed in this thread but with Eucalyptus version 2.0.3, will this fix ever be applied to the main stream code for 2.0.x or do I need to reinstall compiling the source code and applying the patch myself? Since I am about to start creating my volumes, I would like to know whether I should go for the compiling option or the debian binaries downgrading to AoE.
My setup:
FC-CC-SC-CLC-WALRUS on a Debian Squeeze machine: Linux 2.6.32-5-amd64 #1 SMP Tue Jun 14 09:42:28 UTC 2011 x86_64 GNU/Linux
NC on a Debian Squeeze machine: Linux 2.6.32-5-amd64 #1 SMP Tue Jun 14 09:42:28 UTC 2011 x86_64 GNU/Linux

Using Eucalyptus Debian 5.0 image (http://open.eucalyptus.com/wiki/EucalyptusUserImageCreatorGuide).

Hi everyone,

Thanks Neil for your answer, I wasn't aware that a 3.0 version was on its way, that's great news!!!

I was about to build the app from source when I came across this link:
https://bugs.launchpad.net/eucalyptus/+bug/733067

Although this is something that needs to be done every time your SC server is restarted, it seems to me faster than reinstalling the application and a clean work around to the problem. I have being trying it for a while and seems to be working perfectly. By the way, I've added the "/run/lock" part as it also seems to happen every time I restart the server.

Delete all iSCSI targets

tgtadm --mode target --op delete --tid=

Activate lvm2 volumes

vgscan

vgchange -ay

Also, after rebooting the OS we will need to run this command:
mkdir /run/lock

Otherwise we will get this error on cloud-error.log
00:25:26 ERROR [SystemUtil:pool-8-thread-1] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap pvcreate /dev/loop6 error: /var/lock/lvm: mkdir failed: No such file or directory
File-based locking initialisation failed.

Restart SC service

/etc/init.d/eucalyptus-cloud restart