Eucalyptus 1.6.1
Hypervisor: KVM
Separate nodes running clc, cc, walrus and 1 Node.
Networking mode is MANAGED
---------------------------
I created a new VM using ubuntu-vm-builder and uploaded it by this script:
mkdir kernel
euca-bundle-image -i /boot/vmlinuz-2.6.31-20-generic -d ./kernel --kernel true
euca-upload-bundle -b kernel1 -m ./kernel/vmlinuz-2.6.31-20-generic.manifest.xml
EKI=`euca-register kernel1/vmlinuz-2.6.31-20-generic.manifest.xml | awk '{print $2}'`
echo $EKI
mkdir ramdisk
euca-bundle-image -i /boot/initrd.img-2.6.31-20-generic -d ./ramdisk --ramdisk true
euca-upload-bundle -b ramdisk1 -m ramdisk/initrd.img-2.6.31-20-generic.manifest.xml
ERI=`euca-register ramdisk1/initrd.img-2.6.31-20-generic.manifest.xml | awk '{print $2}'`
echo $ERI
mkdir image
euca-bundle-image -i ./ubuntu-kvm/disk0.qcow2 -d ./image --kernel $EKI --ramdisk $ERI
euca-upload-bundle -b image1 -m ./image/disk0.qcow2.manifest.xml
EMI=`euca-register image1/disk0.qcow2.manifest.xml | awk '{print $2}'`
----------------
When i launch an instance with this emi, it remains in pending state forever.
nc.log :
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] StartNetwork(): SUCCESS return from vnetStartNetwork 0
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] StartNetwork(): done
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] doRunInstance() invoked (id=i-40800793 cores=2 disk=10 memory=512)
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] image=emi-57F30E05 at http://128.178.52.58:8773/services/Walrus/image1/disk0.qcow2.manifest.xml
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] krnel=eki-51461288 at http://128.178.52.58:8773/services/Walrus/kernel1/vmlinuz-2.6.31-20-gene...
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] rmdsk=eri-A4A513D9 at http://128.178.52.58:8773/services/Walrus/ramdisk1/initrd.img-2.6.31-20-...
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] vlan=10 priMAC=d0:0d:40:80:07:93 pubMAC=d0:0d:40:80:07:93
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] network started for instance i-40800793
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] retrieving images for instance i-40800793 (disk limit=10240MB)...
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances/admin/i-40800793/kernel-digest
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] from http://128.178.52.58:8773/services/Walrus/kernel1/vmlinuz-2.6.31-20-gene...
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): writing GET output to /var/lib/eucalyptus/instances/admin/i-40800793/kernel-digest
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): wrote 3456 bytes in 1 writes
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances/admin/i-40800793/kernel-digest
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] downloding image into /var/lib/eucalyptus/instances/admin/i-40800793/kernel...
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances/admin/i-40800793/kernel
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] from http://128.178.52.58:8773/services/Walrus/kernel1/vmlinuz-2.6.31-20-gene...
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): writing GET/GetDecryptedImage output to /var/lib/eucalyptus/instances/admin/i-40800793/kernel
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): wrote 3947008 bytes in 313 writes
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances/admin/i-40800793/kernel
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/admin/i-40800793/kernel /var/lib/eucalyptus/instances/eucalyptus/cache/eki-51461288/kernel]
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/admin/i-40800793/kernel-digest /var/lib/eucalyptus/instances/eucalyptus/cache/eki-51461288/kernel-digest]
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk-digest
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] from http://128.178.52.58:8773/services/Walrus/ramdisk1/initrd.img-2.6.31-20-...
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): writing GET output to /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk-digest
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): wrote 3464 bytes in 1 writes
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk-digest
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] downloding image into /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk...
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk
[Thu Apr 15 01:13:54 2010][032083][EUCAINFO ] from http://128.178.52.58:8773/services/Walrus/ramdisk1/initrd.img-2.6.31-20-...
[Thu Apr 15 01:13:54 2010][032083][EUCADEBUG ] walrus_request(): writing GET/GetDecryptedImage output to /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk
[Thu Apr 15 01:13:55 2010][032083][EUCADEBUG ] walrus_request(): wrote 7426233 bytes in 611 writes
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk /var/lib/eucalyptus/instances/eucalyptus/cache/eri-A4A513D9/ramdisk]
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/admin/i-40800793/ramdisk-digest /var/lib/eucalyptus/instances/eucalyptus/cache/eri-A4A513D9/ramdisk-digest]
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances/admin/i-40800793/disk-digest
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] from http://128.178.52.58:8773/services/Walrus/image1/disk0.qcow2.manifest.xml
[Thu Apr 15 01:13:55 2010][032083][EUCADEBUG ] walrus_request(): writing GET output to /var/lib/eucalyptus/instances/admin/i-40800793/disk-digest
[Thu Apr 15 01:13:55 2010][032083][EUCADEBUG ] walrus_request(): wrote 4840 bytes in 1 writes
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances/admin/i-40800793/disk-digest
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] downloding image into /var/lib/eucalyptus/instances/admin/i-40800793/disk...
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances/admin/i-40800793/disk
[Thu Apr 15 01:13:55 2010][032083][EUCAINFO ] from http://128.178.52.58:8773/services/Walrus/image1/disk0.qcow2.manifest.xml
[Thu Apr 15 01:13:55 2010][032083][EUCADEBUG ] walrus_request(): writing GET/GetDecryptedImage output to /var/lib/eucalyptus/instances/admin/i-40800793/disk
[Thu Apr 15 01:13:57 2010][032083][EUCADEBUG ] doDescribeInstances() invoked
[Thu Apr 15 01:13:57 2010][032083][EUCADEBUG ] doDescribeResource() invoked
[Thu Apr 15 01:13:58 2010][032083][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-40800793' (code=42)
And then periodically a Walrus time out message in nc.log:
[Thu Apr 15 01:17:55 2010][032083][EUCADEBUG ] walrus_request(): wrote 30 bytes in 1 writes
[Thu Apr 15 01:17:55 2010][032083][EUCAWARN ] walrus_request(): server responded with HTTP code 408 (timeout), retrying
------------------------------
Relevant portion of cloud-error.log:
01:03:38 [DefaultServiceExceptionStrategy:New I/O server worker #1-23] ERROR
********************************************************************************
Message : Component that caused exception is: BukkitInternal. Message payload is of type: CacheImageType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
783ddffb-1978-4f42-a7c0-095e3ded60a5
admin
image1
disk0.qcow2.manifest.xml
JavaDoc : http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceExce...
********************************************************************************
Exception stack is:
1. Fail (edu.ucsb.eucalyptus.cloud.DecryptionFailedException)
edu.ucsb.eucalyptus.cloud.ws.WalrusImageManager:261 (null)
2. Component that caused exception is: BukkitInternal. Message payload is of type: CacheImageType (org.mule.api.service.ServiceException)
org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceExce...)
----------------------------------------
As on can see, in nc.log, the image retrieval can't proceed after "walrus_request(): writing GET/GetDecryptedImage output to /var/lib/eucalyptus/instances/admin/i-40800793/disk" and cloud-error.log shows an exception about decryption too.
I have read a lots of related topics on this forum and haven't been able to fix it. Please guide me, I need to launch a custom image and I am struck from 3 days on this.
Thanks.
Hi,
I have installed on fresh fedora 12 systems. Eucalyptus 1.6.2. I am using centos 32 bit image provided on eucalyptus website. When I try to run the instance I see following in cloud-error.log on walrus:
12:08:33 [DefaultServiceExceptionStrategy:connector.VM.0.dispatcher.7] ERROR
********************************************************************************
Message : Component that caused exception is: BukkitInternal. Message payload is of type: CacheImageType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
00c12b6e-51ca-4b55-9560-970e4aea474f
admin
S3-bucket-cfs
centos.5-3.x86.img.manifest.xml
JavaDoc : http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceExce...
********************************************************************************
Exception stack is:
1. Fail (edu.ucsb.eucalyptus.cloud.DecryptionFailedException)
edu.ucsb.eucalyptus.cloud.ws.WalrusImageManager:262 (null)
2. Component that caused exception is: BukkitInternal. Message payload is of type: CacheImageType (org.mule.api.service.ServiceException)
org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apidocs/org/mule/api/service/ServiceExce...)
********************************************************************************
Root Exception stack trace:
edu.ucsb.eucalyptus.cloud.DecryptionFailedException: Fail
at edu.ucsb.eucalyptus.cloud.ws.WalrusImageManager.decryptImage(WalrusImageManager.java:262)
at edu.ucsb.eucalyptus.cloud.ws.WalrusImageManager.cacheImage(WalrusImageManager.java:435)
at edu.ucsb.eucalyptus.cloud.ws.WalrusImageManager.cacheImage(WalrusImageManager.java:1048)
at edu.ucsb.eucalyptus.cloud.ws.WalrusControl.CacheImage(WalrusControl.java:380)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolver.invokeMethod(AbstractEntryPointResolver.java:147)
at org.mule.model.resolvers.ReflectionEntryPointResolver.invoke(ReflectionEntryPointResolver.java:127)
at org.mule.model.resolvers.DefaultEntryPointResolverSet.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.intercept(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeComponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall(AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(AbstractComponent.java:87)
at org.mule.model.seda.SedaService.doSend(SedaService.java:234)
at org.mule.service.AbstractService.sendEvent(AbstractService.java:510)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleSession.java:351)
at org.mule.routing.inbound.DefaultInboundRouterCollection.send(DefaultInboundRouterCollection.java:196)
at org.mule.routing.inbound.DefaultInboundRouterCollection.route(DefaultInboundRouterCollection.java:164)
at org.mule.transport.AbstractMessageReceiver$DefaultInternalMessageListener.onMessage(AbstractMessageReceiver.java:604)
at org.mule.transport.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:346)
at org.mule.transport.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:269)
at org.mule.transport.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:262)
at org.mule.transport.vm.VMMessageReceiver.onMessage(VMMessageReceiver.java:98)
at org.mule.transport.vm.VMMessageDispatcher.doDispatch(VMMessageDispatcher.java:66)
at org.mule.transport.AbstractMessageDispatcher$Worker.run(AbstractMessageDispatcher.java:262)
at org.mule.work.WorkerContext.run(WorkerContext.java:310)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:636)
********************************************************************************
On the node's end the walrus times out and the VM remains in pending state only.
Regards
Deependra
Hello,
did you register walrus? What does euca_conf --list-walruses returns? Did you register all other components correctly? Do you have some firewall? Can the NC connect to the walrus machine? What IP did you use for walrus (you can check/change it from the webUI)?
cheers
graziano
Hi,
Yes. I registered every component as mentioned in the setup guide.
euca_conf --register-walrus 10.122.114.164
euca_conf --list-walruses
10.122.114.164
Similarly the same host is running 10.122.114.164 Cloud and SC too.
No there is no firewall it has been disabled. I can see the IP 10.122.114.164 in webUI for eucalyptus too. But I didn't changed it.
When I run the instance at the node end I can see the node is able to download ramdisk and kernel file successfully but when it comes to filesystem image it tries for like a hour with 10 fail tries and downloads only 30bytes.
The error about failing to decrypt comes very early though and then it keeps trying only.
Regards
Deependra
Hi,
Please check if you have enough space to download the image in the node (instance location - /var/eucalyptus/instances). Sometimes the error to download may be because of the non-availability of space in your node.
Regards,
Shivanand B
Hi,
I have 33GB in /usr/local/eucalyptus which is the instance storage location for me. At the Walrus I have 101GB of free space available.
Am I affected by this bug ?
https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/532682
Regards
Deependra
Hi,
Did a complete new installation on a fresh setup of 2 machines with fedora 12 and this patch applied to the sources of 1.6.2
http://bazaar.launchpad.net/~eucalyptus-maintainers/eucalyptus/1.6.2/rev...
Doesn't help I still face the same problem
Regards
Deependra
Hello,
what is the specs of the hardware on which walrus is running? What's the machine load when you start the instance? Walrus needs to prep the image before handing it over to the NC, and the timeout seems to imply that the machine went out for lunch at that time: but we tested internally on some very slow machine, and we don't have problem. Anything on the system logs to imply some sort of error? Where is the bucket repository residing? On a local disk?
cheers
graziano
Hi,
Its been long since I have not been able to solve this issue. I don't know why eucalyptus 1.6.2 refuse to work at this point with fedora.
This setup now is fresh fedora 13 64bit machines where I have compiled and installed eucalyptus 1.6.2 from source code. The machines are having AMD processors with hardware VT and each having 2GB RAM
I am talking about 2 machines here both node controllers as well one of them acting as cluster controller/cloud controller and storage controller (walrus).
When I try to run a instance the walrus keeps timing out and finally it dies. The images are stored in a local disk of one of these 2 machines. Bucket repository is residing on one of the machine's local hard drive.
Have you guys fixed anything related to this in bzr which we don't see in 1.6.2 ?
I tried putting bzr latest revision last night on my laptop but I was not able to get my nodes registered with the cloud controller. Not sure why it is working.
The setup in this context is at the work place.
Thanks
Deependra
Hi,
My host kernels were affected by the following bug:
https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/588861
which I manually patched and rebuild according to the patch mentioned in the above bug details and it works.
I hope to see all the working fedora kernels equipped with this patch. Currently I have found that this patch has made it to upstream in 2.6.35. So I am not sure when it will be stabilized enough to be backported to fedora 12 and 13 kernels but till then eucalyptus setups may not function for people running these distributions.
Thanks
Deependra
Hi,
I didn't tried the downgrade option as that would have required putting a fedora 12 kernel on fedora 13. I applied the patch to my current kernel and built a custom kernel.
Also I am not sure about the exact kernel versions for fedora which have this problem but I have been struggling since kernels from March this year. The problem started in fedora 12 and is still there in fedora 13.
The current kernel in fedora where I see this problem is:
2.6.33.6-147
And some 2.6.31 kernel in fedora 12 (don't remember exactly the version).
I hope 2.6.35 soons get released and is pushed for fedora 13! Maybe the patch can be backported to older kernels and maintained there. Not sure if fedora / kernel developers agree on the same :)
Thanks
Deependra