Join us at engage.eucalyptus.com
Eucalyptus version 1.6.1
Distribution and architecture - Ubuntu 9.10 x64
Hypervisor type - kvm
Topology - 1 front-end, 3 nodes
Networking mode - Managed-NOVLAN
I can start the vm, ssh to it without problem but when I reboot using euca-reboot-instances, it shuts down and then terminates. Here's the nc.log:
[Sun Feb 7 16:26:21 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:26:27 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:26:27 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:26:32 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:26:32 2010][001678][EUCADEBUG ] doStartNetwork() invoked
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] StartNetwork(): SUCCESS return from vnetStartNetwork 0
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] StartNetwork(): done
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] doRunInstance() invoked (id=i-347A06BD cores=1 disk=2 memory=128)
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] image=emi-385E1610 at http://192.100.1.10:8773/services/Walrus/image-ubuntu-bucket/ubuntu.9-04...
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] krnel=eki-AE0417F4 at http://192.100.1.10:8773/services/Walrus/kernel-ubuntu-bucket/vmlinuz-2....
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] rmdsk=eri-1672192E at http://192.100.1.10:8773/services/Walrus/ramdisk-ubuntu-bucket/initrd.im...
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] vlan=10 priMAC=d0:0d:34:7A:06:BD pubMAC=d0:0d:34:7A:06:BD
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] network started for instance i-347A06BD
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] retrieving images for instance i-347A06BD (disk limit=2048MB)...
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] verifying cached file in /var/lib/eucalyptus/instances/eucalyptus/cache/eki-AE0417F4/kernel...
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] walrus_request(): downloading /tmp/walrus-digest-MGGeyX
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] from http://192.100.1.10:8773/services/Walrus/kernel-ubuntu-bucket/vmlinuz-2....
[Sun Feb 7 16:26:32 2010][001678][EUCADEBUG ] walrus_request(): writing GET output to /tmp/walrus-digest-MGGeyX
[Sun Feb 7 16:26:32 2010][001678][EUCADEBUG ] walrus_request(): wrote 3456 bytes in 1 writes
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] walrus_request(): saved image in /tmp/walrus-digest-MGGeyX
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/eucalyptus/cache/eki-AE0417F4/kernel /var/lib/eucalyptus/instances/user1/i-347A06BD/kernel]
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] verifying cached file in /var/lib/eucalyptus/instances/eucalyptus/cache/eri-1672192E/ramdisk...
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] walrus_request(): downloading /tmp/walrus-digest-jgmASE
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] from http://192.100.1.10:8773/services/Walrus/ramdisk-ubuntu-bucket/initrd.im...
[Sun Feb 7 16:26:32 2010][001678][EUCADEBUG ] walrus_request(): writing GET output to /tmp/walrus-digest-jgmASE
[Sun Feb 7 16:26:32 2010][001678][EUCADEBUG ] walrus_request(): wrote 3464 bytes in 1 writes
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] walrus_request(): saved image in /tmp/walrus-digest-jgmASE
[Sun Feb 7 16:26:32 2010][001678][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/eucalyptus/cache/eri-1672192E/ramdisk /var/lib/eucalyptus/instances/user1/i-347A06BD/ramdisk]
[Sun Feb 7 16:26:33 2010][001678][EUCAINFO ] verifying cached file in /var/lib/eucalyptus/instances/eucalyptus/cache/emi-385E1610/disk...
[Sun Feb 7 16:26:33 2010][001678][EUCAINFO ] walrus_request(): downloading /tmp/walrus-digest-8B9rGm
[Sun Feb 7 16:26:33 2010][001678][EUCAINFO ] from http://192.100.1.10:8773/services/Walrus/image-ubuntu-bucket/ubuntu.9-04...
[Sun Feb 7 16:26:33 2010][001678][EUCADEBUG ] walrus_request(): writing GET output to /tmp/walrus-digest-8B9rGm
[Sun Feb 7 16:26:33 2010][001678][EUCADEBUG ] walrus_request(): wrote 5420 bytes in 1 writes
[Sun Feb 7 16:26:33 2010][001678][EUCAINFO ] walrus_request(): saved image in /tmp/walrus-digest-8B9rGm
[Sun Feb 7 16:26:33 2010][001678][EUCAINFO ] walrus_request(): saved image in /tmp/walrus-digest-8B9rGm
[Sun Feb 7 16:26:33 2010][001678][EUCAINFO ] vrun(): [cp -a /var/lib/eucalyptus/instances/eucalyptus/cache/emi-385E1610/disk /var/lib/eucalyptus/instances/user1/i-347A06BD/disk]
[Sun Feb 7 16:26:33 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:26:33 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:26:34 2010][001678][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-347A06BD' (code=42)
[Sun Feb 7 16:26:39 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:26:39 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:26:39 2010][001678][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-347A06BD' (code=42)
[Sun Feb 7 16:26:39 2010][001678][EUCAINFO ] vrun(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/partition2disk /var/lib/eucalyptus/instances/user1/i-347A06BD/disk 512 499]
[Sun Feb 7 16:26:44 2010][001678][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-347A06BD' (code=42)
[Sun Feb 7 16:26:46 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:26:46 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:26:49 2010][001678][EUCAINFO ] preparing images for instance i-347A06BD...
[Sun Feb 7 16:26:49 2010][001678][EUCAINFO ] adding key/tmp/sckey.PjsgIH to the root file system at /var/lib/eucalyptus/instances/user1/i-347A06BD/disk using (//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/add_key.pl //usr/lib/eucalyptus/euca_mountwrap)
[Sun Feb 7 16:26:49 2010][001678][EUCAINFO ] vrun(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/add_key.pl //usr/lib/eucalyptus/euca_mountwrap 32256 /var/lib/eucalyptus/instances/user1/i-347A06BD/disk /tmp/sckey.PjsgIH]
[Sun Feb 7 16:26:49 2010][001678][EUCADEBUG ] system_output(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/gen_kvm_libvirt_xml --ramdisk --ephemeral]
[Sun Feb 7 16:26:50 2010][001678][EUCAINFO ] currently running/booting: i-347A06BD
[Sun Feb 7 16:26:50 2010][001678][EUCAINFO ] started VM instance i-347A06BD
[Sun Feb 7 16:27:57 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:03 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:03 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:09 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:09 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:09 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:15 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:15 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:21 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:21 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:27 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:27 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:33 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:33 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:39 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:39 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:45 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:45 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:51 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:51 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:57 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:28:57 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:28:59 2010][001678][EUCAINFO ] doRebootInstance() invoked (id=i-347A06BD)
[Sun Feb 7 16:28:59 2010][001678][EUCAERROR ] libvirt: Failure while reading monitor startup output: Connection reset by peer (code=38)
[Sun Feb 7 16:28:59 2010][001678][EUCAFATAL ] Failed to restart instance i-347A06BD
[Sun Feb 7 16:29:01 2010][001678][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-347A06BD' (code=42)
[Sun Feb 7 16:29:01 2010][001678][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances/user1/i-347A06BD/]
[Sun Feb 7 16:29:01 2010][001678][EUCAINFO ] stopping the network (vlan=10)
[Sun Feb 7 16:29:03 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:29:03 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:29:09 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:29:09 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
[Sun Feb 7 16:29:09 2010][001678][EUCAINFO ] doTerminateInstance() invoked (id=i-347A06BD)
[Sun Feb 7 16:29:09 2010][001678][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-347A06BD' (code=42)
[Sun Feb 7 16:29:09 2010][001678][EUCAWARN ] warning: domain i-347A06BD to be terminated not running on hypervisor
[Sun Feb 7 16:29:09 2010][001678][EUCADEBUG ] doDescribeResource() invoked
[Sun Feb 7 16:29:11 2010][001678][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-347A06BD' (code=42)
[Sun Feb 7 16:29:11 2010][001678][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances/user1/i-347A06BD/]
[Sun Feb 7 16:29:11 2010][001678][EUCAINFO ] stopping the network (vlan=10)
[Sun Feb 7 16:29:15 2010][001678][EUCADEBUG ] doDescribeInstances() invoked
I saw this recommendation to modify /etc/libvirt/qemu.conf and set:
security_driver = "none"
I restarted all the 3 nodes after setting security_driver = "none" but that didn't help.
Ay help would be greatly appreciated.
Thanks,
spcla09
[
Hello,
this is a weird problem indeed. Could you check the libvirt logs and the system logs for anything which may explain this behavior? And if you modify MANUAL_INSTANCES_CLEANUP=1 on the NC, restart it and create again the problem (you may want to disable manual cleanup after that), there may be a console log in the instance directory: perhaps there is some clue on what happened to the instance from its kernel output.
cheers
graziano
Thanks Graziano for your response.
I checked the console.log file and it doesn't generate any new log entries when I execute the euca-reboot-instances. I haven't seen any useful info related to my problem from the console log.
Here's the libvirt log:
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin /usr/bin/kvm -S -M pc-0.11 -m 128 -smp 1 -name i-3EF506A2 -uuid b307cc95-4e82-8b2d-2ee5-0492cd1556c8 -nographic -monitor unix:/var/run/libvirt/qemu/i-3EF506A2.monitor,server,nowait -boot c -kernel /var/lib/eucalyptus/instances/user2/i-3EF506A2/kernel -initrd /var/lib/eucalyptus/instances/user2/i-3EF506A2/ramdisk -append root=/dev/sda1 console=ttyS0 -drive file=/var/lib/eucalyptus/instances/user2/i-3EF506A2/disk,if=scsi,index=0,boot=on -net nic,macaddr=d0:0d:3e:f5:06:a2,vlan=0,model=e1000,name=e1000.0 -net tap,fd=22,vlan=0,name=tap.0 -serial file:/var/lib/eucalyptus/instances/user2/i-3EF506A2/console.log -parallel none -usb
Below is the only error message generated in the syslog while I was rebooting an instance using the euca-reboot-instances.
Feb 10 20:47:29 euca-node3 libvirtd: 20:47:29.791: error : qemudDomainLookupByName:2677 : Domain not found: no domain with matching name 'i-4AC00871'
Feb 10 20:47:38 euca-node3 libvirtd: 20:47:38.083: error : qemudDomainLookupByName:2677 : Domain not found: no domain with matching name 'i-4AC00871'
Feb 10 20:47:40 euca-node3 libvirtd: 20:47:40.029: error : qemudDomainLookupByName:2677 : Domain not found: no domain with matching name 'i-4AC00871'
I tried logging in to the instance and reboot from there and that seems to work ok.
Please let me know if there's anything else I can do/check.
Thanks.
Hello,
the error you are mentioning in our logs, means that eucalyptus keeps querying the hypervisor looking for the instance.Eventually, once assessed that the instance is gone for good, the state will be clear. So it doesn't help, in the sense that we already know that the instance is gone :). Did you find anything else in the systems log around the time you do the reboot?
cheers
graziano