Hello,

the registration part with avahi is a UEC addition, and we, the upstream, don't know too much about it. The scenario you mentioned should just work: you can see our documentation which have examples that I think are very close to what you are proposing.

If you disable autoregistration, you should be able to follow our documentation and register the component by hand.

Let us know if you find more about the autoregistration.

cheers
graziano

Hello there,

today we performed some more tests, trying to reproduce the problems. And we managed to reproduce most of them indeed, apart from the 0.0.0.0 address. I hope that the following steps we performed will help to shed some light on this:

1 rebooted all controllers
2 first instance started fine
3 attached an existing volume
4 rsynced the system from previous backup
5 rebooted the first VI as usual, everything fine.
6 second instance started fine
7 attached another existing volume
8 rsynced the system from previous backup
9 rebooted the second VI as usual, and..

- the 1st instance lost its attached volume
- the 2nd instance lost its attached volume
- both volumes are still attached according to the cloud, but, from within the VI, only the device /dev/sdb shows up, not the partitions /dev/sdb1 or /dev/sdb2
- now all officially attached volumes show the exact same creation date and time as the last volume that was created/attached, which is not the case of course.
- In order to restore functionality, I have to shut down all instances and reboot the cloud, until the problems occur again.

Does the above give you any idea about what's possibly going on? Can avahi on its own cause all this?
The only thing I'm pretty sure of, is that this problems occur only if I perform a dist-upgrade on a standard 10.04 UEC installation. If no upgrades are performed, then the cloud is kind of stable after I solve the registration troubles, which still occur of course. UEC fails to register properly due to the dual NIC setup that UEC is not capable to understand, at least with static addresses, as it expects DHCP only on the second IF.

Best regards,
TritoLux

Thank you Graziano.

The cloud we are testing on is based on 2 systems only, configured exactly as shown in the first post of this thread.

Do you have multiple SCs on the same subnet?
No, only one SC.

Do you have others applications using AOE on your subnet?
No, but currently there are 3 other non-cloud servers on the same subnet.

Can you find any errors in any log (eucalyptus, and systems)?
Yes, please see below:
REGISTRATION-LOG:

2010-08-09 15:48:58+02:00 | 2183 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 15:48:58+02:00 | 2187 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 15:48:58+02:00 | 2183 -> Node is not a candidate for local cluster.
2010-08-09 15:48:58+02:00 | 2187 -> Node is not a candidate for local cluster.
2010-08-09 16:11:09+02:00 | 3948 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 16:11:09+02:00 | 3947 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 16:11:09+02:00 | 3948 -> Node is not a candidate for local cluster.
2010-08-09 16:11:09+02:00 | 3947 -> Node is not a candidate for local cluster.
2010-08-09 16:22:42+02:00 | 3449 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 16:22:42+02:00 | 3450 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 16:22:42+02:00 | 3449 -> Node is not a candidate for local cluster.
2010-08-09 16:22:42+02:00 | 3450 -> Node is not a candidate for local cluster.
2010-08-09 16:38:19+02:00 | 3423 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 16:38:19+02:00 | 3424 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 16:38:19+02:00 | 3423 -> Node is not a candidate for local cluster.
2010-08-09 16:38:19+02:00 | 3424 -> Node is not a candidate for local cluster.
2010-08-09 16:44:48+02:00 | 3935 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 16:44:48+02:00 | 3934 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 16:44:48+02:00 | 3935 -> Node is not a candidate for local cluster.
2010-08-09 16:44:48+02:00 | 3934 -> Node is not a candidate for local cluster.
2010-08-09 16:50:14+02:00 | 5135 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 16:50:14+02:00 | 5135 -> Node is not a candidate for local cluster.
2010-08-09 16:50:21+02:00 | 5156 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 16:50:21+02:00 | 5156 -> Node is not a candidate for local cluster.
2010-08-09 16:55:11+02:00 | 3788 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 16:55:11+02:00 | 3789 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 16:55:11+02:00 | 3789 -> Node is not a candidate for local cluster.
2010-08-09 16:55:11+02:00 | 3788 -> Node is not a candidate for local cluster.
2010-08-09 17:33:01+02:00 | 3591 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 17:33:01+02:00 | 3592 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 17:33:01+02:00 | 3591 -> Node is not a candidate for local cluster.
2010-08-09 17:33:01+02:00 | 3592 -> Node is not a candidate for local cluster.
2010-08-09 18:03:35+02:00 | 3468 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 18:03:35+02:00 | 3467 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 18:03:35+02:00 | 3468 -> Node is not a candidate for local cluster.
2010-08-09 18:03:35+02:00 | 3467 -> Node is not a candidate for local cluster.
2010-08-09 21:56:35+02:00 | 3581 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 21:56:35+02:00 | 3582 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 21:56:35+02:00 | 3582 -> Node is not a candidate for local cluster.
2010-08-09 21:56:35+02:00 | 3581 -> Node is not a candidate for local cluster.
2010-08-09 23:00:38+02:00 | 3582 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 23:00:38+02:00 | 3583 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 23:00:38+02:00 | 3582 -> Node is not a candidate for local cluster.
2010-08-09 23:00:38+02:00 | 3583 -> Node is not a candidate for local cluster.
2010-08-09 23:08:37+02:00 | 4152 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 23:08:37+02:00 | 4153 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 23:08:37+02:00 | 4152 -> Node is not a candidate for local cluster.
2010-08-09 23:08:37+02:00 | 4153 -> Node is not a candidate for local cluster.
2010-08-09 23:10:56+02:00 | 3494 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-09 23:10:56+02:00 | 3495 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-09 23:10:56+02:00 | 3495 -> Node is not a candidate for local cluster.
2010-08-09 23:10:56+02:00 | 3494 -> Node is not a candidate for local cluster.
2010-08-11 01:21:43+02:00 | 3480 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-11 01:21:43+02:00 | 3479 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-11 01:21:44+02:00 | 3480 -> Node is not a candidate for local cluster.
2010-08-11 01:21:44+02:00 | 3479 -> Node is not a candidate for local cluster.
2010-08-11 09:05:50+02:00 | 3467 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-11 09:05:50+02:00 | 3468 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-11 09:05:50+02:00 | 3467 -> Node is not a candidate for local cluster.
2010-08-11 09:05:50+02:00 | 3468 -> Node is not a candidate for local cluster.
2010-08-11 12:34:09+02:00 | 3471 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-11 12:34:09+02:00 | 3472 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-11 12:34:09+02:00 | 3471 -> Node is not a candidate for local cluster.
2010-08-11 12:34:09+02:00 | 3472 -> Node is not a candidate for local cluster.
2010-08-12 17:06:39+02:00 | 3612 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 17:06:39+02:00 | 3613 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 17:06:39+02:00 | 3613 -> Node is not a candidate for local cluster.
2010-08-12 17:06:39+02:00 | 3612 -> Node is not a candidate for local cluster.
2010-08-12 17:16:55+02:00 | 5740 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 17:16:55+02:00 | 5741 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 17:16:55+02:00 | 5740 -> Node is not a candidate for local cluster.
2010-08-12 17:16:55+02:00 | 5741 -> Node is not a candidate for local cluster.
2010-08-12 17:19:11+02:00 | 3550 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 17:19:11+02:00 | 3549 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 17:19:11+02:00 | 3550 -> Node is not a candidate for local cluster.
2010-08-12 17:19:11+02:00 | 3549 -> Node is not a candidate for local cluster.
2010-08-12 17:27:14+02:00 | 3543 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 17:27:14+02:00 | 3544 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 17:27:14+02:00 | 3543 -> Node is not a candidate for local cluster.
2010-08-12 17:27:14+02:00 | 3544 -> Node is not a candidate for local cluster.
2010-08-12 19:42:04+02:00 | 3936 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 19:42:04+02:00 | 3937 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 19:42:04+02:00 | 3936 -> Node is not a candidate for local cluster.
2010-08-12 19:42:04+02:00 | 3937 -> Node is not a candidate for local cluster.
2010-08-12 19:53:34+02:00 | 3561 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 19:53:34+02:00 | 3560 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 19:53:34+02:00 | 3561 -> Node is not a candidate for local cluster.
2010-08-12 19:53:34+02:00 | 3560 -> Node is not a candidate for local cluster.
2010-08-12 20:00:22+02:00 | 3583 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 20:00:22+02:00 | 3582 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 20:00:22+02:00 | 3583 -> Node is not a candidate for local cluster.
2010-08-12 20:00:22+02:00 | 3582 -> Node is not a candidate for local cluster.
2010-08-12 20:10:59+02:00 | 3544 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-12 20:10:59+02:00 | 3545 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-12 20:10:59+02:00 | 3545 -> Node is not a candidate for local cluster.
2010-08-12 20:10:59+02:00 | 3544 -> Node is not a candidate for local cluster.
2010-08-13 23:15:57+02:00 | 10431 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-13 23:15:57+02:00 | 10431 -> Node is not a candidate for local cluster.
2010-08-14 08:18:06+02:00 | 2550 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-14 08:18:06+02:00 | 2550 -> Node is not a candidate for local cluster.
2010-08-21 18:54:32+02:00 | 3648 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-21 18:54:32+02:00 | 3647 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-21 18:54:32+02:00 | 3648 -> Node is not a candidate for local cluster.
2010-08-21 18:54:32+02:00 | 3647 -> Node is not a candidate for local cluster.
2010-08-21 18:59:55+02:00 | 3662 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-21 18:59:55+02:00 | 3661 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-08-21 18:59:55+02:00 | 3662 -> Node is not a candidate for local cluster.
2010-08-21 18:59:55+02:00 | 3661 -> Node is not a candidate for local cluster.
2010-08-24 05:12:25+02:00 | 24287 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-08-24 05:12:25+02:00 | 24287 -> Node is not a candidate for local cluster.
2010-09-02 15:52:39+02:00 | 11597 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-09-02 15:52:39+02:00 | 11597 -> Node is not a candidate for local cluster.
2010-09-02 15:52:46+02:00 | 11617 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-09-02 15:52:46+02:00 | 11617 -> Node is not a candidate for local cluster.
2010-09-02 15:54:42+02:00 | 3673 -> Calling node Cluster01 node NC.PRIVATE.IP.196
2010-09-02 15:54:42+02:00 | 3672 -> Calling node Cluster01 node NC.PUBLIC.IP.196
2010-09-02 15:54:42+02:00 | 3673 -> Node is not a candidate for local cluster.
2010-09-02 15:54:42+02:00 | 3672 -> Node is not a candidate for local cluster.


NC.LOG

[Thu Sep 2 13:46:53 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-40C0086B' (code=42)
[Thu Sep 2 13:47:06 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-40C0086B' (code=42)
[Thu Sep 2 13:57:03 2010][010959][EUCAERROR ] ERROR: doTerminateInstance() failed error=101
[Thu Sep 2 14:43:37 2010][010959][EUCAERROR ] libvirt: operation failed: disk sdb not found (code=9)
[Thu Sep 2 14:43:37 2010][010959][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML=
[Thu Sep 2 14:43:37 2010][010959][EUCAERROR ] ERROR: doDetachVolume() failed error=1
[Thu Sep 2 14:46:05 2010][010959][EUCAERROR ] libvirt: operation failed: disk sdb not found (code=9)
[Thu Sep 2 14:46:05 2010][010959][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML=
[Thu Sep 2 14:46:05 2010][010959][EUCAERROR ] ERROR: doDetachVolume() failed error=1
[Thu Sep 2 14:49:31 2010][010959][EUCAERROR ] libvirt: operation failed: disk sdb not found (code=9)
[Thu Sep 2 14:49:31 2010][010959][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML=
[Thu Sep 2 14:49:31 2010][010959][EUCAERROR ] ERROR: doDetachVolume() failed error=1
[Thu Sep 2 14:51:41 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-3B52075F' (code=42)
[Thu Sep 2 14:52:10 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-3B52075F' (code=42)
[Thu Sep 2 15:01:58 2010][010959][EUCAERROR ] ERROR: doTerminateInstance() failed error=101
[Thu Sep 2 15:02:06 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-309B065E' (code=42)
[Thu Sep 2 15:02:17 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-309B065E' (code=42)
[Thu Sep 2 15:12:18 2010][010959][EUCAERROR ] ERROR: doTerminateInstance() failed error=101
[Thu Sep 2 15:29:18 2010][010959][EUCAERROR ] libvirt: operation failed: disk sdb not found (code=9)
[Thu Sep 2 15:29:18 2010][010959][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML=
[Thu Sep 2 15:29:18 2010][010959][EUCAERROR ] ERROR: doDetachVolume() failed error=1
[Thu Sep 2 15:30:23 2010][010959][EUCAERROR ] libvirt: operation failed: disk sdb not found (code=9)
[Thu Sep 2 15:30:23 2010][010959][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML=
[Thu Sep 2 15:30:23 2010][010959][EUCAERROR ] ERROR: doDetachVolume() failed error=1
[Thu Sep 2 15:30:57 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-571F0955' (code=42)
[Thu Sep 2 15:31:06 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-571F0955' (code=42)
[Thu Sep 2 15:38:57 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-44B107B9' (code=42)
[Thu Sep 2 15:39:06 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-44B107B9' (code=42)
[Thu Sep 2 15:41:20 2010][010959][EUCAERROR ] ERROR: doTerminateInstance() failed error=101
[Thu Sep 2 15:41:20 2010][010959][EUCAERROR ] libvirt: internal error Invalid harddisk device name: electra-emd (code=1)
[Thu Sep 2 15:41:20 2010][010959][EUCAERROR ] virDomainAttachDevice() failed (err=-1) XML=
[Thu Sep 2 15:41:20 2010][010959][EUCAERROR ] ERROR: doAttachVolume() failed error=1
[Thu Sep 2 15:48:12 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-46B30805' (code=42)
[Thu Sep 2 15:48:28 2010][010959][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-46B30805' (code=42)
[Thu Sep 2 15:49:07 2010][010959][EUCAERROR ] ERROR: doTerminateInstance() failed error=101[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf///etc/eucalyptus/eucalyptus.local.conf
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] SC is looking for configuration in files (//etc/eucalyptus/eucalyptus.conf,//etc/eucalyptus/eucalyptus.local.conf)
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-cert.pem
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-pk.pem
[Thu Sep 2 15:52:33 2010][001441][EUCADEBUG ] doInitialized() invoked
[Thu Sep 2 15:52:33 2010][001441][EUCADEBUG ] system_output(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/get_sys_info]
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] Using 16 cores
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] Using 20348 memory
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] looking for existing domains
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] no currently running domains to adopt
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] vnetInit(): VNET Configuration: eucahome=/, path=//var/run/eucalyptus/net, dhcpdaemon=, dhcpuser=, pubInterface=br1, privInterface=br1, bridgedev=br1, networkMode=MANAGED-NOVLAN
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] checking the integrity of instances directory (/var/lib/eucalyptus/instances/)
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client2/i-309B065E]
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client2/i-40C0086B]
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client2/i-3B52075F]
[Thu Sep 2 15:52:33 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client2/i-571F0955]
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client2/i-46B30805]
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client1/i-534308D7]
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client1/i-435F06CF]
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//client1/i-44B107B9]
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] checking the integrity of the cache directory (/var/lib/eucalyptus/instances//eucalyptus/cache)
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image emi-E2E7147C directory, size=1075849220
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eri-0AF8115B directory, size=4248915
[Thu Sep 2 15:52:34 2010][001441][EUCAWARN ] warning: empty cached image directory /var/lib/eucalyptus/instances//eucalyptus/cache/emi-DF0C1068
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image emi-E3331482 directory, size=1075849148
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eri-D39017EB directory, size=8686967
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eri-0A3E1157 directory, size=3870630
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eri-09BB1152 directory, size=3870630
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eki-EF5A14DD directory, size=3529891
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eki-085C1539 directory, size=3529891
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eki-F66A10EE directory, size=4100791
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eri-047B1888 directory, size=8686967
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image emi-E0501074 directory, size=1512056575
[Thu Sep 2 15:52:34 2010][001441][EUCAWARN ] warning: empty cached image directory /var/lib/eucalyptus/instances//eucalyptus/cache/emi-DFCB1080
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eki-F5F210F8 directory, size=3942359
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] - cached image eki-F53D10E9 directory, size=3942359
[Thu Sep 2 15:52:34 2010][001441][EUCAINFO ] Maximum disk available = 843444 (under /var/lib/eucalyptus/instance


I'm going to post the cloud-errors in a different way, as it crashes the post if I paste it in..

Thank you for your help.

Regards,
TritoLux


15:53:08 [ProxoolDriver:SystemClockTimer] ERROR Problem
org.logicalcobwebs.proxool.ProxoolException: Attempt to refer to a unregistered pool by its alias 'eucalyptus_general'
at org.logicalcobwebs.proxool.ConnectionPoolDefinition.doChange(ConnectionPoolDefinition.java:246)
at org.logicalcobwebs.proxool.ConnectionPoolDefinition.(ConnectionPoolDefinition.java:143)
at org.logicalcobwebs.proxool.ProxoolFacade.registerConnectionPool(ProxoolFacade.java:96)
at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:77)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:222)
at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:75)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
at com.eucalyptus.util.TxHandle.(TxHandle.java:40)
at com.eucalyptus.util.EntityWrapper.(EntityWrapper.java:98)
at com.eucalyptus.util.EntityWrapper.(EntityWrapper.java:91)
at edu.ucsb.eucalyptus.cloud.cluster.VmTypes.update(VmTypes.java:119)
at edu.ucsb.eucalyptus.cloud.cluster.VmTypes.list(VmTypes.java:148)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.trigger(ResourceStateHandler.java:31)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.fireTimedStatefulTrigger(AbstractClusterMessageDispatcher.java:141)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.fireEvent(ResourceStateHandler.java:39)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:80)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:65)
at com.eucalyptus.event.ListenerRegistry.fireEvent(ListenerRegistry.java:69)
at com.eucalyptus.event.SystemClock.run(SystemClock.java:35)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
15:53:08 [TxHandle:SystemClockTimer] ERROR javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
at com.eucalyptus.util.TxHandle.(TxHandle.java:40)
at com.eucalyptus.util.EntityWrapper.(EntityWrapper.java:98)
at com.eucalyptus.util.EntityWrapper.(EntityWrapper.java:91)
at edu.ucsb.eucalyptus.cloud.cluster.VmTypes.update(VmTypes.java:119)
at edu.ucsb.eucalyptus.cloud.cluster.VmTypes.list(VmTypes.java:148)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.trigger(ResourceStateHandler.java:31)
at com.eucalyptus.cluster.handlers.AbstractClusterMessageDispatcher.fireTimedStatefulTrigger(AbstractClusterMessageDispatcher.java:141)
at com.eucalyptus.cluster.handlers.ResourceStateHandler.fireEvent(ResourceStateHandler.java:39)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:80)
at com.eucalyptus.event.ReentrantListenerRegistry.fireEvent(ReentrantListenerRegistry.java:65)
at com.eucalyptus.event.ListenerRegistry.fireEvent(ListenerRegistry.java:69)
at com.eucalyptus.event.SystemClock.run(SystemClock.java:35)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
... 14 more
Caused by: java.sql.SQLException: org.logicalcobwebs.proxool.ProxoolException: Attempt to refer to a unregistered pool by its alias 'eucalyptus_general'
at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:109)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:222)
at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:75)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 19 more
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop5
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop5
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.getLoopback(LVM2Manager.java:162)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:774)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop3
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop3
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.getLoopback(LVM2Manager.java:162)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:774)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop2
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop2
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.getLoopback(LVM2Manager.java:162)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:774)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)
15:54:37 [LVM2Manager:Thread-12] ERROR Backing volume: //var/lib/eucalyptus/volumes/vol-7D850780 not found. Invalidating volume.
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop4
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop4
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.getLoopback(LVM2Manager.java:162)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:774)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop1
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop1
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.getLoopback(LVM2Manager.java:162)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:774)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop0
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop0
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.getLoopback(LVM2Manager.java:162)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:774)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)
15:54:37 [SystemUtil:Thread-12] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap lvchange -ay /dev/vg-Z36Zhg../lv-hjVZ0w..
com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap lvchange -ay /dev/vg-Z36Zhg../lv-hjVZ0w..
at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91)
at edu.ucsb.eucalyptus.storage.LVM2Manager.enableLogicalVolume(LVM2Manager.java:206)
at edu.ucsb.eucalyptus.storage.LVM2Manager.access$900(LVM2Manager.java:105)
at edu.ucsb.eucalyptus.storage.LVM2Manager$VolumeEntityWrapperManager.exportVolume(LVM2Manager.java:867)
at edu.ucsb.eucalyptus.storage.LVM2Manager.reload(LVM2Manager.java:787)
at edu.ucsb.eucalyptus.storage.LVM2Manager.startupChecks(LVM2Manager.java:265)
at edu.ucsb.eucalyptus.storage.BlockStorageChecker$StartupChecker.run(BlockStorageChecker.java:118)


graziano says... Hello,

from the latest NC posts, it looks like the NC either got abruptly stopped or crashed: can you checked other logs even system logs for weirdness around that time?

The other suggestions that comes to mind is to check the webui (as administrator): in the config tab there is a section about the storage controller. Which interfaces is specified there?

cheers
graziano

Hello there,

I am getting back to this problem after
http://open.eucalyptus.com/forum/libvirt-operation-failed-failed-retriev...
The interface configured on the webui is br1.
Here follows a list of all errors I am getting in the logs when the problem as explained in post Fri, 09/03/2010 - 00:14 occurs:

Console from VI with corrupted volumes after instance reboot:

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.31-14-server (buildd@crested) (gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu8) ) #48-Ubuntu SMP Fri Oct 16 15:07:34 UTC 2009 (Ubuntu 2.6.31-14.48-server)
[ 0.000000] Command line: root=/dev/sda1 console=ttyS0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[ 0.000000] BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 00000000383fb000 (usable)
[ 0.000000] BIOS-e820: 00000000383fb000 - 0000000038400000 (reserved)
[ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved)
[ 0.000000] DMI 2.4 present.
[ 0.000000] last_pfn = 0x383fb max_arch_pfn = 0x400000000
[ 0.000000] PAT not supported by CPU.
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] modified physical RAM map:
[ 0.000000] modified: 0000000000000000 - 0000000000001000 (usable)
[ 0.000000] modified: 0000000000001000 - 0000000000006000 (reserved)
[ 0.000000] modified: 0000000000006000 - 000000000009f000 (usable)
[ 0.000000] modified: 000000000009f000 - 00000000000a0000 (reserved)
[ 0.000000] modified: 00000000000f0000 - 0000000000100000 (reserved)
[ 0.000000] modified: 0000000000100000 - 00000000383fb000 (usable)
[ 0.000000] modified: 00000000383fb000 - 0000000038400000 (reserved)
[ 0.000000] modified: 00000000fffbc000 - 0000000100000000 (reserved)
[ 0.000000] init_memory_mapping: 0000000000000000-00000000383fb000
[ 0.000000] Using x86 segment limits to approximate NX protection
[ 0.000000] RAMDISK: 38040000 - 383ef20c
[ 0.000000] ACPI: RSDP 00000000000f8940 00014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 00000000383fde30 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 00000000383ffe70 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 00000000383fdfd0 01E22 (v01 BXPC BXDSDT 00000001 INTL 20090123)
[ 0.000000] ACPI: FACS 00000000383ffe00 00040
[ 0.000000] ACPI: SSDT 00000000383fdf90 00037 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: APIC 00000000383fdeb0 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.000000] ACPI: HPET 00000000383fde70 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at 0000000000000000-00000000383fb000
[ 0.000000] Bootmem setup node 0 0000000000000000-00000000383fb000
[ 0.000000] NODE_DATA [0000000000009000 - 000000000000dfff]
[ 0.000000] bootmap [000000000000e000 - 000000000001507f] pages 8
[ 0.000000] (7 early reservations) ==> bootmem [0000000000 - 00383fb000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
[ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
[ 0.000000] #2 [0001000000 - 00019e0ccc] TEXT DATA BSS ==> [0001000000 - 00019e0ccc]
[ 0.000000] #3 [0038040000 - 00383ef20c] RAMDISK ==> [0038040000 - 00383ef20c]
[ 0.000000] #4 [000009f000 - 0000100000] BIOS reserved ==> [000009f000 - 0000100000]
[ 0.000000] #5 [00019e1000 - 00019e1071] BRK ==> [00019e1000 - 00019e1071]
[ 0.000000] #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000]
[ 0.000000] found SMP MP-table at [ffff8800000f8990] f8990
[ 0.000000] kvm-clock: cpu 0, msr 0:1836801, boot clock
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000000 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal 0x00100000 -> 0x00100000
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[3] active PFN ranges
[ 0.000000] 0: 0x00000000 -> 0x00000001
[ 0.000000] 0: 0x00000006 -> 0x0000009f
[ 0.000000] 0: 0x00000100 -> 0x000383fb
[ 0.000000] ACPI: PM-Timer IO Port: 0xb008
[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] PM: Registered nosave memory: 0000000000001000 - 0000000000006000
[ 0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
[ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
[ 0.000000] Allocating PCI resources starting at 38400000 (gap: 38400000:c7bbc000)
[ 0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 30 pages at ffff8800019ec000, static data 90720 bytes
[ 0.000000] kvm-clock: cpu 0, msr 0:1a01801, primary cpu clock
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 227042
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: root=/dev/sda1 console=ttyS0
[ 0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] Memory: 892680k/921580k available (5303k kernel code, 408k absent, 28492k reserved, 3013k data, 660k init)
[ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS:4352 nr_irqs:256
[ 0.000000] Detected 2500.116 MHz processor.
[ 0.010000] Console: colour VGA+ 80x25
[ 0.010000] console [ttyS0] enabled
[ 0.010000] allocated 10485760 bytes of page_cgroup
[ 0.010000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[ 0.010000] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[ 0.060013] Calibrating delay loop (skipped) preset value.. 5000.23 BogoMIPS (lpj=25001160)
[ 0.061707] Security Framework initialized
[ 0.062509] AppArmor: AppArmor initialized
[ 0.063387] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.066346] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.068344] Mount-cache hash table entries: 256
[ 0.070353] Initializing cgroup subsys ns
[ 0.071192] Initializing cgroup subsys cpuacct
[ 0.072104] Initializing cgroup subsys memory
[ 0.073016] Initializing cgroup subsys freezer
[ 0.073865] Initializing cgroup subsys net_cls
[ 0.074802] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.075717] CPU: L2 cache: 4096K
[ 0.076376] CPU 0/0x0 -> Node 0
[ 0.076991] mce: CPU supports 10 MCE banks
[ 0.077866] Performance Counters: unsupported p6 CPU model 2 no PMU driver, software counters only.
[ 0.080036] SMP alternatives: switching to UP code
[ 0.180007] Freeing SMP alternatives: 39k freed
[ 0.180903] ACPI: Core revision 20090521
[ 0.182600] Setting APIC routing to flat
[ 0.184916] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.186137] CPU0: Intel QEMU Virtual CPU version 0.12.3 stepping 03
[ 0.200000] Brought up 1 CPUs
[ 0.200000] Total of 1 processors activated (5000.23 BogoMIPS).
[ 0.200000] Booting paravirtualized kernel on KVM
[ 0.200000] regulator: core version 0.5
[ 0.200000] Time: 7:23:50 Date: 11/17/10
[ 0.200000] NET: Registered protocol family 16
[ 0.200000] ACPI: bus type pci registered
[ 0.200000] PCI: Using configuration type 1 for base access
[ 0.200000] bio: create slab at 0
[ 0.202168] ACPI: Interpreter enabled
[ 0.202967] ACPI: (supports S0 S3 S4 S5)
[ 0.203908] ACPI: Using IOAPIC for interrupt routing
[ 0.207066] ACPI: No dock devices found.
[ 0.207850] ACPI: PCI Root Bridge [PCI0] (0000:00)
[ 0.211619] pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
[ 0.213109] pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
[ 0.224331] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[ 0.225703] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.227089] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.228474] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[ 0.230407] SCSI subsystem initialized
[ 0.231584] usbcore: registered new interface driver usbfs
[ 0.232813] usbcore: registered new interface driver hub
[ 0.233967] usbcore: registered new device driver usb
[ 0.235279] ACPI: WMI: Mapper loaded
[ 0.236089] PCI: Using ACPI for IRQ routing
[ 0.237327] Bluetooth: Core ver 2.15
[ 0.238193] NET: Registered protocol family 31
[ 0.239120] Bluetooth: HCI device and connection manager initialized
[ 0.240008] Bluetooth: HCI socket layer initialized
[ 0.241029] NetLabel: Initializing
[ 0.241753] NetLabel: domain hash size = 128
[ 0.242627] NetLabel: protocols = UNLABELED CIPSOv4
[ 0.243677] NetLabel: unlabeled traffic allowed by default
[ 0.244999] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 0.246094] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[ 0.263212] pnp: PnP ACPI init
[ 0.263795] ACPI: bus type pnp registered
[ 0.265585] pnp: PnP ACPI: found 7 devices
[ 0.266538] ACPI: ACPI bus type pnp unregistered
[ 0.272591] AppArmor: AppArmor Filesystem Enabled
[ 0.273749] NET: Registered protocol family 2
[ 0.274866] IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 0.277054] TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
[ 0.281287] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.284015] TCP: Hash tables configured (established 131072 bind 65536)
[ 0.285416] TCP reno registered
[ 0.286152] NET: Registered protocol family 1
[ 0.287135] Trying to unpack rootfs image as initramfs...
[ 0.374068] Freeing initrd memory: 3772k freed
[ 0.376502] Scanning for low memory corruption every 60 seconds
[ 0.377929] audit: initializing netlink socket (disabled)
[ 0.379080] type=2000 audit(1289978631.370:1): initialized
[ 0.389852] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.392469] VFS: Disk quotas dquot_6.5.2
[ 0.393320] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.395320] fuse init (API version 7.12)
[ 0.396248] msgmni has been set to 1750
[ 0.397455] alg: No test for stdrng (krng)
[ 0.398278] io scheduler noop registered
[ 0.399132] io scheduler anticipatory registered
[ 0.400109] io scheduler deadline registered (default)
[ 0.401182] io scheduler cfq registered
[ 0.402001] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.403207] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.404405] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.405758] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 0.406854] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 0.408156] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 0.409484] ACPI: Power Button [PWRF]
[ 0.410566] processor LNXCPU:00: registered as cooling_device0
[ 0.413279] Linux agpgart interface v0.103
[ 0.414034] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.415474] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.416994] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.418990] brd: module loaded
[ 0.420031] loop: module loaded
[ 0.420688] input: Macintosh mouse button emulation as /devices/virtual/input/input1
[ 0.422646] scsi0 : ata_piix
[ 0.423291] scsi1 : ata_piix
[ 0.423859] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
[ 0.425102] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
[ 0.427000] Fixed MDIO Bus: probed
[ 0.427801] PPP generic driver version 2.4.2
[ 0.428666] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.429849] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 0.430996] uhci_hcd: USB Universal Host Controller Interface driver
[ 0.432692] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[ 0.433755] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
[ 0.435315] uhci_hcd 0000:00:01.2: UHCI Host Controller
[ 0.436312] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
[ 0.437772] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020
[ 0.438948] usb usb1: configuration #1 chosen from 1 choice
[ 0.440159] hub 1-0:1.0: USB hub found
[ 0.441010] hub 1-0:1.0: 2 ports detected
[ 0.441980] PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[ 0.444191] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 0.445220] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 0.446374] mice: PS/2 mouse device common for all mice
[ 0.447624] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[ 0.449050] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs
[ 0.450513] device-mapper: uevent: version 1.0.3
[ 0.452146] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[ 0.454263] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
[ 0.456018] device-mapper: multipath: version 1.1.0 loaded
[ 0.457160] device-mapper: multipath round-robin: version 1.0.0 loaded
[ 0.458691] cpuidle: using governor ladder
[ 0.459563] cpuidle: using governor menu
[ 0.460757] TCP cubic registered
[ 0.461624] NET: Registered protocol family 10
[ 0.463332] lo: Disabled Privacy Extensions
[ 0.464614] NET: Registered protocol family 17
[ 0.465661] Bluetooth: L2CAP ver 2.13
[ 0.466481] Bluetooth: L2CAP socket layer initialized
[ 0.467664] Bluetooth: SCO (Voice Link) ver 0.6
[ 0.468693] Bluetooth: SCO socket layer initialized
[ 0.469912] Bluetooth: RFCOMM TTY layer initialized
[ 0.471082] Bluetooth: RFCOMM socket layer initialized
[ 0.472253] Bluetooth: RFCOMM ver 1.11
[ 0.473325] registered taskstats version 1
[ 0.474275] Magic number: 2:870:369
[ 0.475126] rtc_cmos 00:01: setting system clock to 2010-11-17 07:23:50 UTC (1289978630)
[ 0.476647] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 0.477777] EDD information not available.
[ 0.610764] ata2.00: ATAPI: QEMU DVD-ROM, 0.12.3, max UDMA/100
[ 0.612294] ata2.00: configured for MWDMA2
[ 0.613947] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 0.12 PQ: 0 ANSI: 5
[ 0.617575] sr0: scsi3-mmc drive: 4x/4x xa/form2 tray
[ 0.618642] Uniform CD-ROM driver Revision: 3.20
[ 0.619730] sr 1:0:0:0: Attached scsi generic sg0 type 5
[ 0.621015] Freeing unused kernel memory: 660k freed
[ 0.622512] Write protecting the kernel read-only data: 7572k
Loading, please wait...
Couldn't get a file descriptor referring to the console
Begin: Loading essential drivers... ...
[ 1.171691] Intel(R) PRO/1000 Network Driver - version 7.3.21-k3-NAPI
[ 1.173080] Copyright (c) 1999-2006 Intel Corporation.
Done.
Begin: Running /scripts/init-premount ...
[ 1.205243] sym53c8xx 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
[ 1.208812] sym0: <895a> rev 0x0 at pci 0000:00:04.0 irq 11
[ 1.212259] sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
[ 1.220018] sym0: SCSI BUS has been reset.
[ 1.229038] scsi2 : sym-2.2.3
[ 1.270987] FDC 0 is a S82078B
[ 1.336429] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
[ 1.337632] e1000 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Waiting for root file system... ...
[ 1.656107] e1000: 0000:00:03.0: e1000_probe: (PCI:33MHz:32-bit) d0:0d:3c:d1:07:5a
[ 1.734689] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[ 4.210218] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 0.12 PQ: 0 ANSI: 3
[ 4.212033] scsi target2:0:0: tagged command queuing enabled, command queue depth 16.
[ 4.213775] scsi target2:0:0: Beginning Domain Validation
[ 4.215446] scsi target2:0:0: Domain Validation skipping write tests
[ 4.216819] scsi target2:0:0: Ending Domain Validation
[ 4.221452] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 4.222824] sd 2:0:0:0: [sda] 21063680 512-byte logical blocks: (10.7 GB/10.0 GiB)
[ 4.224238] sd 2:0:0:0: [sda] Write Protect is off
[ 4.225220] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 4.227214] sda: sda1 sda2 sda3
[ 4.229279] sd 2:0:0:0: [sda] Attached SCSI disk
Done.
Begin: Running /scripts/local-premount ...
Done.
[ 4.406029] EXT3-fs: INFO: recovery required on readonly filesystem.
[ 4.407176] EXT3-fs: write access will be enabled during recovery.
[ 5.890668] kjournald starting. Commit interval 5 seconds
[ 5.891897] EXT3-fs: sda1: orphan cleanup on readonly fs
[ 5.893232] EXT3-fs: sda1: 5 orphan inodes deleted
[ 5.894268] EXT3-fs: recovery complete.
[ 5.956321] EXT3-fs: mounted filesystem with writeback data mode.
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Begin: Starting AppArmor profiles ...
[ 6.042071] type=1505 audit(1289978636.064:2): operation="profile_load" pid=314 name=/sbin/dhclient3
[ 6.044938] type=1505 audit(1289978636.064:3): operation="profile_load" pid=314 name=/usr/lib/NetworkManager/nm-dhcp-client.action
[ 6.047937] type=1505 audit(1289978636.064:4): operation="profile_load" pid=314 name=/usr/lib/connman/scripts/dhclient-script
[ 6.054902] type=1505 audit(1289978636.075:5): operation="profile_load" pid=315 name=/usr/sbin/mysqld
[ 6.061523] type=1505 audit(1289978636.084:6): operation="profile_load" pid=316 name=/usr/sbin/tcpdump
Done.
Done.
[ 6.191727] EXT3-fs warning: checktime reached, running e2fsck is recommended
[ 6.202172] EXT3 FS on sda1, internal journal
* Setting preliminary keymap...
[80G * Starting AppArmor profiles
[80G * Setting up console font and keymap...
[80G * Waiting for EC2 meta-data service
[80G * Starting OpenBSD Secure Shell server sshd
[80G * Starting MySQL database server mysqld
[80G ...done.
* Checking for corrupt, not cleanly closed and upgrade needing tables.
* Starting web server apache2
[80G


Will paste more errors in the following post, otherwise this post crashes..

cat /var/log/eucalyptus/cc.log | grep ERROR

[Wed Nov 17 08:31:24 2010][000522][EUCAERROR ] ERROR: DetachVolume returned an error
[Wed Nov 17 08:31:24 2010][001932][EUCAERROR ] DetachVolume(): call to NC failed: instanceId=i-3CD1075A
[Wed Nov 17 08:31:24 2010] ERROR: doDetachVolume() returned FAIL


$ cat /var/log/eucalyptus/nc.log | grep ERROR

[Wed Nov 17 08:31:37 2010][001368][EUCAERROR ] libvirt: operation failed: disk sdb not found (code=9)
[Wed Nov 17 08:31:37 2010][001368][EUCAERROR ] virDomainDetachDevice() failed (err=-1) XML=
[Wed Nov 17 08:31:37 2010][001368][EUCAERROR ] ERROR: doDetachVolume() failed error=1
[Wed Nov 17 08:39:41 2010][001368][EUCAERROR ] cannot stat console_output file '/var/lib/eucalyptus/instances//admin/i-3CD1075A/console.log'
[Wed Nov 17 08:39:41 2010][001368][EUCAERROR ] ERROR: doGetConsoleOutput() failed error=1
[Wed Nov 17 08:40:55 2010][001368][EUCAERROR ] cannot stat console_output file '/var/lib/eucalyptus/instances//admin/i-3C32069E/console.log'
[Wed Nov 17 08:40:55 2010][001368][EUCAERROR ] ERROR: doGetConsoleOutput() failed error=1


$ cat /var/log/eucalyptus/httpd-cc_error_log

[Tue Nov 16 13:29:28 2010] [notice] Apache/2.2.14 (Ubuntu) Axis2C/1.6.0 configured -- resuming normal operations
[Tue Nov 16 13:29:17 2010] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
iptables-restore: line 8 failed
iptables-restore failed
iptables-restore: line 9 failed
iptables-restore failed
Internet Systems Consortium DHCP Server V3.1.3
Copyright 2004-2009 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
WARNING: Host declarations are global. They are not limited to the scope you declared them in.
Wrote 0 deleted host decls to leases file.
Wrote 0 new dynamic host decls to leases file.
Wrote 0 leases to leases file.
Listening on LPF/br1/00:25:90:05:26:b1/euca
Sending on LPF/br1/00:25:90:05:26:b1/euca
Sending on Socket/fallback/fallback-net
Using intrapositioned negation (`--option ! this`) is deprecated in favor of extrapositioned (`! --option this`).
Using intrapositioned negation (`--option ! this`) is deprecated in favor of extrapositioned (`! --option this`).
Using intrapositioned negation (`--option ! this`) is deprecated in favor of extrapositioned (`! --option this`).
iptables-restore: line 13 failed
iptables-restore failed
iptables-restore: line 14 failed
iptables-restore failed
Internet Systems Consortium DHCP Server V3.1.3
Copyright 2004-2009 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
WARNING: Host declarations are global. They are not limited to the scope you declared them in.
Wrote 0 deleted host decls to leases file.
Wrote 0 new dynamic host decls to leases file.
Wrote 0 leases to leases file.
Listening on LPF/br1/00:25:90:05:26:b1/euca
Sending on LPF/br1/00:25:90:05:26:b1/euca
Sending on Socket/fallback/fallback-net
Using intrapositioned negation (`--option ! this`) is deprecated in favor of extrapositioned (`! --option this`).