Users can't access the instances they run from their workstation, but can from the cloud controller system

I don't know whether this is only my setup or it's something needs attention.
my cloud setup:
OS: centos5.5_x86xen
eucalyptus version: 1.6.2
euca2ools version: 1.2
network mode: managed-novlan
system iptables : disabled

sympton:
If a user download his credential and run the instance, the instances were created as expected, but, the instances are
not accessible from outside of the cloud controller machine.

Found the problem, and can manually fix it:
I think because of the setting of VNET_ADDRESSPERNET="32", the first user (usually with admin credential) will have the first block of address space, (here 192.168.1.1-192.168.1.32) and since issuing euca-authorize command, this entire block is allowed access for group "default".
Now, a new user get his credential downloaded and run instance, his instance will use the 2nd block of ip address, and since the group "default" is used by above user, he can neither issue euca-authorize for the "default" group, nor create a new "group" and run euca-authorize to allow access to his block of subnet.
The iptables output does show their individual chain were contain now rules to allow access.
Maybe iptables find some conflicts and do not automatically setup to allow access to this block of ip? or something else.

Temporary Solution:
To fix this, is to manually append the following, then add to /var/run/eucalyptus/net/iptables-preload:
iptables -A -s 0.0.0.0/0 -d 192.168.1.33/27 -j ACCEPT
iptables -A -s 0.0.0.0/0 -d 192.168.1.65/27 -j ACCEPT
....

This does solve the problem, but this does not sound right, is there any setup that I could be missing?
Can anyone shed some light?
Thank You

clc configuration:
EUCALYPTUS="/"
EUCA_USER="eucalyptus"
DISABLE_DNS="N"
ENABLE_WS_SECURITY="Y"
LOGLEVEL="DEBUG"
CC_PORT="8774"
SCHEDPOLICY="ROUNDROBIN"
POWER_IDLETHRESH="300"
POWER_WAKETHRESH="300"
NODES="10.208.2.11 10.208.2.12"
NC_SERVICE="axis2/services/EucalyptusNC"
NC_PORT="8775"
HYPERVISOR="xen"
INSTANCE_PATH="/usr/local/eucalyptus/"
VNET_PUBINTERFACE="eth0"
VNET_PRIVINTERFACE="eth0"
VNET_BRIDGE="xenbr0"
VNET_DHCPDAEMON="/usr/sbin/dhcpd"
VNET_MODE="MANAGED-NOVLAN"
VNET_SUBNET="192.168.0.0"
VNET_NETMASK="255.255.0.0"
VNET_DNS="xxx.yyy.zzz.36"
VNET_ADDRSPERNET="32"
VNET_PUBLICIPS="10.208.2.32-10.208.2.255"

iptables -L:
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate ESTABLISHED
ACCEPT all -- anywhere !192.168.0.0/16
ACCEPT all -- 192.168.1.0/27 192.168.1.0/27
nehal-default all -- anywhere anywhere
ACCEPT all -- 192.168.1.32/27 192.168.1.32/27
admin-default all -- anywhere anywhere
ACCEPT all -- 192.168.1.64/27 192.168.1.64/27
ACCEPT all -- anywhere anywhere
larryaero-default all -- anywhere anywhere
ACCEPT all -- 192.168.1.64/27 192.168.1.64/27

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain admin-default (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 192.168.1.0/27 tcp dpt:ssh
ACCEPT icmp -- anywhere 192.168.1.0/27

Chain larryaero-default (1 references)
target prot opt source destination

Chain nehal-default (1 references)
target prot opt source destination

NOTE: both users' chains have no rules, I have to manually append the above mentioned rule to allow access from corporate net.

ip route sh:
192.168.1.32/27 dev eth0 proto kernel scope link src 192.168.1.33
192.168.1.0/27 dev eth0 proto kernel scope link src 192.168.1.1
192.168.1.64/27 dev eth0 proto kernel scope link src 192.168.1.65
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
10.208.2.0/23 dev eth0 proto kernel scope link src 10.208.2.8
xxx.yyy.zzz.0/21 dev eth1 proto kernel scope link src xxx.yyy.zzz.76
169.254.0.0/16 dev eth0 scope link
default via 10.208.2.1 dev eth0

ip a sh
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth2: mtu 1500 qdisc noop qlen 1000
link/ether 00:07:43:05:7c:87 brd ff:ff:ff:ff:ff:ff
3: peth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
4: eth1: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:30:48:d1:b1:63 brd ff:ff:ff:ff:ff:ff
inet xxx.yyy.zzz.76/21 brd xxx.yyy.zzz.255 scope global eth1
5: virbr0: mtu 1500 qdisc noqueue
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
6: vif0.0: mtu 1500 qdisc noqueue
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
7: eth0: mtu 1500 qdisc noqueue
link/ether 00:30:48:d1:b1:62 brd ff:ff:ff:ff:ff:ff
inet 169.254.169.254/32 scope link eth0
inet 10.208.2.8/23 brd 10.208.3.255 scope global eth0
inet 192.168.1.1/27 brd 192.168.1.31 scope global eth0
inet 10.208.2.100/32 scope global eth0
inet 10.208.2.101/32 scope global eth0
inet 10.208.2.102/32 scope global eth0
inet 10.208.2.103/32 scope global eth0
inet 192.168.1.33/27 brd 192.168.1.63 scope global eth0
inet 10.208.2.104/32 scope global eth0
inet 10.208.2.105/32 scope global eth0
inet 192.168.1.65/27 brd 192.168.1.95 scope global eth0
inet 10.208.2.106/32 scope global eth0
inet 10.208.2.107/32 scope global eth0
8: vif0.1: mtu 1500 qdisc noop
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
9: veth1: mtu 1500 qdisc noop
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: vif0.2: mtu 1500 qdisc noop
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: veth2: mtu 1500 qdisc noop
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
12: vif0.3: mtu 1500 qdisc noop
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
13: veth3: mtu 1500 qdisc noop
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
14: vif0.4: mtu 1500 qdisc noop
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
15: veth4: mtu 1500 qdisc noop
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
16: xenbr0: mtu 1500 qdisc noqueue
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

Thu, 07/29/2010 - 14:09

larryaero

Offline

Joined: 05/18/2010

Does anyone experience this

Does anyone experience this kind of problem? Any help is really appreciated.

Fri, 07/30/2010 - 15:55

graziano

Joined: 01/14/2010

Hello, every user has a

Hello,

every user has a security group called 'default': if the admin modify her default group, those changes won't be propagate to any user group. So every user needs to add the needed rules to access the instances. Please modify the rules on every user and retry to access the instances.

You have PUBLIC_IPS in the 10.x.x.x range, but your cloud controller has yet another public IP: when you mentioned that the instances weren't accessible, where was the client machine residing (on which subnet)? Instances will not be reachable from the public network, but they should be reachable from the 10.x.x.x network.

cheers
graziano

Fri, 07/30/2010 - 20:47

Graziano, Thanks for

Graziano,
Thanks for reply.

Users did use euca-authorize to set the allow rules for their "default", but that did not make their instances accessible even by themselves.

My 10.x.x.x subnet is accessible from our corporate net, so any client machine in the corporate net can access any 10.x.x.x ip. The VM instances run by users using the "administrator's" credential can be accessed from any client in corporate net with no problem, the problem just happened to individual user who downloaded his/her own x509 credential from the URL are having problem access their VM instances until I do the iptables rules manually on clc.
Any suggestion as to where to look into?

Thanks
larryaero

Tue, 08/10/2010 - 10:58

Hello larryaero, thanks for

Hello larryaero,

thanks for the clafrification. An interesting issue indeed: there shouldn't be anything particurlar about the admin account to warrant this behavior. Can you post the output of euca-describe-groups for both the admin and the user? Then can you post the output of euca-describe-instances for both the admin and the user pointing out the instance that is reachable and the instance that is not? Finally, again with the running instances, can you post iptables -L -n and iptables -L -t nat and ip addr show on the CC machine?

Thu, 08/12/2010 - 18:16

Graziano, Thanks for helping

Graziano,
Thanks for helping on this.
I've removed all regular user groups and just use admin account and have the cloud going. So, at least this is no longer an issue here. Basically, I still have 32 IPs per net set, and as admin, I added 7 more groups for 1 class C, then just run instances with -g to make sure all instances got proper group authorization. Right now I just filled up this class C, I'll added one more class C and will try to have user apply account from the URL and check whether the issue re-surface.

Larryaero