The VMware and VirtualBox start virtual machines with NAT networking enabled by default.
If you are using VirtualBox to run the Stackato VM, configuring "Bridged" networking before starting the VM is the simplest way to set it up. If you need to run in "NAT" mode, configure port forwarding as per the VirtualBox documentation.
Bridged networking exposes the VM through the hypervisor host system to your local network. This makes it easier to connect with Stackato servers running on other hosts, but should only be done on trusted networks.
NAT networking exposes the VM only to the host system (unless routing through the host is configured). This is a preferable configuration if you are operating on an untrusted network.
Note
Multiple hosts broadcasting the same name via mDNS on the same network can cause confusion and network problems. If you plan on running multiple Stackato micro clouds on one network:
- Make sure the MAC address of each virtual machine is unique.
- Change the hostname if not already unique, or
- use NAT networking, or
- configure DNS manually.
The default password for the stackato system user is stackato.
This password is changed to match the one set for the first administrative user created in the Management Console. Once you've set up the primary Stackato admin account, use that account's password when logging in to the VM at the command line.
In a Stackato cluster, this change only happens on the node serving the Management Console pages (which could be one of multiple Controller nodes). In this case, it's best to log in to each node in the cluster to change the password manually with the passwd command.
You may want or need to change the hostname of the Stackato system, either to match a DNS record you've created or just to make the system URLs more convenient.
For a micro cloud using multicast DNS, (mDNS) this can be done using the kato node rename command:
$ kato node rename mynewname
This command will change the system hostname in /etc/hostname and /etc/hosts, as well as performing some internal configuration for Stackato such as generating a new server certificate for the Management Console.
mDNS is only supported with ".local" hostnames. If you want to give the VM a canonical hostname on an existing network, configure DNS and disable the 'mdns' role:
$ kato role remove mdns
Note
Stackato takes a while to configure itself at boot (longer at first boot). Check 'kato status' to see that core services are running before executing 'kato node rename ..'.
In a cluster, you may also need to manually modify the /etc/hosts file.
The Stackato micro cloud server is initially set up for DHCP and multicast DNS. This is often sufficient for local testing, but in this configuration is only a single node and can only be privately routed.
As you move toward production use of the server, further configuration of IP addresses and hostnames will therefore be required. A production Stackato server will most likely be a cluster consisting of several nodes, some of them requiring IP addresses and corresponding hostnames.
If your server is to be exposed to the Internet, these addresses must be routable and the hostnames must appear in the global DNS. Even if your server is to be part of a private PaaS for organizational use only, it must still integrate fully with your network services, DHCP and DNS in particular. Finally, in the rare case that such services are not available, the Stackato server can be configured with static IP addresses and hostnames.
Before we examine these scenarios in detail, let's review the separation of roles in a cluster:
Where you configure these hostnames and IP addresses will depend on how you operate your data center network. You will want to confer with your network administrator about this, starting with the MAC address configured for each VM in the hypervisor. If your site supports a significant number of VMs, DHCP may be set up to map MAC addresses to IP addresses in a particular way. For example, a certain range of MAC adddresses may be used for servers in the DMZ, and another range for internal servers. If you follow this convention, your Stackato server will obtain an appropriate IP address automatically. DNS at your site may establish a similar convention, which you will want to follow when making any name or address changes within the cluster.
Having determined the hostnames of cluster nodes to be managed by DNS, the hostname on the primary node should be set using kato node rename.
Finally, if you must set a static IP on any cluster node, be sure to test it before making the change permanent, otherwise you may not be able to reach the node once it reboots. Assuming that the primary address is on interface eth0, a secondary address 10.0.0.1/24 could be set up temporarily as follows:
$ ipcalc -nb 10.0.0.1/24
Address: 10.0.0.1
Netmask: 255.255.255.0 = 24
Wildcard: 0.0.0.255
=>
Network: 10.0.0.0/24
HostMin: 10.0.0.1
HostMax: 10.0.0.254
Broadcast: 10.0.0.255
Hosts/Net: 254 Class A, Private Internet
$ sudo ifconfig eth0:1 10.0.0.1 netmask 255.255.255.0 broadcast 10.0.0.255 up
Configure another cluster node using a different address on the same subnet, and be sure that ping works correctly on the new addresses. You should also use this opportunity to ping the router and DNS server for this subnet. Check with your network administrator for their addresses.
The easiest way to configure a Stackato VM with a static IP address is to use the kato op static_ip command.
This command will prompt for the following inputs:
kato will verify the IP addresses given are within legal ranges, automatically calculate the network / broadcast addresses for you, and prompt for the 'sudo' password to write the changes.
As a precaution, the command does not automatically restart networking services. To do so, run the following commands:
$ sudo /etc/init.d/networking restart
Note
If you are setting a new static IP after having configured the VM as a Core node in a cluster, you must run the kato node migrate command on each Stackato node to reset the MBUS_IP for the cluster.
Alternatively, these changes could be made by editing the /etc/network/interfaces file manually. For example:
auto eth0
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.0
network 10.0.0.0
broadcast 10.0.0.255
gateway 10.0.0.254
dns-nameservers 10.0.0.252, 10.0.0.253
dns-search example.com, example.org
When DHCP is not used, DNS server IP addresses must be set explicitly using the dns-nameservers directive as shown above. Multiple DNS servers can be specified in a comma separated list.
Note
Stackato clusters running on EC2 will normally be registered with Elastic IP, which will provide local dynamic address and DNS configuration over DHCP while publishing an external static address for the cluster. You do not have to configure the DNS server address in /etc/network/interfaces.
Note that dnsmasq does not necessarily reinitialize on SIGHUP. Therefore, perform the following to reinitialize:
$ sudo /etc/init.d/dnsmasq restart
$ sudo /etc/init.d/networking restart
Or use sudo shutdown -r to exercise a complete restart. Then use ifconfig to check that the interface has been configured, and ping to check routing to other hosts on the subnet and out in the world. Finally, use dig @<DNS SERVER IP> <HOSTNAME> to check that DNS is resolving correctly.
In the event of troubleshooting, you can confirm which DNS servers are being used by dnsmasq by checking the file /var/run/dnsmasq/resolv.conf.
Note
There may be a performance advantage in locally defining a private secondary IP address (RFC 1918) for the controller so that the other nodes can be assured of routing directly to it. See your network administrator for advice on which addresses and subnets are permissible. Once you have this secondary address set up, see the /etc/hosts section for final configuration of the server.
The /etc/hosts file is used to resolve certain essential or local hostnames without calling upon the DNS. Unless you need to change the local hostname, you will in general not have to edit /etc/hosts manually, but when troubleshooting network issues it never hurts to verify that the file is configured correctly.
As well, various components in a Cluster rely on finding the cluster nodes in /etc/hosts: the Cloud Controller and the RabbitMQ service in particular.
Stackato will automatically configure /etc/hosts on the virtual machine with one entry for the localhost loopback address and another for the RFC 1918 private IP address of the cluster's Primary node, for example "10.0.0.1" or "192.168.0.1". All communication between cluster nodes should be strictly through their private IP addresses and not on routable addresses provided by the DNS.
Remember that /etc/hosts does not support wildcards. You must use some form of DNS for that.
Consider a Stackato instance called stackato-test in domain example.com. The following example is what you should expect to see on a micro cloud installation, where all roles are running on the same node:
$ hostname
stackato-test
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 08:00:27:fc:1c:f6
inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fefc:1cf6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:875142 errors:0 dropped:0 overruns:0 frame:0
TX packets:106777 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:191340039 (191.3 MB) TX bytes:23737389 (23.7 MB)
$ cat /etc/hosts
127.0.0.1 localhost stackato-test
10.0.0.1 stackato-test.example.com api.stackato-test.example.com
On a cluster installation, the IP address in /etc/hosts will identify the node hosting the MBUS, usually the same as the Cloud Controller. On this node, you will see a correspondence between the network interface eth0 address and /etc/hosts as in the above example. On each of the other nodes in the cluster, for example DEA and Stager nodes, eth0 will be configured with its own address on the same subnet, but /etc/hosts will remain the same..
If modifying /etc/hosts becomes necessary because of a hostname change, you can simply edit it as in the following example:
$ sudo vi /etc/hosts
The Stackato micro cloud server is configured to support multicast DNS. The mDNS/DNS-SD broadcast address (stackato-xxxx.local) is intended for private micro clouds running on a local machine or a local subnet.
For production use, the server will need:
The wildcard is particularly important for locating your installed apps, because each will run in its own container for privilege separation, and each container is given a separate hostname.
Detailed instructions for configuring DNS on various networks is beyond the scope of this guide, but in essense the resulting DNS zone file should contain something similar to:
stackato-test.example.com IN A 10.0.0.1
*.stackato-test.example.com IN CNAME stackato-test.example.com
If you intend to expose your applications at URLs other than this wildcard through the use of stackato map or stackato push --url then you will want to add the name information to the DNS zone file as well. Firewalls and load balancers at your site may require corresponding adjustments:
app.domain.com IN CNAME stackato-test.example.com
Note
If your site uses DHCP to assign an IP address to the Stackato server, you will want to configure the DHCP server to assign a fixed address corresponding with the MAC address of the virtual machine. Be careful not to change the MAC address accidentally through the hypervisor. If the Stackato server is hosted on a cloud provider, arrange to use a fixed IP address if possible. For example, Amazon EC2 supports this with Elastic IP.
With DNS records in place, the multicast DNS broadcast is no longer necessary. To turn it off on the Stackato server, use the command:
$ kato role remove mdns
If you don't have access to a DNS server, you can use a dynamic DNS provider, such as ChangeIP and others, to provide DNS records. You will need one that provides wildcard subdomain assignment.
Before registering your domain, be sure that your mail server will accept email from the provider (for example support@changeip.com).
Create an account, choose a subdomain, and ensure that a wildcard assignment is made on the subdomain to handle api and related application subdomains. Then wait to receive the authorization email, and verify the zone transfer before proceeding.
For situations where mDNS will not work (e.g. running in a cloud hosting environment or connecting from a Windows system without mDNS support) but which do not merit the effort of manually configuring a DNS record (e.g. a test server) alternative methods are available.
The quickest way to get wildcard DNS resolution is to use the xip.io service.
Change your hostname using kato node rename to match the external IP address with the 'xip.io' domain appended. For example:
$ kato node rename 10.9.8.7.xip.io
This will change the system hostname and reconfigure some internal Stackato settings. The xip.io DNS servers will resolve the domain '10.9.8.7.xip.io' and all sub-domains to '10.9.8.7'. This works for private subnets as well as public IP addresses.
Locally, you can run dnsmasq as a simple DNS proxy which resolves wildcards for *.stackato-test.example.com to 10.9.8.7 when line such as the following is present in any of its configuration files:
address = /.stackato-test.example.com/ 10.9.8.7
You must restart the service to pick up the changed configuration:
$ /etc/init.d/dnsmasq restart
You may need to add site-specific DNS nameservers manually if the Stackato VM or applications running in Stackato containers need to resolve internal hosts using a particular nameserver.
To explicitly add a DNS nameserver to a Stackato VM running under DHCP, edit /etc/dhcp/dhclient.conf and add a line with the DNS server IP. For example:
append domain-name-servers 10.8.8.8;
Reboot to apply the changes.
For Stackato VMs with a static IP, add the nameservers when prompted when running the kato op static_ip command (see Setting a Static IP above).
The Stackato micro cloud runs with the following ports exposed:
Port Type Service 22 tcp ssh 25 tcp smtp 80 tcp http 111 tcp portmapper 111 udp portmapper 443 tcp https 3306 tcp mysql 5432 tcp postgresql 9001 tcp supervisord 12345 tcp netbus
On a production cluster, or a micro cloud running on a cloud hosting provider, only ports 22 (SSH), 80 (HTTPS) and 443 (HTTPS) need to be exposed externally (e.g. for the Router / Core node).
Within the cluster (i.e. behind the firewall), it is advisable to allow communication between the cluster nodes on all ports. This can be done safely by using the security group / security policy tools provided by your hypervisor:
If you wish to restrict ports between some nodes (recommended only if you do not have the option to use security groups), the following summary describes which ports are used by which services:
Port Range Type Config on Inbound Required by 0 - 65535 (all) tcp dea controller stackato ssh 4222 tcp controller all nodes NATS 5454 tcp controller all nodes redis 7000 - 7999 tcp all nodes all nodes kato log tail 8046 tcp controller all nodes doozerd 9001 tcp all nodes controller supervisord 9022 tcp controller dea droplets 9022 tcp dea controller droplets
If you subscribe to the principle of "defense in depth", each node can also be internally firewalled using iptables to apply the above rules.
Comments:
Note
If your network has an HTTP proxy, the stackato client may attempt to use this when connecting to api.stackato-xxxx.local and fail because the changes in /etc/hosts file are not reflected in the proxy. To work around this problem in Windows, enable \*.local in the ProxyOverride registry key HCU/Software/Microsoft/Windows/CurrentVersion/Internet Settings.
In some cases, it may be a requirement that any HTTP request is first handled through an upstream or parent proxy (HTTP requests may not be directly routable otherwise).
In this case it is neccesary to tell Polipo about the proxy so it knows how to handle this correctly.
Open the Polipo config file /etc/polipo/config and add the lines:
parentProxy = <IP>:<PORT>
parentAuthCredentials = "myuser:mypassw"
Then restart Polipo:
$ sudo /etc/init.d/polipo restart
If you are using a SOCKS proxy, edit the file in the same way but with the lines:
socksParentProxy=<IP>:<PORT>
socksProxyType=socks4a | OR | socks5;
Then restart Polipo:
$ sudo /etc/init.d/polipo restart
For log info, any errors reported by Polipo are available on the Stackato server in /var/log/polipo/polipo.log.
Stackato caches all application dependencies that are downloaded by module managers that support the HTTP_PROXY environment variable (e.g. pip, PyPM, PPM, NPM, etc). This is limited to 100MB of in-memory cache.
To disable Stackato from using the local staging proxy cache, remove or comment out the following line from the /lxc/template/rootfs/etc/environment file on all Stager nodes:
HTTP_PROXY=http://localhost:8123
If you have an upstream HTTP proxy that deployed applications and the staging system need to traverse to access the internet, use the kato op upstream_proxy ... command on all DEA and Stager nodes:
$ kato op upstream_proxy set 192.168.0.99:3128
To remove the proxy setting:
$ kato op upstream_proxy delete <proxy_addr>
To set an HTTP proxy exclusively for apps, add an app_http_proxy entry to the /lxc/template/rootfs/etc/environment file on the DEA nodes. For example:
app_http_proxy=http://192.168.0.99:3128
Stackato data services do not offer any built-in redundancy. For business-critical data storage, a high-availability database or cluster is recommended.
To use an external database instead of the data services provided by Stackato, specify the database credentials directly in your application code instead of using the credentials from the VCAP_SERVICES environment variable.
HTTPS mode provides access to the provisioned apps using wild card SSL certificates through the router or Nginx web server.
There are self-signed certificates on the VM to match the default hostname stackato-xxxx.local. These certificates can be found in:
If you change the hostname, you will need to regenerate the certificates or use your own (signed or self-signed) certificate.
For router2g (default router):
On all router nodes, upload your .key file to the /etc/ssl/private/ dirctory and your .crt file to /etc/ssl/certs/. Change the following settings in /s/vcap/stackato-router/config/local.json to point to the new files:
"sslKeyFile": "/etc/ssl/private/example.key",
"sslCertFile": "/etc/ssl/certs/example.crt",
For Nginx (legacy router):
On all router nodes, edit the following block in ~/stackato/etc/nginx/stackato.conf (symlinked to /etc/nginx/sites/enabled/stackato.conf):
server {
listen 443;
server_name *.stackato-xxxx.local;
ssl on;
ssl_certificate /etc/ssl/certs/stackato.crt;
ssl_certificate_key /etc/ssl/private/stackato.key;
Update the server name with current host and domain, then specify your own certificates on the ssl_certificate and ssl_certificate_key lines.
When using a signed certificate for Stackato, the certificates in the chain must be concatenated in a specific order:
For example, to create the final certificate for the chain in Nginx format:
$ sudo su -c "cat /etc/ssl/certs/site.crt /path/to/intermediate.crt /path/to/rootCA.crt > /etc/ssl/certs/stackato.crt"
Once the cert is chained, restart the router processes:
$ kato restart router
Verify that the full chain is being sent by Nginx using openssl. You should see more than one number in the list. For example:
$ openssl s_client -connect api.stacka.to:443
---
Certificate chain
0 s:/C=CA/ST=British Columbia/L=Vancouver/O=ActiveState Software Inc./OU=Stackato/CN=*.stacka.to
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority
You can generate your own self-signed SSL certificate by running the following commands on the Stackato server, substituting "hostname.mydomain.com" with your own details:
$ mkdir ~/hostname.mydomain.com
$ cd ~/hostname.mydomain.com
$ (umask 077 && touch host.key host.cert host.info)
$ openssl genrsa 2048 > host.key
$ openssl req -new -x509 -nodes -sha1 -days 365 -key host.key -multivalue-rdn \
-subj "/C=CA/emailAddress=email@mydomain.com/O=company_name/CN=*.mydomain.com/CN=mydomain.com" \
> host.crt
For specific configurations that can be used in the -subj option, see http://www.openssl.org/docs/apps/req.html.
Following that, run:
$ openssl x509 -noout -fingerprint -text < host.crt > host.info
$ chmod 400 host.key host.crt
Once you have a host.crt and host.key, change the ssl_certificate and ssl_certificate_key lines in stackato.conf to reference them:
# ssl_certificate /etc/ssl/certs/stackato.crt;
ssl_certificate /home/stackato/hostname.mydomain.com/host.crt;
# ssl_certificate_key /etc/ssl/private/stackato.key;
ssl_certificate_key /home/stackato/hostname.mydomain.com/host.key;
With any self-signed SSL certificate, you will get browser warning messages. The certificate will need to be added to the browser exception rules, which you will be prompted to do so when visiting one of your apps via HTTPS for the first time.
Limits for specific users and groups can be set for memory usage, number of apps, number of services, and the ability to run the sudo command within application containers.
You can manage groups, users, and limits in the Users and Groups sections of the Management Console or by using the stackato client administration commands.
The kato data users command is available to import or export user lists in CSV format.
Both individual users and groups can have the use of the sudo command limited (the default is to disallow its use). See the Groups, Users, and Limits documents for details on managing these limits.
Even if sudo is restricted, special access can be given to specific repositories for modules and resources needed during the staging process.
To configure these, modify the allowed_repos: parameter of the cloud_controller.yml file:
allowed_repos:
- "deb mirror://mirrors.ubuntu.com/mirrors.txt natty main restricted universe multiverse"
- "deb mirror://mirrors.ubuntu.com/mirrors.txt natty-updates main restricted universe multiverse"
- "deb http://security.ubuntu.com/ubuntu natty-security main universe"
The file is located on the Stackato server at ~/stackato/vcap/cloud_controller/config/cloud_controller.yml.