Setting up a Multi-node DevStack on OpenStack

I recently needed a multi-node DevStack deployment to test a patch across multiple compute hosts. This article documents how I set up it up on an existing OpenStack cloud.

The installation instructions here follow the official instructions for a multi-node lab. If this guide contradicts the official instructions, assume that I’m wrong and the official documentation is correct.

We will set up one control node, an arbitrary number of compute nodes and one jump host to connect to them. The DevStack nodes won’t be reachable from the internet.

The architecture will look something like this:

                     Workstation
                          │
                     Floating IP
                          │
                      Jump Host
                          │
             10.2.0.0/24 private network
        ┌─────────────────+───────...───────┐
        │                 │                 │
   Control Node     Compute Node 1    Compute Node n

Prerequisites

This guide assumes you already have a working OpenStack project, the OpenStack CLI configured locally, and permission to create networks, routers, floating IPs, and instances.

First, we have to set up some basic security rules:

openstack security group create ssh
openstack security group rule create --protocol tcp --dst-port 22 --ingress ssh

openstack security group create icmp
openstack security group rule create --protocol icmp icmp

We also want a key pair with a local SSH key. You can create a separate one. Just keep in mind that you either have to specify that one with -i when you use ssh or configure that in ~/.config/ssh.

openstack keypair create --public-key ~/.ssh/id_ed25519.pub nicolai

Network

openstack network create devstack 
openstack subnet create devstack --network devstack --subnet-range 10.2.0.0/24 --gateway 10.2.0.1 --allocation-pool start=10.2.0.100,end=10.2.0.200 --dns-nameserver 9.9.9.9
openstack router create devstack
openstack router set --external-gateway shared-public-IPv4 devstack
openstack router add subnet devstack devstack

Jump Host

openstack server create \
    --flavor XXS \
    --image 'Ubuntu 24.04 LTS x64' \
    --key-name nicolai \
    --network devstack \
    --security-group default \
    --security-group ssh \
    --security-group icmp \
    devstack-jump

Assign a floating IP to the jump host so we can ssh into it:

openstack server show devstack-jump | grep addresses 
openstack port list | grep ${IP_ADDRESS}
openstack floating ip create shared-public-IPv4 --port ${PORT}

Set Up the Control and Compute Nodes

Choose a flavor with a decent amount of RAM for the control node and the compute nodes. I picked L.mem+ because it has 16 GB. This is not a hard limit but just based on the fact that I used a setup with 16 GB of RAM before. There’s also a good chance that you can use different settings for the control node and the compute nodes. These are the settings I picked:

openstack server create \
    --flavor L.mem+ \
    --image 'Ubuntu 24.04 LTS x64' \
    --key-name nicolai \
    --network devstack \
    --security-group default \
    --security-group ssh \
    --security-group icmp \
    devstack-ctrl

Create compute nodes in the same way, for example devstack-cmp1, devstack-cmp2, and so on.

The following steps to install and configure the nodes are almost the same for the control node and the compute nodes. I’ll explicitly mention when there’s a difference.

Set Up DevStack User

Connect to the node and switch to the root user:

ssh -J ubuntu@${DEVSTACK_JUMP} ubuntu@${DEVSTACK_NODE}
sudo su

Set up stack user:

useradd -s /bin/bash -d /opt/stack -m stack
chmod +x /opt/stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

Continue as the stack user:

su stack
cd

Set up SSH

Add your SSH key to the authorized keys for the stack user:

mkdir ~/.ssh; chmod 700 ~/.ssh
sudo cat /home/ubuntu/.ssh/authorized_keys >> ~/.ssh/authorized_keys

Configure and Install DevStack

First, we need to clone the DevStack’s Git repository.

git clone https://opendev.org/openstack/devstack
cd devstack

Create a local.conf. For the control node it looks like this:

[[local|localrc]]
HOST_IP=10.2.0.100
FLOATING_RANGE=10.3.0.0/24
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=labstack
DATABASE_PASSWORD=supersecret
RABBIT_PASSWORD=supersecret
SERVICE_PASSWORD=supersecret

The HOST_IP is the fixed internal IP address assigned by OpenStack. You can pick anything for FLOATING_RANGE as long as there are no collisions with the nodes’ IP addresses.

You can create random strings for the passwords with

base64 -w0 < /dev/random | head -c32

For the compute nodes it looks like this:

[[local|localrc]]
HOST_IP=10.2.0.101
FLOATING_RANGE=10.3.0.0/24
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=labstack
DATABASE_PASSWORD=supersecret
RABBIT_PASSWORD=supersecret
SERVICE_PASSWORD=supersecret
DATABASE_TYPE=mysql
SERVICE_HOST=10.2.0.100
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
ENABLED_SERVICES=n-cpu,c-vol,placement-client,ovn-controller,ovs-vswitchd,ovsdb-server,q-ovn-metadata-agent
NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_lite.html"
VNCSERVER_LISTEN=$HOST_IP
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN

The HOST_IP is the IP address assigned by OpenStack. The FLOATING_RANGE should match the one on the control node. SERVICE_HOST is the control node’s IP address. The passwords should match the ones from the control node.

After creating local.conf, you can install the DevStack:

./stack.sh

This will take some time. Afterward, double check that everything is working with:

sudo systemctl status "devstack@*"

You can also check the endpoints with:

source /opt/devstack/openrc admin admin
openstack endpoint list

Verify the Deployment

Finally we can make sure that everything works as expected:

source openrc admin admin
openstack compute service list
openstack hypervisor list