Nomad setup for multi node development server

Nov 13, 2021 · 4 min read · Nomad Containers Orchestration Tool

Overview

Nomad setup with standalone Nomad Server with multiple worker nodes without use of Consul.

Let see at high level, what use cases does Nomad support.

Simple Container Orchestration (Easier to manage containers than k8s in my opinion)
Non-Container based Application Workload Orchestration (This would really excite many organizations that wanted to move quickly to cloud)

Resource Requirements for this setup

3 VMs or 3 LXC (Choice is yours) , you may setup this on your workstation/laptop or you can spin off compute in any cloud provider of your choice. If you want to know more about setting up LXCs ,here is my detailed blog https://livespotty.github.io/kubernetes-the-hard-way-on-lxd/docs/03-compute-resources.html

for this demonstration, I have used Oracle Cloud Infrastructure (OCI), since you get 4 CPU (ARM64) 24 GB always free :)

Note: If you are using any of the cloud providers ensure you open these ports ( 4646,4647,4648,5648 )for inter VM communication. Also linux firewall is disabled.

1sudo ufw status
2
3sudo ufw disable

After creating 3 virtual servers with latest Ubuntu 20.4 OS, download the corresponding version of Nomad under the following link https://www.nomadproject.io/downloads

I have used Nomad v1.1.6. for setting up my development cluster environment.

List of servers and purpose

nomad-server :- This serves as standalone Nomad Control Server that would schedule jobs
nomad-node-1 :- Nomad node, where the jobs will be scheduled based on resource availability
nomad-node-2 :- Second Nomad server node, where the jobs will be scheduled based on resource availability

Step 1:

Create data directory, say "/opt/nomad/data" on each server machines (all three)

Step 2:

Copy the following configuration file to "nomad-nerver" (say, if your server private IP is 10.0.0.1) to "/opt/nomad/config" directory with the filename server.conf

 1data_dir  = "/opt/nomad/data"
 2
 3bind_addr = "10.0.0.1" # the default
 4
 5advertise {
 6  # Defaults to the first private IP address.
 7  http = "10.0.0.1"
 8  rpc  = "10.0.0.1"
 9  serf = "10.0.0.1:5648" # non-default ports may be specified
10}
11
12server {
13  enabled          = true
14  bootstrap_expect = 1
15}
16# if you enable client and set the value to true, you will be able to schedule jobs on server.
17client {
18  enabled       = false
19}
20
21plugin "raw_exec" {
22  config {
23    enabled = true
24  }
25}
26
27ports {
28  http = 4646
29  rpc  = 4647
30  serf = 4648
31}

Step 3:

Log on to "nomad-node-1" (Say your node1 IP is 10.0.0.2) and copy the following node.conf to the location "/opt/nomad/config"

 1data_dir  = "/opt/nomad/data"
 2
 3bind_addr = "10.0.0.2" # the default
 4
 5advertise {
 6  # Defaults to the first private IP address.
 7  http = "10.0.0.2"
 8  rpc  = "10.0.0.2"
 9  serf = "10.0.0.2:5648" # non-default ports may be specified
10}
11
12server {
13  enabled          = false
14  bootstrap_expect = 1
15}
16
17client {
18  enabled       = true
19  servers = ["10.0.0.1:4647"]
20}
21
22plugin "raw_exec" {
23  config {
24    enabled = true
25  }
26}
27
28ports {
29  http = 4646
30  rpc  = 4647
31  serf = 4648
32}

Step 4:

Log on to "nomad-node-2" (Say your node1 IP is 10.0.0.3) and copy the following node.conf to the location "/opt/nomad/config"

 1data_dir  = "/opt/nomad/data"
 2
 3bind_addr = "10.0.0.3" # the default
 4
 5advertise {
 6  # Defaults to the first private IP address.
 7  http = "10.0.0.3"
 8  rpc  = "10.0.0.3"
 9  serf = "10.0.0.3:5648" # non-default ports may be specified
10}
11
12server {
13  enabled          = false
14  bootstrap_expect = 1
15}
16
17client {
18  enabled       = true
19  servers = ["10.0.0.1:4647"]
20}
21
22plugin "raw_exec" {
23  config {
24    enabled = true
25  }
26}
27
28ports {
29  http = 4646
30  rpc  = 4647
31  serf = 4648
32}

Now we have all the configurations ready, now its time to run the nomad process in each server

Step 5:

Ensure you have the correct version of nomad installed, run the following command in "nomad-server"

1sudo nomad agent -config=/opt/nomad/config/server.conf

Step 6:

Run the following command in both the node servers "nomad-node-1" and "nomad-node-2"

1sudo nomad agent -config=/opt/nomad/config/node.conf

How to test the installation

Run the following command and you will see the following output.

1NOMAD_ADDR=http://10.0.0.1:4646 nomad server members
2
3Name                 Address    Port  Status  Leader  Protocol  Build  Datacenter  Region
4nomad-server.global  10.0.0.1   5648  alive   true    2         1.1.6  dc1         global

Testing the node servers

1NOMAD_ADDR=http://10.0.0.1:4646 nomad node status
2
3ID        DC   Name            Class   Drain  Eligibility  Status
48edd9637  dc1  nomad-node-2    <none>  false  eligible     ready
54181323f  dc1  nomad-node-1    <none>  false  eligible     ready

And thats it! you have nomad development cluster up and running. I will cover how to schedule simple jobs on Nomad in my next blog.

For more information, visit https://nomadproject.io