650 device

HostIntel(R) Core(TM) i3-4100U CPU @ 1.80GHz8GB RAM2 cores (1 essentially unused)
UCP/AMIntel Core Processor (Haswell) (kvm)1.5GB RAM1 core (shared with PM)
PMIntel Xeon E312xx (Sandy Bridge) (kvm)1GB RAM1 core (shared with UCP)

Load was generated using 2 hosts:
saturn3500 PAX devices and data loggingIntel(R) Core(TM) i7-7560U CPU @ 2.40GHz - 16GB
loadtest1150 PAX devicesIntel(R) Core(TM) i3-4030U CPU @ 1.90GHz - 8GB

Load was generated in two groups started simultaneously:
Group 1150 devices - spaced at 6 second intervalsloadtest1
Group 2500 devices - spaced at 3 second intervalssaturn3

Each PAX is simulated using phantomjs running a script that:

The PM VM has not consumed the entire 1GB allocated to it. The graphs show it slowly approaching this point, but during this load test it only reaches about 800MB. The PM memory usage (inside the VM) indicates that this isn't really 'real' memory use. The PM is using and freeing memory, but the KVM on the UCP is not re-claiming it. At the end of the run only 370MB is used. (note there are several bash prompts consuming quite a bit of memory not shown. These were used for monitoring during the test run.) Presumably it will level off at 1GB. I could not find a way to force KVM to pre-allocate all of a VM's memory. This is apparently supported in newer verions of KVM than we are running. The long duration testing should show it finally topping out at 1GB. Note that during the whole run the PM's python Manager grows only to 32MB and then levels off. The AM's Manager is approximately 20MB (the PM python has more code included to handle the passenger functions).

The passenger devices are added very quickly, this results in the very spiky CPU loading seen particularly in the PM. As more and more devices are added, the PM lags behind in processing the iptables and PaxUpdate messages that the ground system is sending. As the first group of devices finish up (at around 2:40) the load peaks and then drops back down as the device addition rate drops. Additionally the high load causes the snmpd daemon to be quite active. This SNMP activity also causes the python process to do more work, as it is an Agent-X client.

Devices have a ~10 minute idle timeout time after the Simulated Pax device stops loading content. The devices then are idle-timed-out by the ground system. The ground system is doing this in blocks, which cause the staggered decrease in devices seen on the graph. The ground system is not optimized to remove large numbers of devices 'all at once' like this (it uses a different method to clean up at the end of the flight) so it takes longer to remove these simulated devices than it will on a real aircraft.

Load

Devices (from passenger-manager OIDs)
pm_device_total
pm_device_new
pm_device_wait
pm_device_error
pm_device_transparent
0
100
200
300
400
500
600
700
02:30
02:40
02:50
03:00
03:10
03:20
03:30

Monitoring

PM Host CPU Load
cpu_load_1m
cpu_load_5m
cpu_load_15
0
2
4
6
8
10
12
02:30
02:40
02:50
03:00
03:10
03:20
03:30
PM Processes %CPU
io_wait
python3_cpu
nginx_cpu
snmpd_cpu
p4_cpu
0
10
20
30
40
50
02:30
02:40
02:50
03:00
03:10
03:20
03:30
PM Host Memory (kb)
memory_total_k
memory__used_k
memory_free_k
memory_buffers_k
swap_cached
0
200000
400000
600000
800000
1000000
02:30
02:40
02:50
03:00
03:10
03:20
03:30
PM Process Memory (kb)
python3_mem
nginx_mem
snmpd_mem
p4_mem
0
5000
10000
15000
20000
25000
30000
35000
02:30
02:40
02:50
03:00
03:10
03:20
03:30
AM Host CPU Load
cpu_load_1m
cpu_load_5m
cpu_load_15
0.5
1
1.5
2
02:30
02:40
02:50
03:00
03:10
03:20
03:30
AM Processes %CPU
io_wait
python3_cpu
nginx_cpu
snmpd_cpu
qemu_pm_cpu
0
20
40
60
80
02:30
02:40
02:50
03:00
03:10
03:20
03:30
AM Host Memory (kb)
memory_total_k
memory__used_k
memory_free_k
memory_buffers_k
swap_cached
0
500000
1000000
1500000
02:30
02:40
02:50
03:00
03:10
03:20
03:30
AM Process Memory (kb)
python3_mem
nginx_mem
snmpd_mem
qemu_pm_mem
0
200000
400000
600000
800000
02:30
02:40
02:50
03:00
03:10
03:20
03:30