Predictive load balancing
Why use predictive load balancing?
Predictive load balancing is a tool that can be used to avoid overloading hosts with traffic. Consider the case where a popular event starts at a certain time, let’s say 12 PM. A spike in traffic will be routed to the hosts that are streaming the content at 12 PM, most of them starting at low bitrates. A host might have sufficient bandwidth left to take on more clients but when the recently connected clients start ramping up in video quality and increase their bitrate, the host can quickly become overloaded, possibly dropping incoming requests or going offline. Predictive load balancing solves this issue by considering how many times a host recently been redirected to.
Basic configuration
The router allows predicting incoming load for individual hosts. Through the use of the selection input API, the routing engine can predict incoming load by tracking recent host selections and make routing decisions accordingly.
What is considered recent is configurable and defaults to 500 milliseconds:
$ confcli services.routing.tuning.target.recentDurationMilliseconds
{
"recentDurationMilliseconds": 500
}
$ confcli services.routing.tuning.target.recentDurationMilliseconds 1000
services.routing.tuning.target.recentDurationMilliseconds = 1000
{
"tuning": {
"target_recent_duration_milliseconds": 500
}
}
How to use predictive load balancing
The standard Lua library provides four functions for predictive load balancing
that can be used when constructing conditions/weight functions: host_bitrate()
, host_bitrate_custom()
, host_has_bw()
and host_has_bw_custom()
.
All require data to be supplied to the selection input API and apply
only to leaf nodes in the routing tree. In order for predictive load balancing
to work properly the data must be updated at regular intervals. The data needs
to be supplied by the target system.
These functions are suitable to used as host health checks. To configure host health checks, see configuring CDNs and hosts.
Note that host_bitrate()
and host_has_bw()
rely on data supplied by metrics
agents, detailed in Cache hardware metrics: monitoring and routing.
host_bitrate_custom()
and host_has_bw_custom()
rely on
manually supplied selection input data, detailed in selection input API. The
bitrate unit depends on the data submitted to the selection input API.
host_bitrate()
host_bitrate()
returns the predicted bitrate (in megabits per second) of
the host after the recently connected clients start ramping up in streaming
quality. The function accepts up to four arguments:
interface
: The name of the interface to use for bitrate prediction.- Optional
avg_bitrate
: the average bitrate per client, defaults to 6 megabits per second. - Optional
num_routers
: the number of routers that can route to this host, defaults to 1. This is important to accurately predict the incoming load if multiple routers are used. - Optional
host
: The name of the host to use for bitrate prediction. Defaults to the current host if not provided.
Examples of usage:
host_bitrate({interface="eths0"})
host_bitrate({avg_bitrate=1, interface="eths0"})
host_bitrate({num_routers=2, interface="eths0"})
host_bitrate({avg_bitrate=1, num_routers=4, interface="eths0"})
host_bitrate({avg_bitrate=1, num_routers=4, host="custom_host", interface="eths0"})
host_bitrate()
calculates the predicted bitrate as:
predicted_host_bitrate = current_host_bitrate + (recent_connections * avg_bitrate * num_routers)
host_bitrate_custom()
Same functionality as host_bitrate()
but uses a custom selection input
variable as bitrate input instead of accessing hardware metrics. The function
accepts up to three arguments:
custom_bitrate_var
: The name of the selection input variable to be used for accessing current host bitrate.- Optional
avg_bitrate
: seehost_bitrate()
documentation above. - Optional
num_routers
: seehost_bitrate()
documentation above.
host_bitrate_custom({custom_bitrate_var="host1_current_bitrate"})
host_bitrate_custom({avg_bitrate=1, custom_bitrate_var="host1_current_bitrate"})
host_bitrate_custom({num_routers=4, custom_bitrate_var="host1_current_bitrate"})
host_has_bw()
Instead of accessing the predicted bitrate of a host through host_bitrate()
,
host_has_bw()
returns 1 if the host is predicted to have enough
bandwidth left to take on more clients after recent connections ramp up in
bitrate, otherwise it returns 0. The function accepts up to five arguments:
interface
: seehost_bitrate()
documentation above.- Optional
avg_bitrate
: seehost_bitrate()
documentation above. - Optional
num_routers
: seehost_bitrate()
documentation above. - Optional
host
: seehost_bitrate()
documentation above. - Optional
margin
: the bitrate (megabits per second) headroom that should be taken into account during calculation, defaults to 0.
host_has_bw()
returns whether or not the following statement is true:
predicted_host_bitrate + margin < host_bitrate_capacity
Examples of usage:
host_has_bw({interface="eths0"})
host_has_bw({margin=10, interface="eth0"})
host_has_bw({avg_bitrate=1, interface="eth0"})
host_has_bw({num_routers=4, interface="eth0"})
host_has_bw({host="custom_host", interface="eth0"})
host_has_bw_custom()
Same functionality as host_has_bw()
but uses a custom selection input
variable as bitrate and capacity input. The function accepts up to five
arguments:
custom_capacity_var
: the name of the selection input variable to be used for accessing host capacity.custom_bitrate_var
: seehost_bitrate_custom()
documentation- Optional
margin
: seehost_has_bw()
documentation above. above. - Optional
avg_bitrate
: seehost_bitrate()
documentation above. - Optional
num_routers
: seehost_bitrate()
documentation above.
Examples of usage:
host_has_bw_custom({custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
host_has_bw_custom({margin=10, custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
host_has_bw_custom({avg_bitrate=1, custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
host_has_bw_custom({num_routers=4, custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
Examples in routing
When using confcli
it’s not recommended to use host_bitrate()
since it’s not
possible to make comparisons using <
, >
or similar operators. Using
host_has_bw()
makes it easy to construct conditions utilizing predictive load
balancing.
host_has_bw()
and host_has_bw_custom()
are excellent candidates to be used
as health check functions for hosts, see configuring CDNs and hosts for more details.
$ confcli services.routing.rules -w
Running wizard for resource 'rules'
Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string
rules : [
rule can be one of
1: allow
2: consistentHashing
3: contentPopularity
4: deny
5: firstMatch
6: random
7: rawGroup
8: rawHost
9: split
10: weighted
Choose element index or name: weighted
Adding a 'weighted' element
rule : {
name (default: ): weighted_and_has_bw
type (default: weighted):
targets : [
target : {
target (default: ): host1
weight (default: 100):
condition (default: always()): host_has_bw({interface="eths0"})
}
Add another 'target' element to array 'targets'? [y/N]: y
target : {
target (default: ): host2
weight (default: 100): 100
condition (default: always()): host_has_bw({margin=1000, interface="eths0"})
}
Add another 'target' element to array 'targets'? [y/N]: y
target : {
target (default: ): host3
weight (default: 100):
condition (default: always()): host_has_bw({avg_bitrate=1, num_routers=2, margin=5000, interface="eths0"})
}
Add another 'target' element to array 'targets'? [y/N]: n
]
}
Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
"rules": [
{
"name": "weighted_and_has_bw",
"type": "weighted",
"targets": [
{
"target": "host1",
"weight": "100",
"condition": "host_has_bw({interface="eths0"})"
},
{
"target": "host2",
"weight": "100",
"condition": "host_has_bw({margin=1000, interface="eths0"})"
},
{
"target": "host3",
"weight": "100",
"condition": "host_has_bw({avg_bitrate=1, num_routers=2, margin=5000, interface="eths0"})"
}
]
}
]
}
"routing": {
"id": "weighted_and_has_bw",
"member_order": "sorted",
"members": [
{
"id": "1",
"host_id": "host1",
"weight_function": "return host_bitrate({interface="eths0"}) > 0 and 1 or 0"
},
{
"id": "2",
"host_id": "host2",
"weight_function": "return host_bitrate({avg_bitrate=1, interface="eths0"}) > 10000 and 1 or 0"
},
{
"id": "3",
"host_id": "host3",
"weight_function": "return host_has_bw({avg_bitrate=1, num_routers=2, margin=5000, interface="eths0"}) and 1 or 0"
}
],
"weight_function": "return 1"
}