Health check
You can check the health of the server by making a request to /healthz, which
should return a 200 OK response.
Structured Logging
To aid processing, structured logging is available. This makes the logs more
coherent, and more suitable for processing by external tools. The log formats
journald or consolejson will trigger structured logging.
Log messages related to a specific HTTP request include the log field named
request_id which can be used to filter out specific requests.
The log field named topic is always present and is used to separate different
areas of logging. The topics are access, messages, licensing and error.
It is useful when querying the log.
Log formats
When run from the command line the application can log in one of the formats
consolepretty, consolejson or journald depending on the value of the
-logformat parameter. Logging can be disabled by using the value discard.
$ ew-vod2cbm -upstream http://127.0.0.1 -logformat consolejson
consolepretty logs to standard output (stdout) in a human-readable format.
Each log line starts with the date and the log message followed by the context-
dependent log fields:
2022-02-07T11:54:42Z INF Incoming request bytes_out=27592 latency_ms=6.18251 method=GET ...
consolejson logs to stdout using a JSON object for each message:
{level:info,topic:access,time:2022-02-07T12:06:07Z,bytes_out:27592, ...}
When running as a systemd service, it logs directly to journald. In this
case field names are upper case whereas the values keep their format.
Here are a few examples of querying the logs:
$ journalctl -u ew-vodjit TOPIC=access -o verbose
To see entries with a specific request ID:
$ journalctl -u ew-vod2cbm REQUEST_ID=host/B1Po0eYz1d-000006 -o verbose
To see the call stack in the event of a crash:
$ journalctl -u ew-manip TOPIC=error -o verbose
Using the verbose output includes all available name-value fields. Without
this only the log message is printed. The field named JSON contains the
original structured log message as sent by the service. To filter out only
this field --output-fields=JSON can be used, however this requires a systemd
version of at least 236.
$ journalctl -u ew-vod2cbm --output-fields=JSON TOPIC=access
To continuously monitor the service:
$ journalctl -fu ew-vod2cbm
Log levels
The startup log level is set according to the service configuration.
The log level for a running service can be changed by posting
a form-field named level to the loglevel endpoint. For example:
$ curl -F level=debug ew-vod2cbm-server:8090/loglevel
The available log levels and how they relate to systemd priority values is listed below:
Log level Systemd priority
trace 7
debug 7
info 6
warn 4
error 3
fatal 2
panic 0
To e.g. see all error [3] and warning [4] messages using journalctl:
$ journalctl -u ew-vod2cbm PRIORITY=3 PRIORITY=4
Use a GET request to see the current log level:
$ curl ew-vod2cbm-server:8090/loglevel
Messages with higher level than the configured one will not be logged.
Monitoring
The services generate Prometheus data, available at the endpoint: /metrics.
The respective metrics for the services are:
ew-vod2cbm
ew_audio_segment_requests_total
ew_audio_segment_request_duration_milliseconds
ew_video_segment_requests_total
ew_video_segment_request_duration_milliseconds
ew_subs_segment_requests_total
ew_subs_segment_request_duration_milliseconds
ew_segment_times_requests_total
ew_segment_times_request_duration_milliseconds
ew_content_info_requests_total
ew_content_info_request_duration_milliseconds
ew-manip
ew_hls_requests_total
ew_hls_request_duration_milliseconds
ew_dash_requests_total
ew_dash_request_duration_milliseconds
ew-vodjit
ew_cmaf_segment_requests_total
ew_cmaf_segment_request_duration_milliseconds
ew_dat_segment_requests_total
ew_dat_segment_request_duration_milliseconds
ew_content_info_requests_total
ew_content_info_request_duration_milliseconds
ew_manifest_requests_total
ew_manifest_request_duration_milliseconds