Logging Architecture | Kubernetes - 0 views
-
Application logs can help you understand what is happening inside your application
-
container engines are designed to support logging.
-
The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams.
- ...26 more annotations...
-
In a cluster, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called cluster-level logging.
-
Cluster-level logging architectures require a separate backend to store, analyze, and query logs
-
Kubernetes does not provide a native storage solution for log data.
-
use kubectl logs --previous to retrieve logs from a previous instantiation of a container.
-
A container engine handles and redirects any output generated to a containerized application's stdout and stderr streams
-
The Docker JSON logging driver treats each line as a separate message.
-
By default, if a container restarts, the kubelet keeps one terminated container with its logs.
-
An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node
-
You can also set up a container runtime to rotate an application's logs automatically.
-
The two kubelet flags container-log-max-size and container-log-max-files can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
-
The kubelet and container runtime do not run in containers.
-
On machines with systemd, the kubelet and container runtime write to journald. If systemd is not present, the kubelet and container runtime write to .log files in the /var/log directory.
-
System components inside containers always write to the /var/log directory, bypassing the default logging mechanism.
-
Kubernetes does not provide a native solution for cluster-level logging
-
Use a node-level logging agent that runs on every node.
-
implement cluster-level logging by including a node-level logging agent on each node.
-
the logging agent is a container that has access to a directory with log files from all of the application containers on that node.
-
the logging agent must run on every node, it is recommended to run the agent as a DaemonSet
-
Node-level logging creates only one agent per node and doesn't require any changes to the applications running on the node.
-
Containers write stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation.
-
Each sidecar container prints a log to its own stdout or stderr stream.
-
It is not recommended to write log entries with different formats to the same log stream
-
writing logs to a file and then streaming them to stdout can double disk usage.
-
If you have an application that writes to a single file, it's recommended to set /dev/stdout as the destination
-
it's recommended to use stdout and stderr directly and leave rotation and retention policies to the kubelet.
-
Using a logging agent in a sidecar container can lead to significant resource consumption. Moreover, you won't be able to access those logs using kubectl logs because they are not controlled by the kubelet.
