Group items tagged

a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).

Each Container in the Pod must independently specify where to mount each volume

localnfs

cephfs

awsElasticBlockStore

glusterfs

vsphereVolume

An awsElasticBlockStore volume mounts an Amazon Web Services (AWS) EBS Volume into your Pod.

an EBS volume can be pre-populated with data, and that data can be “handed off” between Pods.

create an EBS volume using aws ec2 create-volume

the nodes on which Pods are running must be AWS EC2 instances

EBS only supports a single EC2 instance mounting a volume

check that the size and EBS volume type are suitable for your use!

A cephfs volume allows an existing CephFS volume to be mounted into your Pod.

the contents of a cephfs volume are preserved and the volume is merely unmounted.

have your own Ceph server running with the share exported

The configMap resource provides a way to inject configuration data into Pods

When referencing a configMap object, you can simply provide its name in the volume to reference it

volumeMounts: - name: config-vol mountPath: /etc/config volumes: - name: config-vol configMap: name: log-config items: - key: log_level path: log_level

create a ConfigMap before you can use it.

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node.

By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment.

you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem)

volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {}

An fc volume allows an existing fibre channel volume to be mounted in a Pod.

configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

Flocker is an open-source clustered Container data volume manager. It provides management and orchestration of data volumes backed by a variety of storage backends.

emptyDir

flocker

A flocker volume allows a Flocker dataset to be mounted into a Pod

have your own Flocker installation running

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your Pod.

Using a PD on a Pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1

A glusterfs volume allows a Glusterfs (an open source networked filesystem) volume to be mounted into your Pod.

have your own GlusterFS installation running

a powerful escape hatch for some applications

access to Docker internals; use a hostPath of /var/lib/docker

allowing a Pod to specify whether a given hostPath should exist prior to the Pod running, whether it should be created, and what it should exist as

specify a type for a hostPath volume

the files or directories created on the underlying hosts are only writable by root.

hostPath: # directory location on host path: /data # this field is optional type: Directory

An iscsi volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.

have your own iSCSI server running

A feature of iSCSI is that it can be mounted as read-only by multiple consumers simultaneously.

A local volume represents a mounted local storage device such as a disk, partition or directory.

Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.

PersistentVolume spec using a local volume and nodeAffinity

PersistentVolume volumeMode can now be set to “Block” (instead of the default value “Filesystem”) to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer

An nfs volume allows an existing NFS (Network File System) share to be mounted into your Pod.

have your own NFS server running with the share exported

A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.

PersistentVolumes are a way for users to “claim” durable storage (such as a GCE PersistentDisk or an iSCSI volume) without knowing the details of the particular cloud environment.

A projected volume maps several existing volume sources into the same directory.

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Each projected volume source is listed in the spec under sources

A Container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.

RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed

A secret volume is used to pass sensitive information, such as passwords, to Pods

create a secret in the Kubernetes API before you can use it

StorageOS runs as a Container within your Kubernetes environment, making local or attached storage accessible from any node within the Kubernetes cluster.

StorageOS provides block storage to Containers, accessible via a file system.

A vsphereVolume is used to mount a vSphere VMDK Volume into your Pod.

supports both VMFS and VSAN datastore.

create VMDK using one of the following methods before using with Pod.

share one volume for multiple uses in a single Pod.

volumeMounts: - name: workdir1 mountPath: /logs subPathExpr: $(POD_NAME)

env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name

Use the subPathExpr field to construct subPath directory names from Downward API environment variables

enable the VolumeSubpathEnvExpansion feature gate

emptyDir and hostPath volumes will be able to request a certain amount of space using a resource specification, and to select the type of media to use, for clusters that have several media types.

the Container Storage Interface (CSI) and Flexvolume. They enable storage vendors to create custom storage plugins without adding them to the Kubernetes repository.

Container Storage Interface (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the csi volume type to attach, mount, etc. the volumes exposed by the CSI driver.

This feature requires CSIInlineVolume feature gate to be enabled:--feature-gates=CSIInlineVolume=true

In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the “Types of Volumes” section above.

Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts.

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the Container will be propagated back to the host and to all Containers of all Pods that use the same volume.

Edit your Docker’s systemd service file. Set MountFlags as follows:MountFlags=shared

for a lot of people, the name “Docker” itself is synonymous with the word “container”.

Docker created a very ergonomic (nice-to-use) tool for working with containers – also called docker.

docker is designed to be installed on a workstation or server and comes with a bunch of tools to make it easy to build and run containers as a developer, or DevOps person.

containerd: This is a daemon process that manages and runs containers.

runc: This is the low-level container runtime (the thing that actually creates and runs containers).

Kubernetes includes a component called dockershim, which allows it to support Docker.

Kubernetes will remove support for Docker directly, and prefer to use only container runtimes that implement its Container Runtime Interface.

Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.

Docker images, are actually images packaged in the Open Container Initiative (OCI) format.

CRI is the API that Kubernetes uses to control the different runtimes that create and manage containers.

CRI makes it easier for Kubernetes to use different container runtimes

containerd is a high-level container runtime that came from Docker, and implements the CRI spec

containerd was separated out of the Docker project, to make Docker more modular.

CRI-O is another high-level container runtime which implements the Container Runtime Interface (CRI).

The idea behind the OCI is that you can choose between different runtimes which conform to the spec.

runc is an OCI-compatible container runtime.

A reference implementation is a piece of software that has implemented all the requirements of a specification or standard.

runc provides all of the low-level functionality for containers, interacting with existing low-level Linux features, like namespaces and control groups.

it is nearly the same thing as an image, except that the top layer is read-write

A running container is defined as a read-write "union view" and

The processes within this process-space may change, delete or create files within the "union view" file that will be captured in the read-write layer

there is no longer a running container

each layer contains a pointer to a parent layer using the Id

The 'docker create' command adds a read-write layer to the top stack based on the image id.  It does not run this container.

The command 'docker start' creates a process space around the union view of the container's layers.

the docker run command starts with an image, creates a container, and starts the container

'git pull' (which is a combination of 'git fetch' and 'git merge')

'docker ps' lists out the inventory of running containers on your system

'docker ps -a' where the 'a' is short for 'all' lists out all the containers on your system, whether stopped or running.

Only those images that have containers attached to them or that have been pulled are considered top-level.

'docker stop' issues a SIGTERM to a running container which politely stops all the processes in that process-space.

results is a normal, but non-running, container

'docker kill' issues a non-polite SIGKILL command to all the processes in a running container.

'docker stop' and 'docker kill' which send actual UNIX signals to a running process

'docker pause' uses a special cgroups feature to freeze/pause a running process-space

'docker rm' removes the read-write layer that defines a container from your host system

'docker rmi' removes the read-layer that defines a "union view" of an image.

'docker commit' takes a container's top-level read-write layer and burns it into a read-only layer.

turns a container (whether running or stopped) into an immutable image

uses the FROM directive in the Dockerfile file as the starting image and iteratively 1) runs (create and start) 2) modifies and 3) commits.

'docker exec' command runs on a running container and executes a process in that running container's process space

'docker inspect' fetches the metadata that has been associated with the top-layer of the container or image

'docker save' creates a single tar file that can be used to import on a different host system

only be run on an image

'docker export' command creates a tar file of the contents of the "union view" and flattens it for consumption for non-Docker usages

'docker history' command takes an image-id and recursively prints out the read-only layers

To use a secret, a pod needs to reference the secret.

--from-file

You can also create a Secret in a file first, in json or yaml format, and then create that object.

The Secret contains two maps: data and stringData.

Kubernetes automatically creates secrets which contain credentials for accessing the API and it automatically modifies your pods to use this type of secret.

kubectl get and kubectl describe avoid showing the contents of a secret by default.

stringData field is provided for convenience, and allows you to provide secret data as unencoded strings.

where you are deploying an application that uses a Secret to store a configuration file, and you want to populate parts of that configuration file during your deployment process.

a field is specified in both data and stringData, the value from stringData is used.

The keys of data and stringData must consist of alphanumeric characters, ‘-’, ‘_’ or ‘.’.

Newlines are not valid within these strings and must be omitted.

When using the base64 utility on Darwin/macOS users should avoid using the -b option to split long lines.

create a Secret from generators and then apply it to create the object on the Apiserver.

The generated Secrets name has a suffix appended by hashing the contents.

Multiple pods can reference the same secret.

each container needs its own volumeMounts block, but only one .spec.volumes is needed per secret

use .spec.volumes[].secret.items field to change target path of each key:

You can also specify the permission mode bits files part of a secret will have. If you don’t specify any, 0644 is used by default.

Kubelet is checking whether the mounted secret is fresh on every periodic sync.

cache propagation delay depends on the chosen cache type

Multiple pods can reference the same secret.

env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base-64 decoded values of the secret data.

An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry password to the Kubelet so it can pull a private image on behalf of your Pod.

Individual secrets are limited to 1MiB in size.

Kubelet only supports use of secrets for Pods it gets from the API server.

Secrets must be created before they are consumed in pods as environment variables unless they are marked as optional.

Once a pod is scheduled, the kubelet will try to fetch the secret value.

Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret.

volumes: - name: secret-volume secret: secretName: ssh-key-secret

You do not need to escape special characters in passwords from files

make that key begin with a dot

Dotfiles in secret volume

.secret-file

a frontend container which handles user interaction and business logic, but which cannot see the private key;

a signer container that can see the private key, and responds to simple signing requests from the frontend

watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided

watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.

additional precautions with secret objects, such as avoiding writing them to disk where possible.

A secret is only sent to a node if a pod on that node requires it

only the secrets that a pod requests are potentially visible within its containers

each container in a pod has to request the secret volume in its volumeMounts for it to be visible within the container.

In the API server secret data is stored in etcdConsistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

limit access to etcd to admin users

A user who can create a pod that uses a secret can also see the value of that secret.

anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet.

Cluster-level logging architectures require a separate backend to store, analyze, and query logs

use kubectl logs --previous to retrieve logs from a previous instantiation of a container.

A container engine handles and redirects any output generated to a containerized application's stdout and stderr streams

An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node

You can also set up a container runtime to rotate an application's logs automatically.

The two kubelet flags container-log-max-size and container-log-max-files can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.

The kubelet and container runtime do not run in containers.

On machines with systemd, the kubelet and container runtime write to journald. If systemd is not present, the kubelet and container runtime write to .log files in the /var/log directory.

Kubernetes does not provide a native solution for cluster-level logging

implement cluster-level logging by including a node-level logging agent on each node.

the logging agent must run on every node, it is recommended to run the agent as a DaemonSet

Node-level logging creates only one agent per node and doesn't require any changes to the applications running on the node.

Containers write stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation.

Each sidecar container prints a log to its own stdout or stderr stream.

It is not recommended to write log entries with different formats to the same log stream

writing logs to a file and then streaming them to stdout can double disk usage.

it's recommended to use stdout and stderr directly and leave rotation and retention policies to the kubelet.

The control plane's components make global decisions about the cluster

Control plane components can be run on any machine in the cluster.

for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine

The API server is the front end for the Kubernetes control plane.

kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances.

Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for those data.

watches for newly created Pods with no assigned node, and selects a node for them to run on.

Factors taken into account for scheduling decisions include: individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, and deadlines.

each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

Node controller

Job controller

Endpoints controller

Service Account & Token controllers

The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.

If you are running Kubernetes on your own premises, or in a learning environment inside your own PC, the cluster does not have a cloud controller manager.

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.

Kubernetes supports several container runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface).

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features

all Kubernetes clusters should have cluster DNS,

Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.

Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.

A cluster-level logging mechanism is responsible for saving container logs to a central log store with search/browsing interface.

On Linux, control groups are used to constrain resources that are allocated to processes.

Both kubelet and the underlying container runtime need to interface with control groups to enforce resource management for pods and containers and set resources such as cpu/memory requests and limits.

When the cgroupfs driver is used, the kubelet and the container runtime directly interface with the cgroup filesystem to configure cgroups.

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager.

Two cgroup managers result in two views of the available and in-use resources in the system.

Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.

The approach to mitigate this instability is to use systemd as the cgroup driver for the kubelet and the container runtime when systemd is the selected init system.

Kubernetes 1.26 defaults to using v1 of the CRI API. If a container runtime does not support the v1 API, the kubelet falls back to using the (deprecated) v1alpha2 API instead.

containers: directories containing everything-your-application

images: snapshots of containers or base OS (e.g. Ubuntu) images

Dockerfiles: scripts automating the building process of images

Docker containers are basically directories which can be packed (e.g. tar-archived) like any other, then shared and run across various different machines and platforms (hosts).

Linux Containers can be defined as a combination various kernel-level features (i.e. things that Linux-kernel can do) which allow management of applications (and resources they use) contained within their own environment

Each container is layered like an onion and each action taken within a container consists of putting another block (which actually translates to a simple change within the file system) on top of the previous one.

Each docker container starts from a docker image which forms the base for other applications and layers to come.

Docker images constitute the base of docker containers from which everything starts to form

a solid, consistent and dependable base with everything that is needed to run the applications

As more layers (tools, applications etc.) are added on top of the base, new images can be formed by committing these changes.

a Dockerfile for automated image building

Dockerfiles are scripts containing a successive series of instructions, directions, and commands which are to be executed to form a new docker image.

As you work with a container and continue to perform actions on it (e.g. download and install software, configure files etc.), to have it keep its state, you need to “commit”.

Please remember to “commit” all your changes.

When you "run" any process using an image, in return, you will have a container.

When the process is not actively running, this container will be a non-running container. Nonetheless, all of them will reside on your system until you remove them via rm command.

To create a new container, you need to use a base image and specify a command to run.

you can not change the command you run after having created a container (hence specifying one during "creation")

If you would like to save the progress and changes you made with a container, you can use “commit”

turns your container to an image

A Pod models an application-specific “logical host”

being executed on the same physical or virtual machine would mean being executed on the same logical host.

The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation

Containers in different Pods have distinct IP addresses and can not communicate by IPC without special configuration. These containers usually communicate with each other via Pod IP addresses.

a Pod is modelled as a group of Docker containers with shared namespaces and shared filesystem volumes

Pods are considered to be relatively ephemeral (rather than durable) entities.

Pods are created, assigned a unique ID (UID), and scheduled to nodes where they remain until termination (according to restart policy) or deletion.

it can be replaced by an identical Pod

When something is said to have the same lifetime as a Pod, such as a volume, that means that it exists as long as that Pod (with that UID) exists.

uses a persistent volume for shared storage between the containers

Pods serve as unit of deployment, horizontal scaling, and replication

flat shared networking space

Containers within the Pod see the system hostname as being the same as the configured name for the Pod.

Volumes enable data to survive container restarts and to be shared among the applications within the Pod.

Individual Pods are not intended to run multiple instances of the same application

The individual containers may be versioned, rebuilt and redeployed independently.

Controllers like StatefulSet can also provide support to stateful Pods.

When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container.

Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server.

grace period

Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers.

When the grace period expires, any processes still running in the Pod are killed with SIGKILL.

By default, all deletes are graceful within 30 seconds.

You must specify an additional flag --force along with --grace-period=0 in order to perform force deletions.

Force deletion of a Pod is defined as deletion of a Pod from the cluster state and etcd immediately.

Processes within the container get almost the same privileges that are available to processes outside a container.

Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure.

a VE there is no preloaded emulation manager software as in a VM.

LXC will boast bare metal performance characteristics because it only packages the needed applications.

a VM, which packages the entire OS and machine setup, including hard drive, virtual processors and network interfaces. The resulting bloated mass usually takes a long time to boot and consumes a lot of CPU and RAM.

don’t offer some other neat features of VM’s such as IaaS setups and live migration.

LXC as supercharged chroot on Linux. It allows you to not only isolate applications, but even the entire OS.

Libvirt, which allows the use of containers through the LXC driver by connecting to 'lxc:///'.

'LXC', is not compatible with libvirt, but is more flexible with more userspace tools.

Portable deployment across machines

Versioning: Docker includes git-like capabilities for tracking successive versions of a container

Component reuse: Docker allows building or stacking of already created packages.

Shared libraries: There is already a public registry (http://index.docker.io/ ) where thousands have already uploaded the useful containers they have created.

Docker taking the devops world by storm since its launch back in 2013.

LXC, while older, has not been as popular with developers as Docker has proven to be

LXC having a focus on sys admins that’s similar to what solutions like the Solaris operating system, with its Solaris Zones, Linux OpenVZ, and FreeBSD, with its BSD Jails virtualization system

it started out being built on top of LXC, Docker later moved beyond LXC containers to its own execution environment called libcontainer.

LXC tooling sticks close to what system administrators running bare metal servers are used to

The LXC command line provides essential commands that cover routine management tasks, including the creation, launch, and deletion of LXC containers.

Docker containers aim to be even lighter weight in order to support the fast, highly scalable, deployment of applications with microservice architecture.

With backing from Canonical, LXC and LXD have an ecosystem tightly bound to the rest of the open source Linux community.

Docker Swarm

Docker Trusted Registry

Docker Compose

Docker Machine

Kubernetes facilitates the deployment of containers in your data center by representing a cluster of servers as a single system.

Swarm is Docker’s clustering, scheduling and orchestration tool for managing a cluster of Docker hosts. 

rkt is a security minded container engine that uses KVM for VM-based isolation and packs other enhanced security features. 

Apache Mesos can run different kinds of distributed jobs, including containers. 

Elastic Container Service is Amazon’s service for running and orchestrating containerized applications on AWS

LXC offers the advantages of a VE on Linux, mainly the ability to isolate your own private workloads from one another. It is a cheaper and faster solution to implement than a VM, but doing so requires a bit of extra learning and expertise.

the containers within a pod can communicate with each other over localhost