Group items tagged

The control plane's components make global decisions about the cluster

Control plane components can be run on any machine in the cluster.

for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine

The API server is the front end for the Kubernetes control plane.

kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances.

Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for those data.

watches for newly created Pods with no assigned node, and selects a node for them to run on.

Factors taken into account for scheduling decisions include: individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, and deadlines.

each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

Node controller

Job controller

Endpoints controller

Service Account & Token controllers

The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.

If you are running Kubernetes on your own premises, or in a learning environment inside your own PC, the cluster does not have a cloud controller manager.

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.

Kubernetes supports several container runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface).

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features

all Kubernetes clusters should have cluster DNS,

Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.

Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.

A cluster-level logging mechanism is responsible for saving container logs to a central log store with search/browsing interface.

To use a secret, a pod needs to reference the secret.

--from-file

You can also create a Secret in a file first, in json or yaml format, and then create that object.

The Secret contains two maps: data and stringData.

Kubernetes automatically creates secrets which contain credentials for accessing the API and it automatically modifies your pods to use this type of secret.

kubectl get and kubectl describe avoid showing the contents of a secret by default.

stringData field is provided for convenience, and allows you to provide secret data as unencoded strings.

where you are deploying an application that uses a Secret to store a configuration file, and you want to populate parts of that configuration file during your deployment process.

a field is specified in both data and stringData, the value from stringData is used.

The keys of data and stringData must consist of alphanumeric characters, ‘-’, ‘_’ or ‘.’.

Newlines are not valid within these strings and must be omitted.

When using the base64 utility on Darwin/macOS users should avoid using the -b option to split long lines.

create a Secret from generators and then apply it to create the object on the Apiserver.

The generated Secrets name has a suffix appended by hashing the contents.

Multiple pods can reference the same secret.

each container needs its own volumeMounts block, but only one .spec.volumes is needed per secret

use .spec.volumes[].secret.items field to change target path of each key:

You can also specify the permission mode bits files part of a secret will have. If you don’t specify any, 0644 is used by default.

Kubelet is checking whether the mounted secret is fresh on every periodic sync.

cache propagation delay depends on the chosen cache type

Multiple pods can reference the same secret.

env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base-64 decoded values of the secret data.

An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry password to the Kubelet so it can pull a private image on behalf of your Pod.

Individual secrets are limited to 1MiB in size.

Kubelet only supports use of secrets for Pods it gets from the API server.

Secrets must be created before they are consumed in pods as environment variables unless they are marked as optional.

Once a pod is scheduled, the kubelet will try to fetch the secret value.

Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret.

volumes: - name: secret-volume secret: secretName: ssh-key-secret

You do not need to escape special characters in passwords from files

make that key begin with a dot

Dotfiles in secret volume

.secret-file

a frontend container which handles user interaction and business logic, but which cannot see the private key;

a signer container that can see the private key, and responds to simple signing requests from the frontend

watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided

watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.

additional precautions with secret objects, such as avoiding writing them to disk where possible.

A secret is only sent to a node if a pod on that node requires it

only the secrets that a pod requests are potentially visible within its containers

each container in a pod has to request the secret volume in its volumeMounts for it to be visible within the container.

In the API server secret data is stored in etcdConsistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

limit access to etcd to admin users

A user who can create a pod that uses a secret can also see the value of that secret.

anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet.

name-based virtual hosting

Edge routerA router that enforces the firewall policy for your cluster.

A Kubernetes ServiceA way to expose an application running on a set of Pods as a network service. that identifies a set of Pods using labelTags objects with identifying attributes that are meaningful and relevant to users. selectors.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.

Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

Ingress frequently uses annotations to configure some options depending on the Ingress controller,

An optional host.

A list of paths

A backend is a combination of Service and port names

has an associated backend

Both the host and path must match the content of an incoming request before the load balancer directs traffic to the referenced Service.

HTTP (and HTTPS) requests to the Ingress that matches the host and path of the rule are sent to the listed backend.

An Ingress with no rules sends all traffic to a single default backend.

Ingress controllers and load balancers may take a minute or two to allocate an IP address.

A fanout configuration routes traffic from a single IP address to more than one Service, based on the HTTP URI being requested.

nginx.ingress.kubernetes.io/rewrite-target: /

describe ingress

get ingress

Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.

an Ingress resource without any hosts defined in the rules, then any web traffic to the IP address of your Ingress controller can be matched without a name based virtual host being required.

secure an Ingress by specifying a SecretStores sensitive information, such as passwords, OAuth tokens, and ssh keys. that contains a TLS private key and certificate.

An Ingress controller is bootstrapped with some load balancing policy settings that it applies to all Ingress, such as the load balancing algorithm, backend weight scheme, and others.

review the controller specific documentation to see how they handle health checks

edit ingress

After you save your changes, kubectl updates the resource in the API server, which tells the Ingress controller to reconfigure the load balancer.

kubectl replace -f on a modified Ingress YAML file.

Node: A worker machine in Kubernetes, part of a cluster.

in most common Kubernetes deployments, nodes in the cluster are not part of the public internet.

Edge router: A router that enforces the firewall policy for your cluster.

Service: A Kubernetes Service that identifies a set of Pods using label selectors.

An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.

The Ingress spec has all the information needed to configure a load balancer or proxy server.

An Ingress with no rules sends all traffic to a single default backend and .spec.defaultBackend is the backend that should handle requests in that case.

If defaultBackend is not set, the handling of requests that do not match any of the rules will be up to the ingress controller

A common usage for a Resource backend is to ingress data to an object storage backend with static assets.

Exact: Matches the URL path exactly and with case sensitivity.

Prefix: Matches based on a URL path prefix split by /. Matching is case sensitive and done on a path element by element basis.

multiple paths within an Ingress will match a request. In those cases precedence will be given first to the longest matching path.

Hosts can be precise matches (for example “foo.bar.com”) or a wildcard (for example “*.foo.com”).

Each Ingress should specify a class, a reference to an IngressClass resource that contains additional configuration including the name of the controller that should implement the class.

secure an Ingress by specifying a Secret that contains a TLS private key and certificate.

The Ingress resource only supports a single TLS port, 443, and assumes TLS termination at the ingress point (traffic to the Service and its Pods is in plaintext).

hosts in the tls section need to explicitly match the host in the rules section.

A controller tracks at least one Kubernetes resource type.

The controller(s) for that resource are responsible for making the current state come closer to that desired state.

in Kubernetes, a controller will send messages to the API server that have useful side effects.

Built-in controllers manage state by interacting with the cluster API server.

By contrast with Job, some controllers need to make changes to things outside of your cluster.

As long as the controllers for your cluster are running and able to make useful changes, it doesn't matter if the overall state is stable or not.

Kubernetes uses lots of controllers that each manage a particular aspect of cluster state.

a particular control loop (controller) uses one kind of resource as its desired state, and has a different kind of resource that it manages to make that desired state happen.

There can be several controllers that create or update the same kind of object.

On Linux, control groups are used to constrain resources that are allocated to processes.

Both kubelet and the underlying container runtime need to interface with control groups to enforce resource management for pods and containers and set resources such as cpu/memory requests and limits.

When the cgroupfs driver is used, the kubelet and the container runtime directly interface with the cgroup filesystem to configure cgroups.

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager.

Two cgroup managers result in two views of the available and in-use resources in the system.

Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.

The approach to mitigate this instability is to use systemd as the cgroup driver for the kubelet and the container runtime when systemd is the selected init system.

Kubernetes 1.26 defaults to using v1 of the CRI API. If a container runtime does not support the v1 API, the kubelet falls back to using the (deprecated) v1alpha2 API instead.

In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service).

If you're able to use Kubernetes APIs for service discovery in your application, you can query the API server for Endpoints, that get updated whenever the set of Pods in a Service changes.

A Service in Kubernetes is a REST object, similar to a Pod.

The name of a Service object must be a valid DNS label name

Kubernetes assigns this Service an IP address (sometimes called the "cluster IP"), which is used by the Service proxies

The default protocol for Services is TCP

As many Services need to expose more than one port, Kubernetes supports multiple port definitions on a Service object. Each port definition can have the same protocol, or a different one.

Because this Service has no selector, the corresponding Endpoints object is not created automatically. You can manually map the Service to the network address and port where it's running, by adding an Endpoints object manually

Endpoint IP addresses cannot be the cluster IPs of other Kubernetes Services

Kubernetes ServiceTypes allow you to specify what kind of Service you want. The default is ClusterIP

ClusterIP: Exposes the Service on a cluster-internal IP.

NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.

LoadBalancer: Exposes the Service externally using a cloud provider's load balancer

ExternalName: Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up.

You can also use Ingress to expose your Service. Ingress is not a Service type, but it acts as the entry point for your cluster.

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767).

The default for --nodeport-addresses is an empty list. This means that kube-proxy should consider all available network interfaces for NodePort.

you need to take care of possible port collisions yourself. You also have to use a valid port number, one that's inside the range configured for NodePort use.

Service is visible as <NodeIP>:spec.ports[*].nodePort and .spec.clusterIP:spec.ports[*].port

NodePort: Exposes the Service on each Node's IP at a static port

In-use IP addresses

run one or two control plane instances per failure zone, scaling those instances vertically first and then scaling horizontally after reaching the point of falling returns to (vertical) scale.

Kubernetes nodes do not automatically steer traffic towards control-plane endpoints that are in the same failure zone

store Event objects in a separate dedicated etcd instance.

start and configure additional etcd instance

Kubernetes resource limits help to minimize the impact of memory leaks and other ways that pods and containers can impact on other components.

Addons' default limits are typically based on data collected from experience running each addon on small or medium Kubernetes clusters.

Many addons scale horizontally - you add capacity by running more pods

The VerticalPodAutoscaler can run in recommender mode to provide suggested figures for requests and limits.

Some addons run as one copy per node, controlled by a DaemonSet: for example, a node-level log aggregator.

VerticalPodAutoscaler is a custom resource that you can deploy into your cluster to help you manage resource requests and limits for pods.

The cluster autoscaler integrates with a number of cloud providers to help you run the right number of nodes for the level of resource demand in your cluster.

The addon resizer helps you in resizing the addons automatically as your cluster's scale changes.

A Pod models an application-specific “logical host”

being executed on the same physical or virtual machine would mean being executed on the same logical host.

The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation

Containers in different Pods have distinct IP addresses and can not communicate by IPC without special configuration. These containers usually communicate with each other via Pod IP addresses.

a Pod is modelled as a group of Docker containers with shared namespaces and shared filesystem volumes

Pods are considered to be relatively ephemeral (rather than durable) entities.

Pods are created, assigned a unique ID (UID), and scheduled to nodes where they remain until termination (according to restart policy) or deletion.

it can be replaced by an identical Pod

When something is said to have the same lifetime as a Pod, such as a volume, that means that it exists as long as that Pod (with that UID) exists.

uses a persistent volume for shared storage between the containers

Pods serve as unit of deployment, horizontal scaling, and replication

flat shared networking space

Containers within the Pod see the system hostname as being the same as the configured name for the Pod.

Volumes enable data to survive container restarts and to be shared among the applications within the Pod.

Individual Pods are not intended to run multiple instances of the same application

The individual containers may be versioned, rebuilt and redeployed independently.

Controllers like StatefulSet can also provide support to stateful Pods.

When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container.

Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server.

grace period

Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers.

When the grace period expires, any processes still running in the Pod are killed with SIGKILL.

By default, all deletes are graceful within 30 seconds.

You must specify an additional flag --force along with --grace-period=0 in order to perform force deletions.

Force deletion of a Pod is defined as deletion of a Pod from the cluster state and etcd immediately.

Processes within the container get almost the same privileges that are available to processes outside a container.

Ephemeral containers differ from other containers in that they lack guarantees for resources or execution, and they will never be automatically restarted, so they are not appropriate for building applications.

distroless images enable you to deploy minimal container images that reduce attack surface and exposure to bugs and vulnerabilities.

Custom objects can contain custom fields. These fields can contain arbitrary JSON.

When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint and delete all custom objects stored in it

CustomResourceDefinitions store validated resource data in the cluster's persistence store, etcd.

By default, all unspecified fields for a custom resource, across all versions, are pruned.

The field json can store any JSON value, without anything being pruned.

Finalizers allow controllers to implement asynchronous pre-delete hooks.

kubeadm uses the network interface associated with the default gateway to set the advertise address for this particular control-plane node's API server. To use a different network interface, specify the --apiserver-advertise-address=<ip-address> argument to kubeadm init

Do not share the admin.conf file with anyone and instead grant users custom permissions by generating them a kubeconfig file using the kubeadm kubeconfig user command.

The token is used for mutual authentication between the control-plane node and the joining nodes. The token included here is secret. Keep it safe, because anyone with this token can add authenticated nodes to your cluster.

You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed.

Make sure that your Pod network plugin supports RBAC, and so do any manifests that you use to deploy it.

The cluster created here has a single control-plane node, with a single etcd database running on it.

The node-role.kubernetes.io/control-plane label is such a restricted label and kubeadm manually applies it using a privileged client after a node has been created.

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

remove the node-role.kubernetes.io/control-plane:NoSchedule taint from any nodes that have it, including the control plane nodes, meaning that the scheduler will then be able to schedule Pods everywhere.