Group items tagged

The control plane's components make global decisions about the cluster

Control plane components can be run on any machine in the cluster.

for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine

The API server is the front end for the Kubernetes control plane.

kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances.

Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for those data.

watches for newly created Pods with no assigned node, and selects a node for them to run on.

Factors taken into account for scheduling decisions include: individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, and deadlines.

each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

Node controller

Job controller

Endpoints controller

Service Account & Token controllers

The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.

If you are running Kubernetes on your own premises, or in a learning environment inside your own PC, the cluster does not have a cloud controller manager.

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.

Kubernetes supports several container runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface).

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features

all Kubernetes clusters should have cluster DNS,

Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.

Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.

A cluster-level logging mechanism is responsible for saving container logs to a central log store with search/browsing interface.

it is nearly the same thing as an image, except that the top layer is read-write

A running container is defined as a read-write "union view" and

The processes within this process-space may change, delete or create files within the "union view" file that will be captured in the read-write layer

there is no longer a running container

each layer contains a pointer to a parent layer using the Id

The 'docker create' command adds a read-write layer to the top stack based on the image id.  It does not run this container.

The command 'docker start' creates a process space around the union view of the container's layers.

the docker run command starts with an image, creates a container, and starts the container

'git pull' (which is a combination of 'git fetch' and 'git merge')

'docker ps' lists out the inventory of running containers on your system

'docker ps -a' where the 'a' is short for 'all' lists out all the containers on your system, whether stopped or running.

Only those images that have containers attached to them or that have been pulled are considered top-level.

'docker stop' issues a SIGTERM to a running container which politely stops all the processes in that process-space.

results is a normal, but non-running, container

'docker kill' issues a non-polite SIGKILL command to all the processes in a running container.

'docker stop' and 'docker kill' which send actual UNIX signals to a running process

'docker pause' uses a special cgroups feature to freeze/pause a running process-space

'docker rm' removes the read-write layer that defines a container from your host system

'docker rmi' removes the read-layer that defines a "union view" of an image.

'docker commit' takes a container's top-level read-write layer and burns it into a read-only layer.

turns a container (whether running or stopped) into an immutable image

uses the FROM directive in the Dockerfile file as the starting image and iteratively 1) runs (create and start) 2) modifies and 3) commits.

'docker exec' command runs on a running container and executes a process in that running container's process space

'docker inspect' fetches the metadata that has been associated with the top-layer of the container or image

'docker save' creates a single tar file that can be used to import on a different host system

only be run on an image

'docker export' command creates a tar file of the contents of the "union view" and flattens it for consumption for non-Docker usages

'docker history' command takes an image-id and recursively prints out the read-only layers

A chart is organized as a collection of files inside of a directory.

values.yaml # The default configuration values for this chart

charts/ # A directory containing any charts upon which this chart depends.

version: A SemVer 2 version (required)

apiVersion: The chart API version, always "v1" (required)

Every chart must have a version number. A version must follow the SemVer 2 standard.

When generating a package, the helm package command will use the version that it finds in the Chart.yaml as a token in the package name.

the appVersion field is not related to the version field. It is a way of specifying the version of the application.

appVersion: The version of the app that this contains (optional). This needn't be SemVer.

If the latest version of a chart in the repository is marked as deprecated, then the chart as a whole is considered to be deprecated.

deprecated: Whether this chart is deprecated (optional, boolean)

one chart may depend on any number of other charts.

the preferred method of declaring dependencies is by using a requirements.yaml file inside of your chart.

A requirements.yaml file is a simple file for listing your dependencies.

The repository field is the full URL to the chart repository.

helm dependency update and it will use your dependency file to download all the specified charts into your charts/ directory for you.

When helm dependency update retrieves charts, it will store them as chart archives in the charts/ directory.

All charts are loaded by default.

The condition field holds one or more YAML paths (delimited by commas). If this path exists in the top parent’s values and resolves to a boolean value, the chart will be enabled or disabled based on that boolean value.

The tags field is a YAML list of labels to associate with this chart.

all charts with tags can be enabled or disabled by specifying the tag and a boolean value.

The --set parameter can be used as usual to alter tag and condition values.

The first condition path that exists wins and subsequent ones for that chart are ignored.

The keys containing the values to be imported can be specified in the parent chart’s requirements.yaml file using a YAML list. Each item in the list is a key which is imported from the child chart’s exports field.

specifying the key data in our import list, Helm looks in the exports field of the child chart for data key and imports its contents.

To access values that are not contained in the exports key of the child chart’s values, you will need to specify the source key of the values to be imported (child) and the destination path in the parent chart’s values (parent).

To drop a dependency into your charts/ directory, use the helm fetch command

A dependency can be either a chart archive (foo-1.2.3.tgz) or an unpacked chart directory.

a single release is created with all the objects for the chart and its dependencies.

When Helm renders the charts, it will pass every file in that directory through the template engine.

Chart users may supply a YAML file that contains values. This can be provided on the command line with helm install.

Template files follow the standard conventions for writing Go templates

{{default "minio" .Values.storage}}

Values that are supplied via a values.yaml file (or via the --set flag) are accessible from the .Values object in a template.

Release.Name: The name of the release (not the chart)

Release.IsUpgrade: This is set to true if the current operation is an upgrade or rollback.

Chart: The contents of the Chart.yaml

Files: A map-like object containing all non-special files in the chart.

Files can be accessed using {{index .Files "file.name"}} or using the {{.Files.Get name}} or {{.Files.GetString name}} functions.

access the contents of the file as []byte using {{.Files.GetBytes}}

Any unknown Chart.yaml fields will be dropped

Chart.yaml cannot be used to pass arbitrarily structured data into the template.

A values file is formatted in YAML.

A chart may include a default values.yaml file

accessible inside of templates using the .Values object

Values files can declare values for the top-level chart, as well as for any of the charts that are included in that chart’s charts/ directory.

Charts at a higher level have access to all of the variables defined beneath.

lower level charts cannot access things in parent charts

Values are namespaced, but namespaces are pruned.

Helm supports special “global” value.

a way of sharing one top-level variable with all subcharts, which is useful for things like setting metadata properties like labels.

global variables of parent charts take precedence over the global variables from subcharts.

helm lint

A chart repository is an HTTP server that houses one or more packaged charts

Any HTTP server that can serve YAML files and tar files and can answer GET requests can be used as a repository server.

Helm does not provide tools for uploading charts to remote repository servers.

the only way to add a chart to $HELM_HOME/starters is to manually copy it there.

Helm provides a hook mechanism to allow chart developers to intervene at certain points in a release’s life cycle.

Hooks are declared as an annotation in the metadata section of a manifest

pre-install

post-install: Executes after all resources are loaded into Kubernetes

pre-delete

post-delete: Executes on a deletion request after all of the release’s resources have been deleted.

pre-upgrade

post-upgrade

pre-rollback

post-rollback: Executes on a rollback request after all resources have been modified.

crd-install

test-success: Executes when running helm test and expects the pod to return successfully (return code == 0).

test-failure: Executes when running helm test and expects the pod to fail (return code != 0).

Hooks allow you, the chart developer, an opportunity to perform operations at strategic points in a release lifecycle

Tiller then loads the hook with the lowest weight first (negative to positive)

Tiller returns the release name (and other data) to the client

If the resources is a Job kind, Tiller will wait until the job successfully runs to completion.

good practice to add a hook weight, and set it to 0 if weight is not important.

leave the hook resource alone.

To destroy such resources, you need to either write code to perform this operation in a pre-delete or post-delete hook or add "helm.sh/hook-delete-policy" annotation to the hook template file.

Hooks are just Kubernetes manifest files with special annotations in the metadata section

One resource can implement multiple hooks

no limit to the number of different resources that may implement a given hook.

Hook weights can be positive or negative numbers but must be represented as strings.

sort those hooks in ascending order.

Hook deletion policies

"before-hook-creation" specifies Tiller should delete the previous hook before the new hook is launched.

By default Tiller will wait for 60 seconds for a deleted hook to no longer exist in the API server before timing out.

Custom Resource Definitions (CRDs) are a special kind in Kubernetes.

The crd-install hook is executed very early during an installation, before the rest of the manifests are verified.

A common reason why the hook resource might already exist is that it was not deleted following use on a previous install/upgrade.

Helm uses Go templates for templating your resource files.

two special template functions: include and required

include function allows you to bring in another template, and then pass the results to other template functions.

The required function allows you to declare a particular values entry as required for template rendering.

Quote Strings, Don’t Quote Integers

when working with integers do not quote the values

env variables values which are expected to be string

to include a template, and then perform an operation on that template’s output, Helm has a special include function

The above includes a template called toYaml, passes it $value, and then passes the output of that template to the nindent function.

Go provides a way for setting template options to control behavior when a map is indexed with a key that’s not present in the map

The required function gives developers the ability to declare a value entry as required for template rendering.

The tpl function allows developers to evaluate strings as templates inside a template.

Rendering a external configuration file

(.Files.Get "conf/app.conf")

Image pull secrets are essentially a combination of registry, username, and password.

Automatically Roll Deployments When ConfigMaps or Secrets change

configmaps or secrets are injected as configuration files in containers

a restart may be required should those be updated with a subsequent helm upgrade

The sha256sum function can be used to ensure a deployment’s annotation section is updated if another file changes

helm upgrade --recreate-pods

"helm.sh/resource-policy": keep

resources that should not be deleted when Helm runs a helm delete

create some reusable parts in your chart

In the templates/ directory, any file that begins with an underscore(_) is not expected to output a Kubernetes manifest file.

The current best practice for composing a complex application from discrete parts is to create a top-level umbrella chart that exposes the global configurations, and then use the charts/ subdirectory to embed each of the components.

SAP’s Converged charts: These charts install SAP Converged Cloud a full OpenStack IaaS on Kubernetes. All of the charts are collected together in one GitHub repository, except for a few submodules.

Deis’s Workflow: This chart exposes the entire Deis PaaS system with one chart. But it’s different from the SAP chart in that this umbrella chart is built from each component, and each component is tracked in a different Git repository.

YAML is a superset of JSON

any valid JSON structure ought to be valid in YAML.

As a best practice, templates should follow a YAML-like syntax unless the JSON syntax substantially reduces the risk of a formatting issue.

There are functions in Helm that allow you to generate random data, cryptographic keys, and so on.

a chart repository is a location where packaged charts can be stored and shared.

A chart repository is an HTTP server that houses an index.yaml file and optionally some packaged charts.

Because a chart repository can be any HTTP server that can serve YAML and tar files and can answer GET requests, you have a plethora of options when it comes down to hosting your own chart repository.

It is not required that a chart package be located on the same server as the index.yaml file.

A valid chart repository must have an index file. The index file contains information about each chart in the chart repository.

The Helm project provides an open-source Helm repository server called ChartMuseum that you can host yourself.

$ helm repo index fantastic-charts --url https://fantastic-charts.storage.googleapis.com

A repository will not be added if it does not contain a valid index.yaml

add the repository to their helm client via the helm repo add [NAME] [URL] command with any name they would like to use to reference the repository.

Helm has provenance tools which help chart users verify the integrity and origin of a package.

Integrity is established by comparing a chart to a provenance record

The provenance file contains a chart’s YAML file plus several pieces of verification information

Chart repositories serve as a centralized collection of Helm charts.

Chart repositories must make it possible to serve provenance files over HTTP via a specific request, and must make them available at the same URI path as the chart.

We don’t want to be “the certificate authority” for all chart signers. Instead, we strongly favor a decentralized model, which is part of the reason we chose OpenPGP as our foundational technology.

The Keybase platform provides a public centralized repository for trust information.

A chart contains a number of Kubernetes resources and components that work together.

A test in a helm chart lives under the templates/ directory and is a pod definition that specifies a container with a given command to run.

The pod definition must contain one of the helm test hook annotations: helm.sh/hook: test-success or helm.sh/hook: test-failure

helm test

nest your test suite under a tests/ directory like <chart-name>/templates/tests/

deployment.yaml: A basic manifest for creating a Kubernetes deployment

using the suffix .yaml for YAML files and .tpl for helpers.

helm get manifest

The helm get manifest command takes a release name (full-coral) and prints out all of the Kubernetes resources that were uploaded to the server. Each file begins with --- to indicate the start of a YAML document

Names should be unique to a release

The name: field is limited to 63 characters because of limitations to the DNS system.

release names are limited to 53 characters

{{ .Release.Name }}

The values that are passed into a template can be thought of as namespaced objects, where a dot (.) separates each namespaced element.

The Release object is one of the built-in objects for Helm

Using --dry-run will make it easier to test your code, but it won’t ensure that Kubernetes itself will accept the templates you generate.

Objects are passed into a template from the template engine.

create new objects within your templates

Objects can be simple, and have just one value. Or they can contain other objects or functions.

Release is one of the top-level objects that you can access in your templates.

Release.Namespace: The namespace to be released into (if the manifest doesn’t override)

Chart: The contents of the Chart.yaml file.

Files: This provides access to all non-special files in a chart.

Files.Get is a function for getting a file by name

Files.GetBytes is a function for getting the contents of a file as an array of bytes instead of as a string. This is useful for things like images.

Template: Contains information about the current template that is being executed

BasePath: The namespaced path to the templates directory of the current chart

Go’s naming convention

use only initial lower case letters in order to distinguish local names from those built-in.

If this is a subchart, the values.yaml file of a parent chart

Individual parameters passed with --set

While structuring data this way is possible, the recommendation is that you keep your values trees shallow, favoring flatness.

If you need to delete a key from the default values, you may override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.

Kubernetes would then fail because you can not declare more than one livenessProbe handler.

When injecting strings from the .Values object into the template, we ought to quote these strings.

quote

Template functions follow the syntax functionName arg1 arg2...

While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates.

Drawing on a concept from UNIX, pipelines are a tool for chaining together a series of template commands to compactly express a series of transformations.

pipelines are an efficient way of getting several things done in sequence

The repeat function will echo the given string the given number of times

default DEFAULT_VALUE GIVEN_VALUE. This function allows you to specify a default value inside of the template, in case the value is omitted.

all static default values should live in the values.yaml, and should not be repeated using the default command

Operators are implemented as functions that return a boolean value.

To use eq, ne, lt, gt, and, or, not etcetera place the operator at the front of the statement followed by its parameters just as you would a function.

if and

if or

with to specify a scope

range, which provides a “for each”-style loop

block declares a special kind of fillable template area

A pipeline is evaluated as false if the value is: a boolean false a numeric zero an empty string a nil (empty or null) an empty collection (map, slice, tuple, dict, array)

incorrect YAML because of the whitespacing

When the template engine runs, it removes the contents inside of {{ and }}, but it leaves the remaining whitespace exactly as is.

{{- (with the dash and space added) indicates that whitespace should be chomped left, while -}} means whitespace to the right should be consumed.

Newlines are whitespace!

an * at the end of the line indicates a newline character that would be removed

the indent function

Scopes can be changed. with can allow you to set the current scope (.) to a particular object.

range

The range function will “range over” (iterate through) the pizzaToppings list.

Just like with sets the scope of ., so does a range operator.

The toppings: |- line is declaring a multi-line string.

not a YAML list. It’s a big string.

the data in ConfigMaps data is composed of key/value pairs, where both the key and the value are simple strings.

The |- marker in YAML takes a multi-line string.

range can be used to iterate over collections that have a key and a value (like a map or dict).

In Helm templates, a variable is a named reference to another object. It follows the form $name

Variables are assigned with a special assignment operator: :=

{{- $relname := .Release.Name -}}

capture both the index and the value

the integer index (starting from zero) to $index and the value to $topping

For data structures that have both a key and a value, we can use range to get both

one variable that is always global - $ - this variable will always point to the root context.

$.

$.

Helm template language is its ability to declare multiple templates and use them together.

A named template (sometimes called a partial or a subtemplate) is simply a template defined inside of a file, and given a name.

If you declare two templates with the same name, whichever one is loaded last will be the one used.

naming convention is to prefix each defined template with the name of the chart: {{ define "mychart.labels" }}

a human-edited configuration file in the Terraform language native syntax could be partially overridden using a programmatically-generated file in JSON syntax.

Terraform initially skips these override files when loading configuration, and then afterwards processes each one in turn (in lexicographical order).

Over-use of override files hurts readability, since a reader looking only at the original files cannot easily see that some portions of those files have been overridden without consulting all of the override files that are present.

A top-level block in an override file merges with a block in a normal configuration file that has the same block header.

Within a top-level block, an attribute argument within an override block replaces any argument of the same name in the original block.

Within a top-level block, any nested blocks within an override block replace all blocks of the same type in the original block.

The contents of nested configuration blocks are not merged.

If more than one override file defines the same top-level block, the overriding effect is compounded, with later blocks taking precedence over earlier blocks

The settings within terraform blocks are considered individually when merging.

If the required_providers argument is set, its value is merged on an element-by-element basis, which allows an override block to adjust the constraint for a single provider without affecting the constraints for other providers.

A controller tracks at least one Kubernetes resource type.

The controller(s) for that resource are responsible for making the current state come closer to that desired state.

in Kubernetes, a controller will send messages to the API server that have useful side effects.

Built-in controllers manage state by interacting with the cluster API server.

By contrast with Job, some controllers need to make changes to things outside of your cluster.

As long as the controllers for your cluster are running and able to make useful changes, it doesn't matter if the overall state is stable or not.

Kubernetes uses lots of controllers that each manage a particular aspect of cluster state.

a particular control loop (controller) uses one kind of resource as its desired state, and has a different kind of resource that it manages to make that desired state happen.

There can be several controllers that create or update the same kind of object.

On Linux, control groups are used to constrain resources that are allocated to processes.

Both kubelet and the underlying container runtime need to interface with control groups to enforce resource management for pods and containers and set resources such as cpu/memory requests and limits.

When the cgroupfs driver is used, the kubelet and the container runtime directly interface with the cgroup filesystem to configure cgroups.

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager.

Two cgroup managers result in two views of the available and in-use resources in the system.

Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.

The approach to mitigate this instability is to use systemd as the cgroup driver for the kubelet and the container runtime when systemd is the selected init system.

Kubernetes 1.26 defaults to using v1 of the CRI API. If a container runtime does not support the v1 API, the kubelet falls back to using the (deprecated) v1alpha2 API instead.

name-based virtual hosting

Edge routerA router that enforces the firewall policy for your cluster.

A Kubernetes ServiceA way to expose an application running on a set of Pods as a network service. that identifies a set of Pods using labelTags objects with identifying attributes that are meaningful and relevant to users. selectors.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.

Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

Ingress frequently uses annotations to configure some options depending on the Ingress controller,

An optional host.

A list of paths

A backend is a combination of Service and port names

has an associated backend

Both the host and path must match the content of an incoming request before the load balancer directs traffic to the referenced Service.

HTTP (and HTTPS) requests to the Ingress that matches the host and path of the rule are sent to the listed backend.

An Ingress with no rules sends all traffic to a single default backend.

Ingress controllers and load balancers may take a minute or two to allocate an IP address.

A fanout configuration routes traffic from a single IP address to more than one Service, based on the HTTP URI being requested.

nginx.ingress.kubernetes.io/rewrite-target: /

describe ingress

get ingress

Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.

an Ingress resource without any hosts defined in the rules, then any web traffic to the IP address of your Ingress controller can be matched without a name based virtual host being required.

secure an Ingress by specifying a SecretStores sensitive information, such as passwords, OAuth tokens, and ssh keys. that contains a TLS private key and certificate.

An Ingress controller is bootstrapped with some load balancing policy settings that it applies to all Ingress, such as the load balancing algorithm, backend weight scheme, and others.