Group items tagged

The control plane's components make global decisions about the cluster

Control plane components can be run on any machine in the cluster.

for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine

The API server is the front end for the Kubernetes control plane.

kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances.

Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for those data.

watches for newly created Pods with no assigned node, and selects a node for them to run on.

Factors taken into account for scheduling decisions include: individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, and deadlines.

each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

Node controller

Job controller

Endpoints controller

Service Account & Token controllers

The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.

If you are running Kubernetes on your own premises, or in a learning environment inside your own PC, the cluster does not have a cloud controller manager.

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.

Kubernetes supports several container runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface).

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features

all Kubernetes clusters should have cluster DNS,

Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.

Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.

A cluster-level logging mechanism is responsible for saving container logs to a central log store with search/browsing interface.

initialize the local CLI

install Tiller into your Kubernetes cluster

helm install

helm init --upgrade

By default, when Tiller is installed, it does not have authentication enabled.

helm repo update

Without a max history set the history is kept indefinitely, leaving a large number of records for helm and tiller to maintain.

helm init --upgrade

Whenever you install a chart, a new release is created.

helm list function will show you a list of all deployed releases.

helm delete

helm status

the Helm server (Tiller).

The Helm client (helm)

brew install kubernetes-helm

it can also be run locally, and configured to talk to a remote Kubernetes cluster.

Role-Based Access Control - RBAC for short

run Tiller in an RBAC-enabled Kubernetes cluster.

run kubectl get pods --namespace kube-system and see Tiller running.

helm inspect

Helm will look for Tiller in the kube-system namespace unless --tiller-namespace or TILLER_NAMESPACE is set.

For development, it is sometimes easier to work on Tiller locally, and configure it to connect to a remote Kubernetes cluster.

helm version should show you both the client and server version.

The --node-selectors flag allows us to specify the node labels required for scheduling the Tiller pod.

--override allows you to specify properties of Tiller’s deployment manifest.

helm init --override manipulates the specified properties of the final manifest (there is no “values” file).

The --output flag allows us skip the installation of Tiller’s deployment manifest and simply output the deployment manifest to stdout in either JSON or YAML format.

switch from the default backend to the secrets backend, you’ll have to do the migration for this on your own.

a beta SQL storage backend that stores release information in an SQL database (only postgres has been tested so far).

Helm requires that kubelet have access to a copy of the socat program to proxy connections to the Tiller API.

A Release is an instance of a chart running in a Kubernetes cluster. One chart can often be installed many times into the same cluster.

helm init --client-only

helm init --dry-run --debug

A panic in Tiller is almost always the result of a failure to negotiate with the Kubernetes API server

Tiller and Helm have to negotiate a common version to make sure that they can safely communicate without breaking API assumptions

helm delete --purge

Helm stores some files in $HELM_HOME, which is located by default in ~/.helm

A Chart is a Helm package. It contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster.

A Repository is the place where charts can be collected and shared.

Set the $HELM_HOME environment variable

Helm installs charts into Kubernetes, creating a new release for each installation. And to find new charts, you can search Helm chart repositories.

chart repository is named stable by default

helm search shows you all of the available charts

helm inspect

To install a new package, use the helm install command. At its simplest, it takes only one argument: The name of the chart.

If you want to use your own release name, simply use the --name flag on helm install

additional configuration steps you can or should take.

Helm does not wait until all of the resources are running before it exits. Many charts require Docker images that are over 600M in size, and may take a long time to install into the cluster.

helm status

helm inspect values

helm inspect values stable/mariadb

override any of these settings in a YAML formatted file, and then pass that file during installation.

helm install -f config.yaml stable/mariadb

--values (or -f): Specify a YAML file with overrides.

--set (and its variants --set-string and --set-file): Specify overrides on the command line.

Values that have been --set can be cleared by running helm upgrade with --reset-values specified.

Chart designers are encouraged to consider the --set usage when designing the format of a values.yaml file.

--set-file key=filepath is another variant of --set. It reads the file and use its content as a value.

inject a multi-line text into values without dealing with indentation in YAML.

An unpacked chart directory

When a new version of a chart is released, or when you want to change the configuration of your release, you can use the helm upgrade command.

Kubernetes charts can be large and complex, Helm tries to perform the least invasive upgrade.

$ helm upgrade -f panda.yaml happy-panda stable/mariadb

deployment

If both are used, --set values are merged into --values with higher precedence.

The helm get command is a useful tool for looking at a release in the cluster.

A release version is an incremental revision. Every time an install, upgrade, or rollback happens, the revision number is incremented by 1.

helm history

you can rollback a deleted resource, and have it re-activate.

helm repo list

helm repo add

helm repo update

helm create

helm lint

helm package

Charts that are archived can be loaded into chart repositories.

chart repository server

Tiller can be installed into any namespace.

Limiting Tiller to only be able to install into specific namespaces and/or resource types is controlled by Kubernetes RBAC roles and rolebindings

Release names are unique PER TILLER INSTANCE

Charts should only contain resources that exist in a single namespace.

not recommended to have multiple Tillers configured to manage resources in the same namespace.

a client-side Helm plugin. A plugin is a tool that can be accessed through the helm CLI, but which is not part of the built-in Helm codebase.

Helm plugins are add-on tools that integrate seamlessly with Helm. They provide a way to extend the core feature set of Helm, but without requiring every new feature to be written in Go and added to the core tool.

Helm plugins live in $(helm home)/plugins

The Helm plugin model is partially modeled on Git’s plugin model

helm referred to as the porcelain layer, with plugins being the plumbing.

helm plugin install https://github.com/technosophos/helm-template

command is the command that this plugin will execute when it is called.

Environment variables are interpolated before the plugin is executed.

The command itself is not executed in a shell. So you can’t oneline a shell script.

Helm is able to fetch Charts using HTTP/S

Variables like KUBECONFIG are set for the plugin if they are set in the outer environment.

In Kubernetes, granting a role to an application-specific service account is a best practice to ensure that your application is operating in the scope that you have specified.

restrict Tiller’s capabilities to install resources to certain namespaces, or to grant a Helm client running access to a Tiller instance.

Service account with cluster-admin role

The cluster-admin role is created by default in a Kubernetes cluster

Deploy Tiller in a namespace, restricted to deploying resources only in that namespace

Deploy Tiller in a namespace, restricted to deploying resources in another namespace

When running a Helm client in a pod, in order for the Helm client to talk to a Tiller instance, it will need certain privileges to be granted.

SSL Between Helm and Tiller

The Tiller authentication model uses client-side SSL certificates.

creating an internal CA, and using both the cryptographic and identity functions of SSL.

Helm is a powerful and flexible package-management and operations tool for Kubernetes.

default installation applies no security configurations

with a cluster that is well-secured in a private network with no data-sharing or no other users or teams.

With great power comes great responsibility.

Choose the Best Practices you should apply to your helm installation

Role-based access control, or RBAC

Tiller’s gRPC endpoint and its usage by Helm

Kubernetes employ a role-based access control (or RBAC) system (as do modern operating systems) to help mitigate the damage that can be done if credentials are misused or bugs exist.

In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied.

Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.

release information

charts

charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.

Helm’s provenance tools to ensure the provenance and integrity of charts

For synchronization of data in the data nodes, MySQL cluster uses a special data engine called NDB (Network Database).

it uses automatic shrading aka splitting of a large database into small units.

MySQL cluster avoids single point failure and ensures 99.99% availability.

MySQL cluster can provide a response time as low as less than 3 ms.

Galera Cluster consists of a database server and uses the Galera Replication Plugin to manage replication.

a multi-master database cluster that supports synchronous replication.

it provides multiple, up-to-date copies of the data.

there is a need for instant fail-over.

Galera cluster include guaranteed write consistency, automatic node provisioning, etc.

Upon restoring the connection, the separated nodes will sync back and rejoin the cluster automatically.

there is no need to have management node like MySQL cluster.

it gives best results with the InnoDB storage engine.

Kubernetes is fully controlled through this API

the main job of kubectl is to carry out HTTP requests to the Kubernetes API

Kubernetes maintains an internal state of resources, and all Kubernetes operations are CRUD operations on these resources.

Kubernetes is a fully resource-centred system

Kubernetes API reference is organised as a list of resource types with their associated operations.

This is how kubectl works for all commands that interact with the Kubernetes cluster.

kubectl simply makes HTTP requests to the appropriate Kubernetes API endpoints.

it's totally possible to control Kubernetes with a tool like curl by manually issuing HTTP requests to the Kubernetes API.

Kubernetes consists of a set of independent components that run as separate processes on the nodes of a cluster.

components on the master nodes

Storage backend: stores resource definitions (usually etcd is used)

API server: provides Kubernetes API and manages storage backend

Controller manager: ensures resource statuses match specifications

Scheduler: schedules Pods to worker nodes

component on the worker nodes

Kubelet: manages execution of containers on a worker node

triggers the ReplicaSet controller, which is a sub-process of the controller manager.

the scheduler, who watches for Pod definitions that are not yet scheduled to a worker node.

creating and updating resources in the storage backend on the master node.

However, these components do not access the storage backend directly, but only through the Kubernetes API.

All other Kubernetes components and users read, watch, and manipulate the state (i.e. resources) of Kubernetes through the Kubernetes API

The storage backend stores the state (i.e. resources) of Kubernetes.

command completion is a shell feature that works by the means of a completion script.

A completion script is a shell script that defines the completion behaviour for a specific command. Sourcing a completion script enables completion for the corresponding command.

kubectl completion zsh

/etc/bash_completion.d directory (create it, if it doesn't exist)

source <(kubectl completion bash)

source <(kubectl completion zsh)

autoload -Uz compinit compinit

the API reference, which contains the full specifications of all resources.

kubectl api-resources

displays the resource names in their plural form (e.g. deployments instead of deployment). It also displays the shortname (e.g. deploy) for those resources that have one. Don't worry about these differences. All of these name variants are equivalent for kubectl.

.spec

custom columns output format comes in. It lets you freely define the columns and the data to display in them. You can choose any field of a resource to be displayed as a separate column in the output

kubectl get pods -o custom-columns='NAME:metadata.name,NODE:spec.nodeName'

kubectl explain pod.spec.

kubectl explain pod.metadata.

browse the resource specifications and try it out with any fields you like!

with kubectl explain, only a subset of the JSONPath capabilities is supported

Many fields of Kubernetes resources are lists, and this operator allows you to select items of these lists. It is often used with a wildcard as [*] to select all items of the list.

kubectl get pods -o custom-columns='NAME:metadata.name,IMAGES:spec.containers[*].image'

a Pod may contain more than one container.

The availability zones for each node are obtained through the special failure-domain.beta.kubernetes.io/zone label.

kubectl get nodes -o yaml kubectl get nodes -o json

The default kubeconfig file is ~/.kube/config

with multiple clusters, then you have connection parameters for multiple clusters configured in your kubeconfig file.

overwrite the default kubeconfig file with the --kubeconfig option for every kubectl command.

Namespace: the namespace to use when connecting to the cluster

a one-to-one mapping between clusters and contexts.

When kubectl reads a kubeconfig file, it always uses the information from the current context.

just change the current context in the kubeconfig file

kubectl also provides the --cluster, --user, --namespace, and --context options that allow you to overwrite individual elements and the current context itself, regardless of what is set in the kubeconfig file.

for switching between clusters and namespaces is kubectx.

kubectl config get-contexts

just have to download the shell scripts named kubectl-ctx and kubectl-ns to any directory in your PATH and make them executable (for example, with chmod +x)

kubectl proxy

kubectl get roles

kubectl get pod

Kubectl plugins are distributed as simple executable files with a name of the form kubectl-x. The prefix kubectl- is mandatory,

To install a plugin, you just have to copy the kubectl-x file to any directory in your PATH and make it executable (for example, with chmod +x)

krew itself is a kubectl plugin

check out the kubectl-plugins GitHub topic

The executable can be of any type, a Bash script, a compiled Go program, a Python script, it really doesn't matter. The only requirement is that it can be directly executed by the operating system.

kubectl plugins can be written in any programming or scripting language.

you can write more sophisticated plugins with real programming languages, for example, using a Kubernetes client library. If you use Go, you can also use the cli-runtime library, which exists specifically for writing kubectl plugins.

a kubeconfig file consists of a set of contexts

changing the current context means changing the cluster, if you have only a single context per cluster.

name-based virtual hosting

Edge routerA router that enforces the firewall policy for your cluster.

A Kubernetes ServiceA way to expose an application running on a set of Pods as a network service. that identifies a set of Pods using labelTags objects with identifying attributes that are meaningful and relevant to users. selectors.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.

Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

Ingress frequently uses annotations to configure some options depending on the Ingress controller,

An optional host.

A list of paths

A backend is a combination of Service and port names

has an associated backend

Both the host and path must match the content of an incoming request before the load balancer directs traffic to the referenced Service.

HTTP (and HTTPS) requests to the Ingress that matches the host and path of the rule are sent to the listed backend.

An Ingress with no rules sends all traffic to a single default backend.

Ingress controllers and load balancers may take a minute or two to allocate an IP address.

A fanout configuration routes traffic from a single IP address to more than one Service, based on the HTTP URI being requested.

nginx.ingress.kubernetes.io/rewrite-target: /

describe ingress

get ingress

Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.

an Ingress resource without any hosts defined in the rules, then any web traffic to the IP address of your Ingress controller can be matched without a name based virtual host being required.

secure an Ingress by specifying a SecretStores sensitive information, such as passwords, OAuth tokens, and ssh keys. that contains a TLS private key and certificate.

An Ingress controller is bootstrapped with some load balancing policy settings that it applies to all Ingress, such as the load balancing algorithm, backend weight scheme, and others.

review the controller specific documentation to see how they handle health checks

edit ingress

After you save your changes, kubectl updates the resource in the API server, which tells the Ingress controller to reconfigure the load balancer.

kubectl replace -f on a modified Ingress YAML file.

Node: A worker machine in Kubernetes, part of a cluster.

in most common Kubernetes deployments, nodes in the cluster are not part of the public internet.

Edge router: A router that enforces the firewall policy for your cluster.

Service: A Kubernetes Service that identifies a set of Pods using label selectors.

An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.

The Ingress spec has all the information needed to configure a load balancer or proxy server.

An Ingress with no rules sends all traffic to a single default backend and .spec.defaultBackend is the backend that should handle requests in that case.

If defaultBackend is not set, the handling of requests that do not match any of the rules will be up to the ingress controller

A common usage for a Resource backend is to ingress data to an object storage backend with static assets.

Exact: Matches the URL path exactly and with case sensitivity.

Prefix: Matches based on a URL path prefix split by /. Matching is case sensitive and done on a path element by element basis.

multiple paths within an Ingress will match a request. In those cases precedence will be given first to the longest matching path.

Hosts can be precise matches (for example “foo.bar.com”) or a wildcard (for example “*.foo.com”).

Each Ingress should specify a class, a reference to an IngressClass resource that contains additional configuration including the name of the controller that should implement the class.

secure an Ingress by specifying a Secret that contains a TLS private key and certificate.

The Ingress resource only supports a single TLS port, 443, and assumes TLS termination at the ingress point (traffic to the Service and its Pods is in plaintext).

hosts in the tls section need to explicitly match the host in the rules section.

Once you have a cluster running and you want to add/reconnect another node to it, you must supply an address of one of the cluster members in the cluster address URL

It will automatically retrieve the cluster map and reconnect to the rest of the nodes

it's better to list all nodes of the cluster so that any node can join a cluster connecting to any other node, even if one or more are down

The wsrep_cluster_address parameter should be added in my.cnf of each node, listing all the nodes of the cluster,

the minimum recommended number of nodes in a cluster is 3

While two of the members will be engaged in state transfer, the remaining member(s) will be able to keep on serving client requests.

Pods are immutable, which means that when they die, they are not resurrected. The Kubernetes cluster creates new pods in the same node or in a new node once a pod dies. 

For internal application access within a Kubernetes cluster, ClusterIP is the preferred method

Kubernetes Ingress is an API object that provides routing rules to manage external users' access to the services in a Kubernetes cluster, typically via HTTPS/HTTP.

content-based routing, support for multiple protocols, and authentication.

Ingress is made up of an Ingress API object and the Ingress Controller.

Kubernetes Ingress is an API object that describes the desired state for exposing services to the outside of the Kubernetes cluster.

An Ingress Controller reads and processes the Ingress Resource information and usually runs as pods within the Kubernetes cluster.  

If Kubernetes Ingress is the API object that provides routing rules to manage external access to services, Ingress Controller is the actual implementation of the Ingress API.

Layer 7 (L7) refers to the application level of the OSI stack—external connections load-balanced across pods, based on requests.

if Kubernetes Ingress is a computer, then Ingress Controller is a programmer using the computer and taking action.

the Ingress Controller is an application that runs in a Kubernetes cluster and configures an HTTP load balancer according to Ingress Resources.

ClusterIP is the preferred option for internal service access and uses an internal IP address to access the service

A NodePort is a virtual machine (VM) used to expose a service on a Static Port number.

a NodePort would be used to expose a single service (with no load-balancing requirements for multiple services).

Ingress enables you to consolidate the traffic-routing rules into a single resource and runs as part of a Kubernetes cluster.

An application is accessed from the Internet via Port 80 (HTTP) or Port 443 (HTTPS), and Ingress is an object that allows access to your Kubernetes services from outside the Kubernetes cluster. 

To implement Ingress, you need to configure an Ingress Controller in your cluster—it is responsible for processing Ingress Resource information and allowing traffic based on the Ingress Rules.

You can use role-based access control (RBAC) and other security mechanisms to make sure that users and workloads can get access to the resources they need, while keeping workloads, and the cluster itself, secure. You can set limits on the resources that users and workloads can access by managing policies and container resources.

you need to plan how to scale to relieve increased pressure from more requests to the control plane and worker nodes or scale down to reduce unused resources.

Managed control plane: Let the provider manage the scale and availability of the cluster's control plane, as well as handle patches and upgrades.

The simplest Kubernetes cluster has the entire control plane and worker node services running on the same machine.

You can deploy a control plane using tools such as kubeadm, kops, and kubespray.

Secure communications between control plane services are implemented using certificates.

Separate and backup etcd service: The etcd services can either run on the same machines as other control plane services or run on separate machines

Create multiple control plane systems: For high availability, the control plane should not be limited to a single machine

Some deployment tools set up Raft consensus algorithm to do leader election of Kubernetes services. If the primary goes away, another service elects itself and take over.

Groups of zones are referred to as regions.

if you installed with kubeadm, there are instructions to help you with Certificate Management and Upgrading kubeadm clusters.

Production-quality workloads need to be resilient and anything they rely on needs to be resilient (such as CoreDNS).

Add nodes to the cluster: If you are managing your own cluster you can add nodes by setting up your own machines and either adding them manually or having them register themselves to the cluster’s apiserver.

Set up node health checks: For important workloads, you want to make sure that the nodes and pods running on those nodes are healthy.

Authentication: The apiserver can authenticate users using client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth.

Authorization: When you set out to authorize your regular users, you will probably choose between RBAC and ABAC authorization.

Role-based access control (RBAC): Lets you assign access to your cluster by allowing specific sets of permissions to authenticated users. Permissions can be assigned for a specific namespace (Role) or across the entire cluster (ClusterRole).

Attribute-based access control (ABAC): Lets you create policies based on resource attributes in the cluster and will allow or deny access based on those attributes.

Set limits on workload resources

Set namespace limits: Set per-namespace quotas on things like memory and CPU

Prepare for DNS demand: If you expect workloads to massively scale up, your DNS service must be ready to scale up as well.

In-use IP addresses

run one or two control plane instances per failure zone, scaling those instances vertically first and then scaling horizontally after reaching the point of falling returns to (vertical) scale.

Kubernetes nodes do not automatically steer traffic towards control-plane endpoints that are in the same failure zone

store Event objects in a separate dedicated etcd instance.

start and configure additional etcd instance

Kubernetes resource limits help to minimize the impact of memory leaks and other ways that pods and containers can impact on other components.

Addons' default limits are typically based on data collected from experience running each addon on small or medium Kubernetes clusters.

Many addons scale horizontally - you add capacity by running more pods

The VerticalPodAutoscaler can run in recommender mode to provide suggested figures for requests and limits.

Some addons run as one copy per node, controlled by a DaemonSet: for example, a node-level log aggregator.

VerticalPodAutoscaler is a custom resource that you can deploy into your cluster to help you manage resource requests and limits for pods.

The cluster autoscaler integrates with a number of cloud providers to help you run the right number of nodes for the level of resource demand in your cluster.

The addon resizer helps you in resizing the addons automatically as your cluster's scale changes.

A client application is connected to the primary via MySQL Router

MySQL Shell also requires Python 2.7 and above to run cluster provisioning scripts

AdminAPI, which enables you to create and administer an InnoDB cluster, using either JavaScript or Python scripting