Group items tagged

The control plane's components make global decisions about the cluster

Control plane components can be run on any machine in the cluster.

for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine

The API server is the front end for the Kubernetes control plane.

kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances.

Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for those data.

watches for newly created Pods with no assigned node, and selects a node for them to run on.

Factors taken into account for scheduling decisions include: individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, and deadlines.

each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

Node controller

Job controller

Endpoints controller

Service Account & Token controllers

The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.

If you are running Kubernetes on your own premises, or in a learning environment inside your own PC, the cluster does not have a cloud controller manager.

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.

Kubernetes supports several container runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface).

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features

all Kubernetes clusters should have cluster DNS,

Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.

Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.

A cluster-level logging mechanism is responsible for saving container logs to a central log store with search/browsing interface.

決定每個 shard 要被分配到哪個 data node 上

為 cluster 設置多個 master node

一旦發現被選中的 master node 出現問題，就會選出新的 master node

處理 request 的 node 稱為 Coordinating Node，其功能是將 request 轉發到合適的 node 上

所有的 node 都預設是 Coordinating Node

可以保存資料的 node，每個 node 啟動後都會預設是 data node，可以透過設定 node.data: false 停用 data node 功能

由 master node 決定如何把分片分發到不同的 data node 上

每個 node 上都保存了 cluster state

只有 master 才可以修改 cluster state 並負責同步給其他 node

每個 node 都會詳細紀錄本身的狀態資訊

shard 是 Elasticsearch 分散式儲存的基礎，包含 primary shard & replica shard

每一個 shard 就是一個 Lucene instance

primary shard 功能是將一份被索引後的資料，分散到多個 data node 上存放，實現儲存方面的水平擴展

當 primary shard 遺失時，replica shard 就可以被 promote 成 primary shard 來保持資料完整性

replica shard 數量可以動態調整，讓每個 data node 上都有完整的資料

ES 7.0 開始，primary shard 預設為 1，replica shard 預設為 0

replica shard 若設定過多，會降低 cluster 整體的寫入效能

所有的 primary shard 可以在同一個 data node 上

透過 GET _cluster/health/<target> 可以取得目前 cluster 的健康狀態

Yellow：表示 primary shard 可以正常分配，但 replica shard 分配有問題

透過 GET /_cat/shards/<target> 可以取得目前的 shard 狀態

replica shard 無法被分配，因此 cluster 健康狀態為黃色

若是擔心 reboot 機器造成 failover 動作開始執行，可以設定將 replication 延遲一段時間後再執行(透過調整 settings 中的 index.unassigned.node_left.delayed_timeout 參數)，避免無謂的 data copy 動作 (此功能稱為 delay allocation)

集群變紅，代表有 primary shard 丟失，這個時候會影響讀寫。

如果 node 重新回來，會從 translog 中恢復沒有寫入的資料

設定 index settings 之後，primary shard 數量無法隨意變更

不建議直接發送請求到master節點，雖然也會工作，但是大量請求發送到 master，會有潛在的性能問題

shard 是 ES 中最小的工作單元

shard 是一個 Lucene 的 index

將 Index Buffer 中的內容寫入 Segment，而這寫入的過程就稱為 Refresh

當 document 被 refresh 進入到 segment 之後，就可以被搜尋到了

在進行 refresh 時先將 segment 寫入 cache 以開放查詢

將 document 進行索引時，同時也會寫入 transaction log，且預設都會寫入磁碟中

每個 shard 都會有對應的 transaction log

由於 transaction log 都會寫入磁碟中，因此當 node 從故障中恢復時，就會優先讀取 transaction log 來恢復資料

Terraform reads data resources during the planning phase when possible, but announces in the plan when it must defer reading resources until the apply phase to preserve the order of operations.

local-only data sources exist for rendering templates, reading local files, and rendering AWS IAM policies.

As with managed resources, when count or for_each is present it is important to distinguish the resource itself from the multiple resource instances it creates. Each instance will separately read from its data source with its own variant of the constraint arguments, producing an indexed result.

Data instance arguments may refer to computed values, in which case the attributes of the instance itself cannot be resolved until all of its arguments are defined. I

When a Workspace is declared in a job, one or more files or directories can be added. Each addition creates a new layer in the Workspace filesystem. Downstreams jobs can then use this Workspace for its own needs or add more layers on top.

Unlike caching, Workspaces are not shared between runs as they no longer exists once a Workflow is complete.

A prime example is package dependency managers such as Yarn, Bundler, or Pip.

Caches are global within a project, a cache saved on one branch will be used by others so they should only be used for data that is OK to share across Branches

Artifacts are used for longer-term storage of the outputs of your build process.

If your project needs to be packaged in some form or fashion, say an Android app where the .apk file is uploaded to Google Play, that’s a great example of an artifact.

To use a secret, a pod needs to reference the secret.

--from-file

You can also create a Secret in a file first, in json or yaml format, and then create that object.

The Secret contains two maps: data and stringData.

Kubernetes automatically creates secrets which contain credentials for accessing the API and it automatically modifies your pods to use this type of secret.

kubectl get and kubectl describe avoid showing the contents of a secret by default.

stringData field is provided for convenience, and allows you to provide secret data as unencoded strings.

where you are deploying an application that uses a Secret to store a configuration file, and you want to populate parts of that configuration file during your deployment process.

a field is specified in both data and stringData, the value from stringData is used.

The keys of data and stringData must consist of alphanumeric characters, ‘-’, ‘_’ or ‘.’.

Newlines are not valid within these strings and must be omitted.

When using the base64 utility on Darwin/macOS users should avoid using the -b option to split long lines.

create a Secret from generators and then apply it to create the object on the Apiserver.

The generated Secrets name has a suffix appended by hashing the contents.

Multiple pods can reference the same secret.

each container needs its own volumeMounts block, but only one .spec.volumes is needed per secret

use .spec.volumes[].secret.items field to change target path of each key:

You can also specify the permission mode bits files part of a secret will have. If you don’t specify any, 0644 is used by default.

Kubelet is checking whether the mounted secret is fresh on every periodic sync.

cache propagation delay depends on the chosen cache type

Multiple pods can reference the same secret.

env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base-64 decoded values of the secret data.

An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry password to the Kubelet so it can pull a private image on behalf of your Pod.

Individual secrets are limited to 1MiB in size.

Kubelet only supports use of secrets for Pods it gets from the API server.

Secrets must be created before they are consumed in pods as environment variables unless they are marked as optional.

Once a pod is scheduled, the kubelet will try to fetch the secret value.

Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret.

volumes: - name: secret-volume secret: secretName: ssh-key-secret

You do not need to escape special characters in passwords from files

make that key begin with a dot

Dotfiles in secret volume

.secret-file

a frontend container which handles user interaction and business logic, but which cannot see the private key;

a signer container that can see the private key, and responds to simple signing requests from the frontend

watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided

watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.

additional precautions with secret objects, such as avoiding writing them to disk where possible.

A secret is only sent to a node if a pod on that node requires it

only the secrets that a pod requests are potentially visible within its containers

each container in a pod has to request the secret volume in its volumeMounts for it to be visible within the container.

In the API server secret data is stored in etcdConsistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

limit access to etcd to admin users

A user who can create a pod that uses a secret can also see the value of that secret.

anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet.

A controller tracks at least one Kubernetes resource type.

The controller(s) for that resource are responsible for making the current state come closer to that desired state.

in Kubernetes, a controller will send messages to the API server that have useful side effects.

Built-in controllers manage state by interacting with the cluster API server.

By contrast with Job, some controllers need to make changes to things outside of your cluster.

As long as the controllers for your cluster are running and able to make useful changes, it doesn't matter if the overall state is stable or not.

Kubernetes uses lots of controllers that each manage a particular aspect of cluster state.

a particular control loop (controller) uses one kind of resource as its desired state, and has a different kind of resource that it manages to make that desired state happen.

There can be several controllers that create or update the same kind of object.

On Linux, control groups are used to constrain resources that are allocated to processes.

Both kubelet and the underlying container runtime need to interface with control groups to enforce resource management for pods and containers and set resources such as cpu/memory requests and limits.

When the cgroupfs driver is used, the kubelet and the container runtime directly interface with the cgroup filesystem to configure cgroups.

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager.

Two cgroup managers result in two views of the available and in-use resources in the system.

Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.

The approach to mitigate this instability is to use systemd as the cgroup driver for the kubelet and the container runtime when systemd is the selected init system.

Kubernetes 1.26 defaults to using v1 of the CRI API. If a container runtime does not support the v1 API, the kubelet falls back to using the (deprecated) v1alpha2 API instead.

Persistent volumes are long-term storage in your Kubernetes cluster.

A pod uses a persistent volume claim to to get read and write access to the persistent volume.

NFS stands for Network File System – it's a shared filesystem that can be accessed over the network.

what's already stored in the NFS is not deleted when a pod is destroyed. Data is persistent.

an NFS can be accessed from multiple pods at the same time. An NFS can be used to share data between pods!

volumes: - name: nfs-volume nfs: # URL for the NFS server server: 10.108.211.244 # Change this! path: /

volumeMounts: - name: nfs-volume mountPath: /var/nfs

Just add the volume to each pod, and add a volume mount to use the NFS volume from each container.

Percona XtraBackup works by remembering the log sequence number (LSN) when it starts, and then copying away the data files.

Percona XtraBackup runs a background process that watches the transaction log files, and copies changes from it.

Percona XtraBackup uses Backup locks where available as a lightweight alternative to FLUSH TABLES WITH READ LOCK.

Locking is only done for MyISAM and other non-InnoDB tables after Percona XtraBackup finishes backing up all InnoDB/XtraDB data and logs.

When backup locks are supported by the server, xtrabackup first copies InnoDB data, runs the LOCK TABLES FOR BACKUP and then copies the MyISAM tables.

the STDERR of xtrabackup is not written in any file. You will have to redirect it to a file, e.g., xtrabackup OPTIONS 2> backupout.log

During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files, using the copied transaction log file. After this is done, the database is ready to restore and use.

the tools enable you to do operations such as streaming and incremental backups with various combinations of copying the data files, copying the log files, and applying the logs to the data.

To restore a backup with xtrabackup you can use the --copy-back or --move-back options.

a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).

Each Container in the Pod must independently specify where to mount each volume

localnfs

cephfs

awsElasticBlockStore

glusterfs

vsphereVolume

An awsElasticBlockStore volume mounts an Amazon Web Services (AWS) EBS Volume into your Pod.

an EBS volume can be pre-populated with data, and that data can be “handed off” between Pods.

create an EBS volume using aws ec2 create-volume

the nodes on which Pods are running must be AWS EC2 instances

EBS only supports a single EC2 instance mounting a volume

check that the size and EBS volume type are suitable for your use!

A cephfs volume allows an existing CephFS volume to be mounted into your Pod.

the contents of a cephfs volume are preserved and the volume is merely unmounted.

have your own Ceph server running with the share exported

The configMap resource provides a way to inject configuration data into Pods

When referencing a configMap object, you can simply provide its name in the volume to reference it

volumeMounts: - name: config-vol mountPath: /etc/config volumes: - name: config-vol configMap: name: log-config items: - key: log_level path: log_level

create a ConfigMap before you can use it.

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node.

By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment.

you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem)

volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {}

An fc volume allows an existing fibre channel volume to be mounted in a Pod.

configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

Flocker is an open-source clustered Container data volume manager. It provides management and orchestration of data volumes backed by a variety of storage backends.

emptyDir

flocker

A flocker volume allows a Flocker dataset to be mounted into a Pod

have your own Flocker installation running

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your Pod.

Using a PD on a Pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1

A glusterfs volume allows a Glusterfs (an open source networked filesystem) volume to be mounted into your Pod.

have your own GlusterFS installation running

a powerful escape hatch for some applications

access to Docker internals; use a hostPath of /var/lib/docker

allowing a Pod to specify whether a given hostPath should exist prior to the Pod running, whether it should be created, and what it should exist as

specify a type for a hostPath volume

the files or directories created on the underlying hosts are only writable by root.

hostPath: # directory location on host path: /data # this field is optional type: Directory

An iscsi volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.

have your own iSCSI server running

A feature of iSCSI is that it can be mounted as read-only by multiple consumers simultaneously.

A local volume represents a mounted local storage device such as a disk, partition or directory.

Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.

PersistentVolume spec using a local volume and nodeAffinity

PersistentVolume volumeMode can now be set to “Block” (instead of the default value “Filesystem”) to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer

An nfs volume allows an existing NFS (Network File System) share to be mounted into your Pod.

have your own NFS server running with the share exported

A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.

PersistentVolumes are a way for users to “claim” durable storage (such as a GCE PersistentDisk or an iSCSI volume) without knowing the details of the particular cloud environment.

A projected volume maps several existing volume sources into the same directory.

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Each projected volume source is listed in the spec under sources

A Container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.

RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed

A secret volume is used to pass sensitive information, such as passwords, to Pods

create a secret in the Kubernetes API before you can use it

StorageOS runs as a Container within your Kubernetes environment, making local or attached storage accessible from any node within the Kubernetes cluster.

StorageOS provides block storage to Containers, accessible via a file system.

A vsphereVolume is used to mount a vSphere VMDK Volume into your Pod.

supports both VMFS and VSAN datastore.

create VMDK using one of the following methods before using with Pod.

share one volume for multiple uses in a single Pod.

volumeMounts: - name: workdir1 mountPath: /logs subPathExpr: $(POD_NAME)

env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name

Use the subPathExpr field to construct subPath directory names from Downward API environment variables

enable the VolumeSubpathEnvExpansion feature gate

emptyDir and hostPath volumes will be able to request a certain amount of space using a resource specification, and to select the type of media to use, for clusters that have several media types.

the Container Storage Interface (CSI) and Flexvolume. They enable storage vendors to create custom storage plugins without adding them to the Kubernetes repository.

Container Storage Interface (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the csi volume type to attach, mount, etc. the volumes exposed by the CSI driver.

This feature requires CSIInlineVolume feature gate to be enabled:--feature-gates=CSIInlineVolume=true

In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the “Types of Volumes” section above.

Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts.

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the Container will be propagated back to the host and to all Containers of all Pods that use the same volume.

Edit your Docker’s systemd service file. Set MountFlags as follows:MountFlags=shared

globalLock.totalTime: If this is higher than the total database uptime, the database has been in a lock state for too long.

Unlike relational databases such as MySQL or PostgreSQL, MongoDB uses JSON-like documents for storing data.

Databases operate in an environment that consists of numerous reads, writes, and updates.

When a lock occurs, no other operation can read or modify the data until the operation that initiated the lock is finished.

locks.deadlockCount: Number of times the lock acquisitions have encountered deadlocks

Is the database frequently locking from queries? This might indicate issues with the schema design, query structure, or system architecture.

For version 3.2 on, WiredTiger is the default.

MMAPv1 locks whole collections, not individual documents.

When the MMAPv1 storage engine is in use, MongoDB will use memory-mapped files to store data.

db.serverStatus().mem

mem.resident: Roughly equivalent to the amount of RAM in megabytes that the database process uses

If mem.resident exceeds the value of system memory and there’s a large amount of unmapped data on disk, we’ve most likely exceeded system capacity.

If the value of mem.mapped is greater than the amount of system memory, some operations will experience page faults.

The WiredTiger storage engine is a significant improvement over MMAPv1 in performance and concurrency.

By default, MongoDB will reserve 50 percent of the available memory for the WiredTiger data cache.

wiredTiger.cache.bytes currently in the cache – This is the size of the data currently in the cache.

wiredTiger.cache.tracked dirty bytes in the cache – This is the size of the dirty data in the cache.

we can look at the wiredTiger.cache.bytes read into cache value for read-heavy applications. If this value is consistently high, increasing the cache size may improve overall read performance.

check whether the application is read-heavy. If it is, increase the size of the replica set and distribute the read operations to secondary members of the set.

Replication is the propagation of data from one node to another

Replication sets handle this replication.

Sometimes, data isn’t replicated as quickly as we’d like.

use the db.printSlaveReplicationInfo() or the rs.printSlaveReplicationInfo() command to see the status of a replica set from the perspective of the secondary member of the set.

shows how far behind the secondary members are from the primary. This number should be as low as possible.

monitor this metric closely.

watch for any spikes in replication delay.

One replica set is primary. All others are secondary.

use the profiler to gain a deeper understanding of the database’s behavior.

Enabling the profiler can affect system performance, due to the additional activity.

In full backups, two types of operations are performed to make the database consistent: committed transactions are replayed from the log file against the data files, and uncommitted transactions are rolled back.

You should use the --apply-log-only option to prevent the rollback phase.

An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s LSN.

Incremental backups do not actually compare the data files to the previous backup’s data files.

Incremental backups simply read the pages and compare their LSN to the last backup’s LSN.

The xtrabackup binary writes a file called xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn, which is the database’s LSN at the end of the backup.

from_lsn is the starting LSN of the backup and for incremental it has to be the same as to_lsn (if it is the last checkpoint) of the previous/base backup.

If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups will be useless.

If you restore it and start MySQL, InnoDB will detect that the rollback phase was not performed, and it will do that in the background, as it usually does for a crash recovery upon start.

xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc1

The final data is in /data/backups/base, not in the incremental directory.

xtrabackup --prepare --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc2

Even if the --apply-log-only was used on the last step, backup would still be consistent but in that case server would perform the rollback phase.

replication can provide increased read capacity as clients can send read operations to different servers.

A replica set contains several data bearing nodes and optionally one arbiter node.

A replica set can have only one primary capable of confirming writes with { w: "majority" } write concern; although in some circumstances, another mongod instance may transiently believe itself to also be primary.

The secondaries replicate the primary’s oplog and apply the operations to their data sets such that the secondaries’ data sets reflect the primary’s data set

add a mongod instance to a replica set as an arbiter. An arbiter participates in elections but does not hold data

An arbiter will always be an arbiter whereas a primary may step down and become a secondary and a secondary may become the primary during an election.

Secondaries replicate the primary’s oplog and apply the operations to their data sets asynchronously.

These slow oplog messages are logged for the secondaries in the diagnostic log under the REPL component with the text applied op: <oplog entry> took <num>ms.

Replication lag refers to the amount of time that it takes to copy (i.e. replicate) a write operation on the primary to a secondary.

When a primary does not communicate with the other members of the set for more than the configured electionTimeoutMillis period (10 seconds by default), an eligible secondary calls for an election to nominate itself as the new primary.

The replica set cannot process write operations until the election completes successfully.

The median time before a cluster elects a new primary should not typically exceed 12 seconds, assuming default replica configuration settings.

Factors such as network latency may extend the time required for replica set elections to complete, which in turn affects the amount of time your cluster may operate without a primary.

MongoDB drivers can detect the loss of the primary and automatically retry certain write operations a single time, providing additional built-in handling of automatic failovers and elections

By default, clients read from the primary [1]; however, clients can specify a read preference to send read operations to secondaries.