Group items tagged

Persistent volumes are long-term storage in your Kubernetes cluster.

A pod uses a persistent volume claim to to get read and write access to the persistent volume.

NFS stands for Network File System – it's a shared filesystem that can be accessed over the network.

what's already stored in the NFS is not deleted when a pod is destroyed. Data is persistent.

an NFS can be accessed from multiple pods at the same time. An NFS can be used to share data between pods!

volumes: - name: nfs-volume nfs: # URL for the NFS server server: 10.108.211.244 # Change this! path: /

volumeMounts: - name: nfs-volume mountPath: /var/nfs

Just add the volume to each pod, and add a volume mount to use the NFS volume from each container.

a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).

Each Container in the Pod must independently specify where to mount each volume

localnfs

cephfs

awsElasticBlockStore

glusterfs

vsphereVolume

An awsElasticBlockStore volume mounts an Amazon Web Services (AWS) EBS Volume into your Pod.

an EBS volume can be pre-populated with data, and that data can be “handed off” between Pods.

create an EBS volume using aws ec2 create-volume

the nodes on which Pods are running must be AWS EC2 instances

EBS only supports a single EC2 instance mounting a volume

check that the size and EBS volume type are suitable for your use!

A cephfs volume allows an existing CephFS volume to be mounted into your Pod.

the contents of a cephfs volume are preserved and the volume is merely unmounted.

have your own Ceph server running with the share exported

The configMap resource provides a way to inject configuration data into Pods

When referencing a configMap object, you can simply provide its name in the volume to reference it

volumeMounts: - name: config-vol mountPath: /etc/config volumes: - name: config-vol configMap: name: log-config items: - key: log_level path: log_level

create a ConfigMap before you can use it.

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node.

By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment.

you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem)

volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {}

An fc volume allows an existing fibre channel volume to be mounted in a Pod.

configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

Flocker is an open-source clustered Container data volume manager. It provides management and orchestration of data volumes backed by a variety of storage backends.

emptyDir

flocker

A flocker volume allows a Flocker dataset to be mounted into a Pod

have your own Flocker installation running

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your Pod.

Using a PD on a Pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1

A glusterfs volume allows a Glusterfs (an open source networked filesystem) volume to be mounted into your Pod.

have your own GlusterFS installation running

a powerful escape hatch for some applications

access to Docker internals; use a hostPath of /var/lib/docker

allowing a Pod to specify whether a given hostPath should exist prior to the Pod running, whether it should be created, and what it should exist as

specify a type for a hostPath volume

the files or directories created on the underlying hosts are only writable by root.

hostPath: # directory location on host path: /data # this field is optional type: Directory

An iscsi volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.

have your own iSCSI server running

A feature of iSCSI is that it can be mounted as read-only by multiple consumers simultaneously.

A local volume represents a mounted local storage device such as a disk, partition or directory.

Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.

PersistentVolume spec using a local volume and nodeAffinity

PersistentVolume volumeMode can now be set to “Block” (instead of the default value “Filesystem”) to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer

An nfs volume allows an existing NFS (Network File System) share to be mounted into your Pod.

have your own NFS server running with the share exported

A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.

PersistentVolumes are a way for users to “claim” durable storage (such as a GCE PersistentDisk or an iSCSI volume) without knowing the details of the particular cloud environment.

A projected volume maps several existing volume sources into the same directory.

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Each projected volume source is listed in the spec under sources

A Container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.

RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed

A secret volume is used to pass sensitive information, such as passwords, to Pods

create a secret in the Kubernetes API before you can use it

StorageOS runs as a Container within your Kubernetes environment, making local or attached storage accessible from any node within the Kubernetes cluster.

StorageOS provides block storage to Containers, accessible via a file system.

A vsphereVolume is used to mount a vSphere VMDK Volume into your Pod.

supports both VMFS and VSAN datastore.

create VMDK using one of the following methods before using with Pod.

share one volume for multiple uses in a single Pod.

volumeMounts: - name: workdir1 mountPath: /logs subPathExpr: $(POD_NAME)

env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name

Use the subPathExpr field to construct subPath directory names from Downward API environment variables

enable the VolumeSubpathEnvExpansion feature gate

emptyDir and hostPath volumes will be able to request a certain amount of space using a resource specification, and to select the type of media to use, for clusters that have several media types.

the Container Storage Interface (CSI) and Flexvolume. They enable storage vendors to create custom storage plugins without adding them to the Kubernetes repository.

Container Storage Interface (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the csi volume type to attach, mount, etc. the volumes exposed by the CSI driver.

This feature requires CSIInlineVolume feature gate to be enabled:--feature-gates=CSIInlineVolume=true

In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the “Types of Volumes” section above.

Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts.

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the Container will be propagated back to the host and to all Containers of all Pods that use the same volume.

Edit your Docker’s systemd service file. Set MountFlags as follows:MountFlags=shared

管理者可能會直接在 NFS Server 上進行 MDADM 來設定相關的 Block Device 並且基於上面提供 Export 供 NFS 使用，甚至底層套用不同的檔案系統 (EXT4/BTF4) 來獲取不同的功能與效能。

Kubernetes 就只是 NFS Client 的角色

CSI(Container Storage Interface)。CSI 本身作為 Kubernetes 與 Storage Solution 的中介層。

基本上 Pod 裡面每個 Container 會使用 Volume 這個物件來代表容器內的掛載點，而在外部實際上會透過 PVC 以及 PV 的方式來描述這個 Volume 背後的儲存方案伺服器的資訊。

整體會透過 CSI 的元件們與最外面實際上的儲存設備連接，所有儲存相關的功能是否有實現，有支援全部都要仰賴最後面的實際提供者， kubernetes 只透過 CSI 的標準去執行。

在網路部分也有與之對應的 CNI(Container Network Interface). kubernetes 透過 CNI 這個介面來與後方的 網路解決方案 溝通

CNI 最基本的要求就是在在對應的階段為對應的容器提供網路能力

目前最常見也是 IPv4 + TCP/UDP 的傳輸方式，因此才會看到大部分的 CNI 都在講這些。

希望所有容器彼此之間可以透過 IPv4 來互相存取彼此，不論是同節點或是跨節點的容器們都要可以滿足這個需求。

容器間到底怎麼傳輸的，需不需要封裝，透過什麼網卡，要不要透過 NAT 處理? 這一切都是 CNI 介面背後的實現

外部網路存取容器服務 (Service/Ingress)

kubernetes 在 Service/Ingress 中間自行實現了一個模組，大抵上稱為 kube-proxy, 其底層可以使用 iptables, IPVS, user-space software 等不同的實現方法，這部分是跟 CNI 完全無關。

CNI 一般會處理的部份，包含了容器內的 網卡數量,網卡名稱,網卡IP, 以及容器與外部節點的連接能力等

CRI (Container Runtime Interface) 或是 Device Plugin

去思考到底為什麼自己本身的服務需要容器化，容器化可以帶來什麼優點

最後就會發現根本把 Container 當作 Virtual Machine 來使用，然後再補一句 Contaienr 根本不好用啊

容器化 不是把直接 Virtual Machine 的使用習慣換個環境使用就叫做 容器化，而是要從概念上去暸解與使用

VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

A vSphere datastore is an abstraction which hides storage details (such as LUNs) and provides a uniform interface for storing persistent data.

the datastores can be of the type vSAN, VMFS, NFS & VVol.

VMFS (Virtual Machine File System) is a cluster file system that allows virtualization to scale beyond a single node for multiple VMware ESX servers.

NFS (Network File System) is a distributed file protocol to access storage over network like local storage.

vSphere Cloud Provider supports every storage primitive exposed by Kubernetes

Kubernetes PVs are defined in Pod specifications.

Workloads running in a Docker service that require access to low latency/high IOPs persistent storage, such as a database engine, can use a relocatable Cloudstor volume backed by EBS.

If a swarm task using a relocatable Cloudstor volume gets rescheduled to another node within the same availability zone as the original node where the task was running, Cloudstor detaches the backing EBS volume from the original node and attaches it to the new target node automatically.

in a different availability zone,

Cloudstor transfers the contents of the backing EBS volume to the destination availability zone using a snapshot, and cleans up the EBS volume in the original availability zone.

Typically the snapshot-based transfer process across availability zones takes between 2 and 5 minutes unless the work load is write-heavy.

A swarm task is not started until the volume it mounts becomes available

Sharing/mounting the same Cloudstor volume backed by EBS among multiple tasks is not a supported scenario and leads to data loss.

a volume and its contents can be mounted by multiple swarm service tasks without the risk of data loss

the persistent data backed by EFS volumes is always available.

shared Cloudstor volumes only work in those AWS regions where EFS is supported.

从集群中获取每个 pod ip 地址，然后也能在集群内部直接通过 podIP:Port 来获取对应的服务。

pod 是经常变化的，每次更新 ip 地址都可能会发生变化，如果直接访问容器 ip 的话，会有很大的问题。

“服务”（service），每个服务都一个固定的虚拟 ip（这个 ip 也被称为 cluster IP），自动并且动态地绑定后面的 pod，所有的网络请求直接访问服务 ip，服务会自动向后端做转发。

实现 service 这一功能的关键，就是 kube-proxy。

kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1

iptables 完全实现 iptables 来实现 service，是目前默认的方式，也是推荐的方式，效率很高（只有内核中 netfilter 一些损耗）。

可以在终端上启动 kube-proxy，也可以使用诸如 systemd 这样的工具来管理它

Make sure that the br_netfilter module is loaded.

you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config,

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you.

one minor version skew between the kubelet and the control plane is supported, but the kubelet version may never exceed the API server version.

Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.