Group items tagged

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

Cluster-level logging architectures require a separate backend to store, analyze, and query logs

use kubectl logs --previous to retrieve logs from a previous instantiation of a container.

A container engine handles and redirects any output generated to a containerized application's stdout and stderr streams

An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node

You can also set up a container runtime to rotate an application's logs automatically.

The two kubelet flags container-log-max-size and container-log-max-files can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.

The kubelet and container runtime do not run in containers.

On machines with systemd, the kubelet and container runtime write to journald. If systemd is not present, the kubelet and container runtime write to .log files in the /var/log directory.

Kubernetes does not provide a native solution for cluster-level logging

implement cluster-level logging by including a node-level logging agent on each node.

the logging agent must run on every node, it is recommended to run the agent as a DaemonSet

Node-level logging creates only one agent per node and doesn't require any changes to the applications running on the node.

Containers write stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation.

Each sidecar container prints a log to its own stdout or stderr stream.

It is not recommended to write log entries with different formats to the same log stream

writing logs to a file and then streaming them to stdout can double disk usage.

it's recommended to use stdout and stderr directly and leave rotation and retention policies to the kubelet.

The upgrade procedure on control plane nodes should be executed one node at a time.

Manually upgrade your CNI provider plugin.

sudo systemctl daemon-reload sudo systemctl restart kubelet

If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again.

To recover from a bad state, you can also run kubeadm upgrade apply --force without changing the version that your cluster is running.

kubeadm-backup-etcd contains a backup of the local etcd member data for this control plane Node.

the contents of this folder can be manually restored in /var/lib/etcd

kubeadm-backup-manifests contains a backup of the static Pod manifest files for this control plane Node.

the contents of this folder can be manually restored in /etc/kubernetes/manifests

Enforces the version skew policies.

Upgrades the control plane components or rollbacks if any of them fails to come up.

一句话形容下Loki就是like Prometheus, but for logs。

Promtail是一个日志收集的代理，它会将本地日志的内容发送到一个Loki实例，它通常部署到需要监视应用程序的每台机器/容器上。Promtail主要是用来发现目标、将标签附加到日志流以及将日志推送到Loki。

Loki中的日志带有一组标签名和值，其中只有标签对被索引，这种权衡使得它比完整索引的操作成本更低，但是针对基于内容的查询，需要通过LogQL再单独查询。

和Fluentd相比，Promtail是专门为Loki量身定制的，它可以为运行在同一节点上的Kubernetes Pods做服务发现，从指定文件夹读取日志。

亚马逊云科技也提供了Grafana和Prometheus的托管服务Amazon Managed Service for Grafana（AMG）和Amazon Managed Service for Prometheus（AMP）

choosing a leader for a distributed application without an internal member election process

publishing a Service to applications that don't support Kubernetes APIs to discover them

The core of the Operator is code to tell the API server how to make reality match the configured resources.

to deploy an Operator is to add the Custom Resource Definition and its associated Controller to your cluster.

Once you have an Operator deployed, you'd use it by adding, modifying or deleting the kind of resource that the Operator uses.

算數邏輯單元與控制單元。其中算數邏輯單元主要負責程式運算與邏輯判斷，控制單元則主要在協調各周邊元件與各單元間的工作。

資料會先寫入到主記憶體，然後 CPU 才能開始應用

CPU 其實內部已經含有一些微指令，我們所使用的軟體都要經過 CPU 內部的微指令集來達成才行。

世界上常見到的兩種主要 CPU 架構， 分別是：精簡指令集 (RISC) 與複雜指令集 (CISC) 系統。

微指令集較為精簡，每個指令的執行時間都很短，完成的動作也很單純，指令的執行效能較佳； 但是若要做複雜的事情，就要由多個指令來完成。

CISC在微指令集的每個小指令可以執行一些較低階的硬體操作，指令數目多而且複雜， 每條指令的長度並不相同。因為指令執行較為複雜所以每條指令花費的時間較長， 但每條個別指令可以處理的工作較為豐富。

最早的那顆Intel發展出來的CPU代號稱為8086

AMD依此架構修改新一代的CPU為64位元

CPU 位元指的是CPU一次資料讀取的最大量！64位元CPU代表CPU一次可以讀寫64bits這麼多的資料

因為CPU讀取資料量有限制，因此能夠從記憶體中讀寫的資料也就有所限制

：(1)北橋：負責連結速度較快的CPU、主記憶體與顯示卡界面等元件；

(2)南橋：負責連接速度較慢的裝置介面， 包括硬碟、USB、網路卡等等。不過由於北橋最重要的就是 CPU 與主記憶體之間的橋接，因此目前的主流架構中， 大多將北橋記憶體控制器整合到 CPU 封裝當中

CPU 與可接受的晶片組是有搭配性的，尤其目前每種 CPU 的腳位都不一樣

時脈越高，代表單位時間內可進行的工作越多

大概都只看總頻寬或基準時脈

CPU 時脈越高的情況下，會產生很多諸如散熱、設計及與週邊元件溝通的問題。

其他的程序還在等待 CPU ，但是 CPU 又在等待 I/O

Intel 在同一個運算核心底下安裝兩個暫存器來模擬出另一個核心，實際上是兩個暫存器 (可以想成是程式的執行器) 共用一個實體核心。 這就是超執行緒的簡易認知

時脈的意思是每秒鐘能夠進行的工作次數，而每次工作能夠讀進的資料量就是位元數

每個用戶端大多是短時間的大運算，所以通常不會有一隻程序一直佔用著系統資源。

CPU 的所有資料都是從記憶體讀寫來的，所以記憶體的容量與頻寬就相當的重要了

DRAM 根據技術的更新又分好幾代，而使用上較廣泛的有所謂的 SDRAM 與 DDR SDRAM 兩種。 這兩種記憶體的差別除了在於腳位與工作電壓上的不同之外，DDR 是所謂的雙倍資料傳送速度 (Double Data Rate)

晶片組廠商就將兩個主記憶體彙整在一起，如果一支記憶體可達 64 位元，兩支記憶體就可以達到 128 位元了，這就是雙通道的設計理念。 在某些比較多核心的伺服器主機上面，甚至還使用了 3 通道到 4 通道的設計

CPU 內部的暫存器與少量記憶體被稱為第一、第二層快取 (L1, L2 cache)，而放在核心間的快取記憶體就被稱為第三層快取記憶體 (L3 cache)

CPU 的暫存器是直接設計在 CPU 核心內部，可以用來幫助 CPU 的運算。

在多核心 CPU 的核心之間，會有另一個共享的快取記憶體存在，讓所有的 CPU 核心可以共享某些資源。

靜態隨機存取記憶體 (Static Random Access Memory, SRAM)

各項參數， 是被記錄到主機板上頭的一個稱為 CMOS 的晶片上，這個晶片需要藉著額外的電源來發揮記錄功能， 這也是為什麼你的主機板上面會有一顆電池的緣故。

BIOS(Basic Input Output System)是一套程式，這套程式是寫死到主機板上面的一個記憶體晶片中， 這個記憶體晶片在沒有通電時也能夠將資料記錄下來，那就是唯讀記憶體(Read Only Memory, ROM)。

BIOS對於個人電腦來說是非常重要的， 因為他是系統在開機的時候首先會去讀取的一個小程式喔

韌體(firmware)很多也是使用ROM來進行軟體的寫入的。 韌體像軟體一樣也是一個被電腦所執行的程式，然而他是對於硬體內部而言更加重要的部分。例如BIOS就是一個韌體， BIOS雖然對於我們日常操作電腦系統沒有什麼太大的關係，但是他卻控制著開機時各項硬體參數的取得！

現在的 BIOS 通常是寫入類似快閃記憶體 (flash) 或 EEPROM

顯示卡能夠傳輸的資料量當然也是越大越好！這時就得要考慮到傳輸的界面了！當前 (2018) 主流的傳輸界面為 PCI-E，這個 PCI-E 又有三種版本， version 1, 2, 3 ，這三個版本的速度並不相同

PCI-E (PCI-Express) 使用的是類似管線的概念來處理，在 PCI-E 第一版中，每條管線可以具有 250MBytes/s 的頻寬效能，管線越多(通常設計到 x16 管線)則總頻寬越高

通常 CPU 製造商會根據 CPU 來設計搭配的南橋晶片，由於現在只有一個晶片 (北橋整合到 CPU 內了)，所以目前只有一顆主機板晶片組。

插槽上面的資料要傳輸到 CPU 就得要經過晶片組，然後透過 DMI 通道傳上去！所以，在某些情況下，系統的效能會被卡住在這條通道上喔！

晶片組接的設備相當多喔！有 PCI-E 插槽、USB設備、網路設備、音效設備、硬碟設備等，這全部的裝置共用那一條 DMI 喔！ 所以，整個系統效能的瓶頸通常不在 CPU 啦！通常就是在南橋接的設備上面！所以，當你有非常複雜的程式要運作的時候，讓這些程式越少通過南橋， 他的系統效能就會比較好一點

物理組成

讀寫主要是透過在機械手臂上的讀取頭(head)來達成

由於磁碟盤是圓的，並且透過機器手臂去讀寫資料，磁碟盤要轉動才能夠讓機器手臂讀寫。

磁碟的最小物理儲存單位，稱之為磁區 (sector)，那同一個同心圓的磁區組合成的圓就是所謂的磁軌(track)。 由於磁碟裡面可能會有多個磁碟盤，因此在所有磁碟盤上面的同一個磁軌可以組合成所謂的磁柱 (cylinder)。

通常資料的讀寫會由外圈開始往內寫

目前主流的硬碟連接界面就是 SATA

雖然 SATA III 界面理論傳輸可到達 600Mbytes/s，不過傳統硬碟由於物理設計的限制因素，通常存取速度效能大多在 150~200Mbytes/s 之間

串列式 SCSI (Serial Attached SCSI, SAS)

傳統硬碟有個很致命的問題，就是需要驅動馬達去轉動磁碟盤～這會造成很嚴重的磁碟讀取延遲

快閃記憶體去製作成高容量的設備

沒有讀寫頭與磁碟盤啊！都是記憶體！

各個繪圖卡的運算晶片 (GPU) 設計的理念不同，加上驅動程式與軟體搭配的問題，並不是一張高效能繪圖卡就可以完勝其他的繪圖卡， 還得要注意相關的軟體才行。

透過卡片與讀卡機，將程式碼一次性的讀進大機器，然後就是等待大機器的運作， 結果再交由印表機印出。如果打卡紙打洞錯誤呢？只好重新打洞，重新排隊去運作程式了。

允許兩個以上的程序在記憶體中等待被 CPU 執行，當 CPU 執行完其中一隻程式後， 第二隻程式就可以立刻被執行，因此效能會比較好。

程序的狀態進入中斷狀態，CPU 不會理會該程序

CPU 的排程 (cpu scheduling)

早期單核 CPU 的運作中，CPU 一次只能運作一個工作，因此，若有多個工作要同時進行， 那麼 CPU 就得要安排一個 CPU 運作時間給所有的工作，當該程序達到最大工作時間後，CPU 就會將該工作排回佇列，讓下一隻程序接著運作。

分時系統其實與多元程式處理系統有點類似， 只是工作的輸入改為透過終端機操作輸入，CPU 可以在各個用戶操作間切換工作，於是每個使用者感覺似乎都是在同步操作電腦系統一般， 這就是分時系統。

早期的程式設計師要設計程式是件苦差事，因為得要了解電腦硬體，並根據該電腦硬體來選擇程式語言，然後根據程式語言來設計運算工作、記憶體讀寫工作、 磁碟與影像輸入輸出工作、檔案存取工作等。等於從硬體、軟體、輸入輸出行為都得要在自己的程式碼裡面一口氣完成才行。

在 1971 年開始的 unix 系統開發後，後續的系統大多使用 unix 的概念

程式的執行

作業系統需要將使用者交付的軟體程序分配到記憶體中， 然後透過 CPU 排程持續的交錯的完成各項任務才行。

CPU 中斷 (interrupt) 的功能

CPU 根據硬體擁有許多與週邊硬體的中斷通道， 當接收到中斷訊號時，CPU 就會嘗試將該程序列入等待的狀態下，讓該硬體自行完成相關的任務後，然後再接管系統。

記憶體管理模組

系統會自動去偵測與管理主記憶體的使用狀態，避免同一個記憶體位址同時被兩個程序所使用而讓程序工作損毀

虛擬記憶體 (virtual memory)

主記憶體當中的資料並不是連續的，主記憶體的資料就像磁碟一樣，重複讀、刪、寫之後， 記憶區段是不會連續的

CPU 主要讀出虛擬記憶體，記憶體管理模組就會主動讀出資料

作業系統好不好的重要指標之一！如何讓 CPU 在多工的情況下以最快速的方式將所有的工作完成，這方面的演算法是目前各主要作業系統持續在進步的部份。

磁碟存取與檔案系統

作業系統則需要驅動磁碟(不論是傳統硬碟還是 SSD)，然後也需要了解該磁碟內的檔案系統格式， 之後透過檔案系統這個子系統來進行資料的處理。

裝置的驅動程式

作業系統必須要能夠接受硬體裝置的驅動，所以硬體製造商可以推出給各個不同作業系統使用的驅動程式 (dirver / modules)， 這樣作業系統直接將該驅動程式載入後，即可開始使用該硬體，而不需要重新編譯作業系統。

網路子系統

使用者界面

現代 CPU 設計的主要思考依據，讓一個 CPU 封裝 (單一一顆 CPU 硬體) 裡面，整合多個 CPU 核心，也就是多核心 CPU 製造的思考方向。

對於單執行緒的程式來說， 多核心的 CPU 不見得會跑得比單核的快！這是因為單執行緒只有一個程序在進行，所以 CPU 時脈越高，代表會越快執行完畢。

軟體會將單一工作拆分成數個小工作，分別交給不同的核心去執行，這樣每個核心只要負責一小段任務， 當然 CPU 時脈不用高，只要數量夠大，效能就會提昇很明顯

所謂的平行處理功能，讓一件工作可以拆分成數個部份，讓這些不同的部份丟給不同的 CPU 去運算， 然後再透過一支監控程式，將各別的計算在一定的時間內收回統整後，再次的細分小工作發派出去，持續這些動作後，直到程式執行完畢為止。

對於 Linux 來說，大部分都可以支援到 4096 個 CPU 核心數。

銀行商用大型主機 Unix 系統

記憶體管理（Memory management）

核心 - 作業系統之最核心部分，通常執行在最高特權級，負責提供基礎性、結構性的功能。

驅動程式 - 最底層的、直接控制和監視各類硬體的部分，它們的職責是隱藏硬體的具體細節，並向其他部分提供一個抽象的、通用的介面。

作業系統的分類沒有一個單一的標準，可以根據工作方式分為批次處理作業系統、分時作業系統、即時作業系統、網路作業系統和分散式作業系統等

根據帕金森定律：「你給程式再多記憶體，程式也會想盡辦法耗光」

大部分的現代電腦記憶體架構都是階層式的，最快且數量最少的暫存器為首，然後是快取、記憶體以及最慢的磁碟儲存裝置。

虛擬記憶體管理的功能大幅增加每個行程可獲得的記憶空間

當年運用馮·諾伊曼結構建造電腦時，每個中央處理器最多只能同時執行一個行程。

現代的作業系統，即使只擁有一個CPU，也可以利用多行程（multitask）功能同時執行多個行程。行程管理指的是作業系統調整多個行程的功能。

作業系統尚有擔負起行程間通訊（IPC）、行程異常終止處理以及死結（Dead Lock）偵測及處理等較為艱深的問題。

檔案系統，通常指稱管理磁碟資料的系統，可將資料以目錄或檔案的型式儲存。每個檔案系統都有自己的特殊格式與功能，例如日誌管理或不需磁碟重整。

現代的作業系統都具備操作主流網路通訊協定TCP/IP的能力。也就是說這樣的作業系統可以進入網路世界，並且與其他系統分享諸如檔案、印表機與掃描器等資源。

作業系統提供外界直接或間接存取數種資源的管道

作業系統有能力認證資源存取的請求

一個高安全等級的系統也會提供記錄選項，允許記錄各種請求對資源存取的行為（例如「誰曾經讀了這個檔案？」）

大部分的作業系統都包含圖形化使用者介面（GUI）。有幾類較舊的作業系統將圖形化使用者介面與核心緊密結合，例如最早的Windows與Mac OS實作產品。

驅動程式（Device driver）是指某類設計來與硬體互動的電腦軟體。通常是一設計完善的裝置互動介面，利用與此硬體連接的電腦匯排流或通訊子系統，提供對此裝置下令與接收資訊的功能；以及最終目的，將訊息提供給作業系統或應用程式。

適合的驅動程式一旦安裝，相對應的新裝置就可以無誤地執行。此新驅動程式可以讓此裝置完美地切合在作業系統中，讓使用者察覺不到這是作業系統原本沒有的功能。

从集群中获取每个 pod ip 地址，然后也能在集群内部直接通过 podIP:Port 来获取对应的服务。

pod 是经常变化的，每次更新 ip 地址都可能会发生变化，如果直接访问容器 ip 的话，会有很大的问题。

“服务”（service），每个服务都一个固定的虚拟 ip（这个 ip 也被称为 cluster IP），自动并且动态地绑定后面的 pod，所有的网络请求直接访问服务 ip，服务会自动向后端做转发。

实现 service 这一功能的关键，就是 kube-proxy。

kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1

iptables 完全实现 iptables 来实现 service，是目前默认的方式，也是推荐的方式，效率很高（只有内核中 netfilter 一些损耗）。

可以在终端上启动 kube-proxy，也可以使用诸如 systemd 这样的工具来管理它

kubectl provides the most flexible management of Node Problem Detector.

run the Node Problem Detector in your cluster to monitor node health.

Since source code changes often, the previously cached layer for the ADD instruction is invalidated due to the mismatching checksums.

The ARG instruction in the Dockerfile defines RAILS_ENV variable and is implicitly used as an environment variable by the rest of the instructions defined just after that ARG instruction.

RUN instructions are used to install gems and precompile static assets using sprockets

Instead, Docker automatically re-uses the previously built layer for the RUN bundle install instruction if the Gemfile.lock file remains unchanged.

everyday we need to build a lot of Docker images containing source code from varying Git branches as well as with varying environments.

it is hard for Docker to cache layers for bundle install and rake assets:precompile tasks and re-use those layers during every docker build command run with different application source code and a different environment.

By default, Bundler installs gems at the location which is set by Rubygems.

This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.

differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.

Filebeat is just a tiny binary with no dependencies.

For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

For example, the apache module will point Filebeat to default access.log and error.log paths

Filebeat’s scope is very limited,

Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Filebeat can parse JSON

you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.

You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off

For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well

The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.

rsyslog is the fastest shipper

Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).

It’s also one of the lightest parsers you can find, depending on the configured memory buffers.

rsyslog requires more work to get the configuration right

the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.

rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).

rsyslog also works well when you need that ultimate performance.

syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).

a modular syslog daemon, that can do much more than just syslog

Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.

Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.

syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance

Fluentd plugins are in Ruby and very easy to write.

structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).

Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.

Splunk isn’t a log shipper, it’s a commercial logging solution

everything goes through graylog-server, from authentication to queries.

Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.

it depends

deployment.yaml: A basic manifest for creating a Kubernetes deployment

using the suffix .yaml for YAML files and .tpl for helpers.

helm get manifest

The helm get manifest command takes a release name (full-coral) and prints out all of the Kubernetes resources that were uploaded to the server. Each file begins with --- to indicate the start of a YAML document

Names should be unique to a release

The name: field is limited to 63 characters because of limitations to the DNS system.

release names are limited to 53 characters

{{ .Release.Name }}

The values that are passed into a template can be thought of as namespaced objects, where a dot (.) separates each namespaced element.

The Release object is one of the built-in objects for Helm

Using --dry-run will make it easier to test your code, but it won’t ensure that Kubernetes itself will accept the templates you generate.

Objects are passed into a template from the template engine.

create new objects within your templates

Objects can be simple, and have just one value. Or they can contain other objects or functions.

Release is one of the top-level objects that you can access in your templates.

Release.Namespace: The namespace to be released into (if the manifest doesn’t override)

Chart: The contents of the Chart.yaml file.

Files: This provides access to all non-special files in a chart.

Files.Get is a function for getting a file by name

Files.GetBytes is a function for getting the contents of a file as an array of bytes instead of as a string. This is useful for things like images.

Template: Contains information about the current template that is being executed

BasePath: The namespaced path to the templates directory of the current chart

Go’s naming convention

use only initial lower case letters in order to distinguish local names from those built-in.

If this is a subchart, the values.yaml file of a parent chart

Individual parameters passed with --set

While structuring data this way is possible, the recommendation is that you keep your values trees shallow, favoring flatness.

If you need to delete a key from the default values, you may override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.

Kubernetes would then fail because you can not declare more than one livenessProbe handler.

When injecting strings from the .Values object into the template, we ought to quote these strings.

quote

Template functions follow the syntax functionName arg1 arg2...

While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates.

Drawing on a concept from UNIX, pipelines are a tool for chaining together a series of template commands to compactly express a series of transformations.

pipelines are an efficient way of getting several things done in sequence

The repeat function will echo the given string the given number of times

default DEFAULT_VALUE GIVEN_VALUE. This function allows you to specify a default value inside of the template, in case the value is omitted.

all static default values should live in the values.yaml, and should not be repeated using the default command

Operators are implemented as functions that return a boolean value.

To use eq, ne, lt, gt, and, or, not etcetera place the operator at the front of the statement followed by its parameters just as you would a function.

if and

if or

with to specify a scope

range, which provides a “for each”-style loop

block declares a special kind of fillable template area

A pipeline is evaluated as false if the value is: a boolean false a numeric zero an empty string a nil (empty or null) an empty collection (map, slice, tuple, dict, array)

incorrect YAML because of the whitespacing

When the template engine runs, it removes the contents inside of {{ and }}, but it leaves the remaining whitespace exactly as is.

{{- (with the dash and space added) indicates that whitespace should be chomped left, while -}} means whitespace to the right should be consumed.

Newlines are whitespace!

an * at the end of the line indicates a newline character that would be removed

the indent function

Scopes can be changed. with can allow you to set the current scope (.) to a particular object.

range

The range function will “range over” (iterate through) the pizzaToppings list.

Just like with sets the scope of ., so does a range operator.

The toppings: |- line is declaring a multi-line string.

not a YAML list. It’s a big string.

the data in ConfigMaps data is composed of key/value pairs, where both the key and the value are simple strings.

The |- marker in YAML takes a multi-line string.

range can be used to iterate over collections that have a key and a value (like a map or dict).

In Helm templates, a variable is a named reference to another object. It follows the form $name

Variables are assigned with a special assignment operator: :=

{{- $relname := .Release.Name -}}

capture both the index and the value

the integer index (starting from zero) to $index and the value to $topping

For data structures that have both a key and a value, we can use range to get both

one variable that is always global - $ - this variable will always point to the root context.

$.

$.

Helm template language is its ability to declare multiple templates and use them together.

A named template (sometimes called a partial or a subtemplate) is simply a template defined inside of a file, and given a name.

If you declare two templates with the same name, whichever one is loaded last will be the one used.

naming convention is to prefix each defined template with the name of the chart: {{ define "mychart.labels" }}

Once connected to the server, in order to perform a backup you will need READ and EXECUTE permissions at a filesystem level in the server’s datadir.

/sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location.

an userspace library that processes can use to query their memory usage and available memory.

we might need to encourage people to stop using those tools inside containers.

In full backups, two types of operations are performed to make the database consistent: committed transactions are replayed from the log file against the data files, and uncommitted transactions are rolled back.

You should use the --apply-log-only option to prevent the rollback phase.

An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s LSN.

Incremental backups do not actually compare the data files to the previous backup’s data files.

Incremental backups simply read the pages and compare their LSN to the last backup’s LSN.

The xtrabackup binary writes a file called xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn, which is the database’s LSN at the end of the backup.

from_lsn is the starting LSN of the backup and for incremental it has to be the same as to_lsn (if it is the last checkpoint) of the previous/base backup.

If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups will be useless.

If you restore it and start MySQL, InnoDB will detect that the rollback phase was not performed, and it will do that in the background, as it usually does for a crash recovery upon start.

xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc1

The final data is in /data/backups/base, not in the incremental directory.