Group items tagged

adding pre-commit plugins to your project is done with the .pre-commit-config.yaml configuration file.

The pre-commit config file describes what repositories and hooks are installed.

This configuration says to download the pre-commit-hooks project and run its trailing-whitespace hook

The environment variables are defined as name/value pairs and are injected at runtime.

The CircleCI Docker Layer Caching feature allows builds to reuse Docker image layers

Image layers are stored in separate volumes in the cloud and are not shared between projects.

Environment variables store customer data that is used by a project.

Defines the underlying technology to run a job.

machine to run your job inside a full virtual machine.

docker to run your job inside a Docker container with a specified image

A job is a collection of steps.

The first image listed in config.yml

A CircleCI project shares the name of the code repository for which it automates workflows, tests, and deployment.

must be added with the Add Project button

Following a project enables a user to subscribe to email notifications for the project build status and adds the project to their CircleCI dashboard.

A step is a collection of executable commands

Users must be added to a GitHub or Bitbucket org to view or follow associated CircleCI projects.

A Workflow is a set of rules for defining a collection of jobs and their run order.

Workflows are implemented as a directed acyclic graph (DAG) of jobs for greatest flexibility.

A workspace is a workflows-aware storage mechanism.

a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).

Each Container in the Pod must independently specify where to mount each volume

localnfs

cephfs

awsElasticBlockStore

glusterfs

vsphereVolume

An awsElasticBlockStore volume mounts an Amazon Web Services (AWS) EBS Volume into your Pod.

an EBS volume can be pre-populated with data, and that data can be “handed off” between Pods.

create an EBS volume using aws ec2 create-volume

the nodes on which Pods are running must be AWS EC2 instances

EBS only supports a single EC2 instance mounting a volume

check that the size and EBS volume type are suitable for your use!

A cephfs volume allows an existing CephFS volume to be mounted into your Pod.

the contents of a cephfs volume are preserved and the volume is merely unmounted.

have your own Ceph server running with the share exported

The configMap resource provides a way to inject configuration data into Pods

When referencing a configMap object, you can simply provide its name in the volume to reference it

volumeMounts: - name: config-vol mountPath: /etc/config volumes: - name: config-vol configMap: name: log-config items: - key: log_level path: log_level

create a ConfigMap before you can use it.

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node.

By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment.

you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem)

volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {}

An fc volume allows an existing fibre channel volume to be mounted in a Pod.

configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

Flocker is an open-source clustered Container data volume manager. It provides management and orchestration of data volumes backed by a variety of storage backends.

emptyDir

flocker

A flocker volume allows a Flocker dataset to be mounted into a Pod

have your own Flocker installation running

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your Pod.

Using a PD on a Pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1

A glusterfs volume allows a Glusterfs (an open source networked filesystem) volume to be mounted into your Pod.

have your own GlusterFS installation running

a powerful escape hatch for some applications

access to Docker internals; use a hostPath of /var/lib/docker

allowing a Pod to specify whether a given hostPath should exist prior to the Pod running, whether it should be created, and what it should exist as

specify a type for a hostPath volume

the files or directories created on the underlying hosts are only writable by root.

hostPath: # directory location on host path: /data # this field is optional type: Directory

An iscsi volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.

have your own iSCSI server running

A feature of iSCSI is that it can be mounted as read-only by multiple consumers simultaneously.

A local volume represents a mounted local storage device such as a disk, partition or directory.

Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.

PersistentVolume spec using a local volume and nodeAffinity

PersistentVolume volumeMode can now be set to “Block” (instead of the default value “Filesystem”) to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer

An nfs volume allows an existing NFS (Network File System) share to be mounted into your Pod.

have your own NFS server running with the share exported

A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.

PersistentVolumes are a way for users to “claim” durable storage (such as a GCE PersistentDisk or an iSCSI volume) without knowing the details of the particular cloud environment.

A projected volume maps several existing volume sources into the same directory.

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Each projected volume source is listed in the spec under sources

A Container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.

RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed

A secret volume is used to pass sensitive information, such as passwords, to Pods

create a secret in the Kubernetes API before you can use it

StorageOS runs as a Container within your Kubernetes environment, making local or attached storage accessible from any node within the Kubernetes cluster.

StorageOS provides block storage to Containers, accessible via a file system.

A vsphereVolume is used to mount a vSphere VMDK Volume into your Pod.

supports both VMFS and VSAN datastore.

create VMDK using one of the following methods before using with Pod.

share one volume for multiple uses in a single Pod.

volumeMounts: - name: workdir1 mountPath: /logs subPathExpr: $(POD_NAME)

env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name

Use the subPathExpr field to construct subPath directory names from Downward API environment variables

enable the VolumeSubpathEnvExpansion feature gate

emptyDir and hostPath volumes will be able to request a certain amount of space using a resource specification, and to select the type of media to use, for clusters that have several media types.

the Container Storage Interface (CSI) and Flexvolume. They enable storage vendors to create custom storage plugins without adding them to the Kubernetes repository.

Container Storage Interface (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the csi volume type to attach, mount, etc. the volumes exposed by the CSI driver.

This feature requires CSIInlineVolume feature gate to be enabled:--feature-gates=CSIInlineVolume=true

In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the “Types of Volumes” section above.

Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts.

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the Container will be propagated back to the host and to all Containers of all Pods that use the same volume.

Edit your Docker’s systemd service file. Set MountFlags as follows:MountFlags=shared

其實 Auto DevOps 就是一份官方寫好的 gitlab-ci.yml，在啟動 Auto DevOps 的專案裡，如果找不到 gitlab-ci.yml 檔，那就會直接用官方 gitlab-ci.yml 去跑 CI / CD 流程。

Pod 是 K8S 中可以被部署的最小元件，一個 Pod 是由一到多個 Container 組成，同個 Pod 的不同 Container 之間彼此共享網路資源。

Node 又分成 Worker Node 和 Master Node 兩種

Helm 透過參數 (parameter) 跟模板 (template) 的方式，讓我們可以在只修改參數的方式重複利用模板。

為了要有 CI CD 的功能我們會把 .gitlab-ci.yml 放在專案的根目錄裡， GitLab 會依造 .gitlab-ci.yml 的設定產生 CI/CD Pipeline，每個 Pipeline 裡面可能有多個 Job，這時候就會需要有 GitLab Runner 來執行這些 Job 並把執行的結果回傳給 GitLab 讓它知道這個 Job 是否有正常執行。

把專案打包成 Docker Image 這工作又或是 helm 的操作都會在 Container 內執行

CI/CD Pipeline 是由 stage 還有 job 組成的，stage 是有順序性的，前面的 stage 完成後才會開始下一個 stage。

每個 stage 裡面包含一到多個 Job

Auto Devops 裡也會大量用到這種在指定 Container 內運行的工作。

可以通過 health checks

開 private 的話還要注意使用 Container Registry 的權限問題

申請好的 wildcard 的 DNS

Auto Devops 也提供只要設定環境變數就能一定程度客製化的選項

特別注意 namespace 有沒有設定對，不然會找不到資料喔

Auto Devops，如果想要進一步的客製化，而且是改 GitLab 環境變數都無法實現的客製化，這時候還是得回到 .gitlab-ci.yml 設定檔

在 Docker in Docker 的環境用 Dockerfile 打包 Image

用 helm upgrade 把 chart 部署到 K8S 上

把專案打包成 Docker Image 首先需要在專案下新增一份 Dockerfile

在 Runner 的環境中是沒有 docker 指令可以用的，所以這邊啟動一個 Docker Container 在裡面執行就可以用 docker 指令了。

其中 $CI_COMMIT_SHA $CI_COMMIT_BEFORE_SHA 這兩個都是 GitLab 預設環境變數，代表這次 commit 還有上次 commit 的 SHA 值。

dind 則是直接啟動 docker daemon，此外 dind 還會自動產生 TLS certificates

為了在 Docker Container 內運行 Docker，會把 Host 上面的 Docker API 分享給 Container。

docker:stable 有執行 docker 需要的執行檔，他裡面也包含要啟動 docker 的程式(docker daemon)，但啟動 Container 的 entrypoint 是 sh

docker:dind 繼承自 docker:stable，而且它 entrypoint 就是啟動 docker 的腳本，此外還會做完 TLS certificates

Container 要去連 Host 上的 Docker API 。但現在連線失敗卻是找 http://docker:2375，現在的 dind 已經不是被當做 services 來用了，而是要直接在裡面跑 Docker，所以他應該是要 unix:///var/run/docker.sock 用這種連線，於是把環境變數 DOCKER_HOST 從 tcp://docker:2375 改成空字串，讓 docker daemon 走預設連線就能成功囉！

auto-deploy preparationhelm init 建立 helm 專案設定 tiller 在背景執行設定 cluster 的 namespace

auto-deploy deploy使用 helm upgrade 部署 chart 到 K8S 上透過 --set 來設定要注入 template 的參數

set -x，這樣就能在執行前，顯示指令內容。

用 helm repo list 看看現在有註冊哪些 Chart Repository

helm fetch gitlab/auto-deploy-app --untar

nohup 可以讓你在離線或登出系統後，還能夠讓工作繼續進行

在不特別設定 CI_APPLICATION_REPOSITORY 的情況下，image_repository 的值就是預設環境變數 CI_REGISTRY_IMAGE/CI_COMMIT_REF_SLUG

A:-B 的意思是如果有 A 就用它，沒有就用 B

研究 Auto Devops 難度最高的地方就是太多工具整合在一起，搞不清楚他們之間的關係，出錯也不知道從何查起

The lock file is always named .terraform.lock.hcl, and this name is intended to signify that it is a lock file for various items that Terraform caches in the .terraform

Terraform automatically creates or updates the dependency lock file each time you run the terraform init command.

If a particular provider has no existing recorded selection, Terraform will select the newest available version that matches the given version constraint, and then update the lock file to include that selection.

the "trust on first use" model

you can pre-populate checksums for a variety of different platforms in your lock file using the terraform providers lock command, which will then allow future calls to terraform init to verify that the packages available in your chosen mirror match the official packages from the provider's origin registry.

The h1: and zh: prefixes on these values represent different hashing schemes, each of which represents calculating a checksum using a different algorithm.

zh:: a mnemonic for "zip hash"

h1:: a mnemonic for "hash scheme 1", which is the current preferred hashing scheme.

To determine whether there still exists a dependency on a given provider, Terraform uses two sources of truth: the configuration itself, and the state.

Version constraints within the configuration itself determine which versions of dependencies are potentially compatible, but after selecting a specific version of each dependency Terraform remembers the decisions it made in a dependency lock file so that it can (by default) make the same decisions again in future.

The lock file is always named .terraform.lock.hcl

Developers rely heavily on app logs via syslog (mounted /dev/log) and Alpine uses busybox syslog by default.

Ubuntu officially launched minimal ubuntu images for cloud / container use

use all rules keywords, like if, changes, and exists, in the same rule. The rule evaluates to true only when all included keywords evaluate to true.

use parentheses with && and || to build more complicated variable expressions.

Use workflow to specify which types of pipelines can run.

every push to an open merge request’s source branch causes duplicated pipelines.

avoid duplicate pipelines by changing the job rules to avoid either push (branch) pipelines or merge request pipelines.

For behavior similar to the only/except keywords, you can check the value of the $CI_PIPELINE_SOURCE variable

commonly used variables for if clauses

rules:changes expressions to determine when to add jobs to a pipeline

Use !reference tags to reuse rules in different jobs.

Use except to define when a job does not run.

only or except used without refs is the same as only:refs / except/refs

If you change multiple files, but only one file ends in .md, the build job is still skipped.

If you use multiple keywords with only or except, the keywords are evaluated as a single conjoined expression.

only includes the job if all of the keys have at least one condition that matches.

except excludes the job if any of the keys have at least one condition that matches.

To specify a job as manual, add when: manual to the job in the .gitlab-ci.yml file.

Use protected environments to define a list of users authorized to run a manual job.

Use when: delayed to execute scripts after a waiting period, or if you want to avoid jobs immediately entering the pending state.

To split a large job into multiple smaller jobs that run in parallel, use the parallel keyword

run a trigger job multiple times in parallel in a single pipeline, but with different variable values for each instance of the job.

The @ symbol denotes the beginning of a ref’s repository path. To match a ref name that contains the @ character in a regular expression, you must use the hex character code match \x40.

Compare a variable to a string

Check if a variable is undefined

Check if a variable is empty

Check if a variable exists

Check if a variable is empty

Matches are found when using =~.

Matches are not found when using !~.

Join variable expressions together with && or ||