Skip to main content

Home/ Larvata/ Group items tagged linux

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
張 旭

打造高效的工作环境 - Shell 篇 | | 酷 壳 - CoolShell - 0 views

coolshell.cn/19219.html

system shell linux devops

shared by 張 旭 on 25 Mar 19 - No Cached
  • 对于一些字符串变量，使用引号括起，避免其中有空格或是别的什么诡异字符。
  • 使用 $() 而不是 “ 来获得命令行的输出，主要原因是易读。
  • 对于 rm -rf 这样的高危操作，需要检查后面的变量名是否为空
  • ...2 more annotations...
  • find . -type f | while read F ; do echo $F ; done 不但可以容忍空格，而且还更快。
  • 防御式编程，在正式执行命令前，把相关的东西都检查好，比如，文件目录有没有存在。
張 旭

Baseimage-docker: A minimal Ubuntu base image modified for Docker-friendliness - 0 views

phusion.github.io/baseimage-docker

docker base system linux

shared by 張 旭 on 29 Oct 14 - No Cached
  • We encourage you to use multiple processes.
  • Baseimage-docker is a special Docker image that is configured for correct use within Docker containers.
  • When your Docker container starts, only the CMD command is run.
  • ...16 more annotations...
  • You're not running them, you're only running your app.
  • You have Ubuntu installed in Docker. The files are there. But that doesn't mean Ubuntu's running as it should.
  • The only processes that will be running inside the container is the CMD command, and all processes that it spawns.
  • A proper Unix system should run all kinds of important system services.
  • Ubuntu is not designed to be run inside Docker
  • When a system is started, the first process in the system is called the init process, with PID 1. The system halts when this processs halts.
  • Runit (written in C) is much lighter weight than supervisord (written in Python).
  • Docker runs fine with multiple processes in a container.
  • Baseimage-docker encourages you to run multiple processes through the use of runit.
  • If your init process is your app, then it'll probably only shut down itself, not all the other processes in the container.
  • a Docker container, which is a locked down environment with e.g. no direct access to many kernel resources.
  • Used for service supervision and management.
  • A custom tool for running a command as another user.
  • add additional daemons (e.g. your own app) to the image by creating runit entries.
  • write a small shell script which runs your daemon, and runit will keep it up and running for you, restarting it when it crashes, etc.
  • the shell script must run the daemon without letting it daemonize/fork it.
張 旭

你到底知不知道什麼是 Kubernetes? | Hwchiu Learning Note - 0 views

www.hwchiu.com/kubernetes-concept.html

kubernetes storage networking programming system

shared by 張 旭 on 03 Feb 19 - No Cached
  • Storage(儲存) 實際上一直都不是一個簡單處理的問題，從軟體面來看實際上牽扯到非常多的層級，譬如 Linux Kernel, FileSystem, Block/File-Level, Cache, Snapshot, Object Storage 等各式各樣的議題可以討論。
  • DRBD
  • 異地備援，容錯機制，快照，重複資料刪除等超多相關的議題基本上從來沒有一個完美的解法能夠滿足所有使用情境。
  • ...20 more annotations...
  • 管理者可能會直接在 NFS Server 上進行 MDADM 來設定相關的 Block Device 並且基於上面提供 Export 供 NFS 使用，甚至底層套用不同的檔案系統 (EXT4/BTF4) 來獲取不同的功能與效能。
  • Kubernetes 就只是 NFS Client 的角色
  • CSI(Container Storage Interface)。CSI 本身作為 Kubernetes 與 Storage Solution 的中介層。
  • 基本上 Pod 裡面每個 Container 會使用 Volume 這個物件來代表容器內的掛載點，而在外部實際上會透過 PVC 以及 PV 的方式來描述這個 Volume 背後的儲存方案伺服器的資訊。
  • 整體會透過 CSI 的元件們與最外面實際上的儲存設備連接，所有儲存相關的功能是否有實現，有支援全部都要仰賴最後面的實際提供者， kubernetes 只透過 CSI 的標準去執行。
  • 在網路部分也有與之對應的 CNI(Container Network Interface). kubernetes 透過 CNI 這個介面來與後方的 網路解決方案 溝通
  • CNI 最基本的要求就是在在對應的階段為對應的容器提供網路能力
  • 目前最常見也是 IPv4 + TCP/UDP 的傳輸方式，因此才會看到大部分的 CNI 都在講這些。
  • 希望所有容器彼此之間可以透過 IPv4 來互相存取彼此，不論是同節點或是跨節點的容器們都要可以滿足這個需求。
  • 容器間到底怎麼傳輸的，需不需要封裝，透過什麼網卡，要不要透過 NAT 處理? 這一切都是 CNI 介面背後的實現
  • 外部網路存取容器服務 (Service/Ingress)
  • kubernetes 在 Service/Ingress 中間自行實現了一個模組，大抵上稱為 kube-proxy, 其底層可以使用 iptables, IPVS, user-space software 等不同的實現方法，這部分是跟 CNI 完全無關。
  • CNI 跟 Service/Ingress 是會衝突的，也有可能彼此沒有配合，這中間沒有絕對的穩定整合。
  • CNI 一般會處理的部份，包含了容器內的 網卡數量,網卡名稱,網卡IP, 以及容器與外部節點的連接能力等
  • CRI (Container Runtime Interface) 或是 Device Plugin
  • 對於 kubernetes 來說，其實本身並不在意到底底下的容器化技術實際上是怎麼實現的，你要用 Docker, rkt, CRI-O 都無所謂，甚至背後是一個偽裝成 Container 的 Virtaul Machine virtlet 都可以。
  • 去思考到底為什麼自己本身的服務需要容器化，容器化可以帶來什麼優點
  • 太多太多的人都認為只要寫一個 Dockerfile 將原先的應用程式們全部包裝起來放在一起就是一個很好的容器 來使用了。
  • 最後就會發現根本把 Container 當作 Virtual Machine 來使用，然後再補一句 Contaienr 根本不好用啊
  • 容器化 不是把直接 Virtual Machine 的使用習慣換個環境使用就叫做 容器化，而是要從概念上去暸解與使用
crazylion lee

Idle Time » Blog Archive » Colorized man pages: Understood and customized - 0 views

boredzo.org/...ages-understood-and-customized

shell unix linux pages man zsh

shared by crazylion lee on 24 Aug 16 - No Cached
  • crazylion lee on 24 Aug 16
     
    "Colorized man pages: Understood and customized"
張 旭

redis cluster管理工具redis-trib.rb详解 | 魏子珺的博客 - 0 views

weizijun.cn/...edis-trib-rb%E8%AF%A6%E8%A7%A3

linux redis cache cluster

shared by 張 旭 on 26 Feb 18 - No Cached
  • reshard命令可以在线把集群的一些slot从集群原来slot负责节点迁移到新的节点，利用reshard可以完成集群的在线横向扩容和缩容。
張 旭

Pods - Kubernetes - 0 views

kubernetes.io/...pod

kubernetes pod system

shared by 張 旭 on 05 Jul 19 - No Cached
  • Pods are the smallest deployable units of computing
  • A Pod (as in a pod of whales or pea pod) is a group of one or more containersA lightweight and portable executable image that contains software and all of its dependencies. (such as Docker containers), with shared storage/network, and a specification for how to run the containers.
  • A Pod’s contents are always co-located and co-scheduled, and run in a shared context.
  • ...32 more annotations...
  • A Pod models an application-specific “logical host”
  • application containers which are relatively tightly coupled
  • being executed on the same physical or virtual machine would mean being executed on the same logical host.
  • The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation
  • Containers within a Pod share an IP address and port space, and can find each other via localhost
  • Containers in different Pods have distinct IP addresses and can not communicate by IPC without special configuration. These containers usually communicate with each other via Pod IP addresses.
  • Applications within a Pod also have access to shared volumesA directory containing data, accessible to the containers in a pod. , which are defined as part of a Pod and are made available to be mounted into each application’s filesystem.
  • a Pod is modelled as a group of Docker containers with shared namespaces and shared filesystem volumes
    • 張 旭
      張 旭 on 05 Jul 19
       
      類似 docker-compose 裡面宣告的同一坨？
  • Pods are considered to be relatively ephemeral (rather than durable) entities.
  • Pods are created, assigned a unique ID (UID), and scheduled to nodes where they remain until termination (according to restart policy) or deletion.
  • it can be replaced by an identical Pod
  • When something is said to have the same lifetime as a Pod, such as a volume, that means that it exists as long as that Pod (with that UID) exists.
  • uses a persistent volume for shared storage between the containers
  • Pods serve as unit of deployment, horizontal scaling, and replication
  • The applications in a Pod all use the same network namespace (same IP and port space), and can thus “find” each other and communicate using localhost
  • flat shared networking space
  • Containers within the Pod see the system hostname as being the same as the configured name for the Pod.
  • Volumes enable data to survive container restarts and to be shared among the applications within the Pod.
  • Individual Pods are not intended to run multiple instances of the same application
  • The individual containers may be versioned, rebuilt and redeployed independently.
  • Pods aren’t intended to be treated as durable entities.
  • Controllers like StatefulSet can also provide support to stateful Pods.
  • When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container.
  • Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server.
  • grace period
  • Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers.
  • When the grace period expires, any processes still running in the Pod are killed with SIGKILL.
  • By default, all deletes are graceful within 30 seconds.
  • You must specify an additional flag --force along with --grace-period=0 in order to perform force deletions.
  • Force deletion of a Pod is defined as deletion of a Pod from the cluster state and etcd immediately.
  • StatefulSet Pods
  • Processes within the container get almost the same privileges that are available to processes outside a container.
張 旭

如何在 Ubuntu 18.04 下正确配置网络 - 运维之美 - 0 views

www.hi-linux.com/...49513.html

networking ubuntu server network

shared by 張 旭 on 09 Apr 20 - No Cached
  • 張 旭 on 09 Apr 20
     
    "systemd-resolve --status"
張 旭

Rails Environment Variables · RailsApps - 1 views

railsapps.github.io/ils-environment-variables.html

rails environment system variable

shared by 張 旭 on 13 Mar 17 - No Cached
  • You can pass local configuration settings to an application using environment variables.
  • Operating systems (Linux, Mac OS X, Windows) provide mechanisms to set local environment variables, as does Heroku and other deployment platforms.
  • In general, you shouldn’t save email account credentials or private API keys to a shared git repository.
  • ...10 more annotations...
  • You could “hardcode” your Gmail username and password into the file but that would expose it to everyone who has access to your git repository.
  • It’s important to learn to use the Unix shell if you’re commited to improving your skills as a developer.
  • The gem reads a config/application.yml file and sets environment variables before anything else is configured in the Rails application.
  • make sure this file is listed in the .gitignore file so it isn’t checked into the git repository
  • Rails provides a config.before_configuration
  • YAML.load(File.open(env_file)).each do |key, value| ENV[key.to_s] = value end if File.exists?(env_file)
  • Heroku is a popular choice for low cost, easily configured Rails application hosting.
  • heroku config:add
  • the dotenv Ruby gem
  • Foreman is a tool for starting and configuring multiple processes in a complex application
張 旭

Upgrading kubeadm clusters | Kubernetes - 0 views

kubernetes.io/...kubeadm-upgrade

kubernetes system

shared by 張 旭 on 02 Jun 21 - No Cached
  • Swap must be disabled.
  • read the release notes carefully.
  • back up any important components, such as app-level state stored in a database.
  • ...16 more annotations...
  • All containers are restarted after upgrade, because the container spec hash value is changed.
  • The upgrade procedure on control plane nodes should be executed one node at a time.
  • /etc/kubernetes/admin.conf
  • kubeadm upgrade also automatically renews the certificates that it manages on this node. To opt-out of certificate renewal the flag --certificate-renewal=false can be used.
  • Manually upgrade your CNI provider plugin.
  • sudo systemctl daemon-reload sudo systemctl restart kubelet
  • If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again.
  • To recover from a bad state, you can also run kubeadm upgrade apply --force without changing the version that your cluster is running.
  • kubeadm-backup-etcd contains a backup of the local etcd member data for this control plane Node.
  • the contents of this folder can be manually restored in /var/lib/etcd
  • kubeadm-backup-manifests contains a backup of the static Pod manifest files for this control plane Node.
  • the contents of this folder can be manually restored in /etc/kubernetes/manifests
  • Enforces the version skew policies.
  • Upgrades the control plane components or rollbacks if any of them fails to come up.
  • Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days.
  • backup folders under /etc/kubernetes/tmp
張 旭

第 06 章 - 計算機概論 - 作業系統概論 - 0 views

dic.vbird.tw/...2020unit06.php

system

shared by 張 旭 on 22 Jun 21 - No Cached
  • 自行參考電腦硬體來設計出運算的軟體，當時的系統並沒有『作業系統』的概念，因為應用程式與作業系統是同時設計的。
  • 電腦裡面有儲存設備 (不論是硬碟還是記憶體)， 所以電腦硬體裡面會執行一隻監督程式 (monitor)，使用者可以預先將自己的程式讀進系統，系統先儲存該程式到佇列 (queue)，等到輪到該程式運作後， 就將該程式讀入讓 CPU 開始運作，直到運作結束輸出到印表機之後，將該工作丟棄，然後開始讀入在 queue 裡面的新的程式，依序執行。
  • 將 CPU 與 I/O 分離開
  • ...38 more annotations...
  • 透過卡片與讀卡機，將程式碼一次性的讀進大機器，然後就是等待大機器的運作， 結果再交由印表機印出。如果打卡紙打洞錯誤呢？只好重新打洞，重新排隊去運作程式了。
  • 允許兩個以上的程序在記憶體中等待被 CPU 執行，當 CPU 執行完其中一隻程式後， 第二隻程式就可以立刻被執行，因此效能會比較好。
  • 程序的狀態進入中斷狀態，CPU 不會理會該程序
  • CPU 的排程 (cpu scheduling)
  • 早期單核 CPU 的運作中，CPU 一次只能運作一個工作，因此，若有多個工作要同時進行， 那麼 CPU 就得要安排一個 CPU 運作時間給所有的工作，當該程序達到最大工作時間後，CPU 就會將該工作排回佇列，讓下一隻程序接著運作。
  • 你會覺得 CPU 是同時運作所有的程序，其實不是的！而是 CPU 在各個程序之間切換工作而已。
  • 分時系統其實與多元程式處理系統有點類似， 只是工作的輸入改為透過終端機操作輸入，CPU 可以在各個用戶操作間切換工作，於是每個使用者感覺似乎都是在同步操作電腦系統一般， 這就是分時系統。
  • 早期的程式設計師要設計程式是件苦差事，因為得要了解電腦硬體，並根據該電腦硬體來選擇程式語言，然後根據程式語言來設計運算工作、記憶體讀寫工作、 磁碟與影像輸入輸出工作、檔案存取工作等。等於從硬體、軟體、輸入輸出行為都得要在自己的程式碼裡面一口氣完成才行。
  • 在 1971 年開始的 unix 系統開發後，後續的系統大多使用 unix 的概念
  • 將硬體管理的工作統一交給一組程式碼去進行，而且這組程式碼還提供了一個開發界面
  • 軟體工程師只要依據這組程式碼規範的開發界面後，該軟體開發完成就能夠在這組程式碼上面運作了
  • 程式的執行
  • 作業系統需要將使用者交付的軟體程序分配到記憶體中， 然後透過 CPU 排程持續的交錯的完成各項任務才行。
  • CPU 中斷 (interrupt) 的功能
  • CPU 根據硬體擁有許多與週邊硬體的中斷通道， 當接收到中斷訊號時，CPU 就會嘗試將該程序列入等待的狀態下，讓該硬體自行完成相關的任務後，然後再接管系統。
  • 記憶體管理模組
  • 舊的環境底下，程式設計師需要自己判斷自己的程式會用到多少記憶體，然後自行指定記憶體使用位址的任務。
  • 系統會自動去偵測與管理主記憶體的使用狀態，避免同一個記憶體位址同時被兩個程序所使用而讓程序工作損毀
  • 作業系統核心也在記憶體中， 因此核心也會被這個子系統放入受保護的記憶體區段，一般用戶是無法直接操作該受保護的記憶區段的。
  • 虛擬記憶體 (virtual memory)
  • 主記憶體當中的資料並不是連續的，主記憶體的資料就像磁碟一樣，重複讀、刪、寫之後， 記憶區段是不會連續的
  • CPU 主要讀出虛擬記憶體，記憶體管理模組就會主動讀出資料
  • 一隻程序的資料是連續的 (左側)，但是實際上對應的是在主記憶體或其他位置上
  • CPU 排程
  • 作業系統好不好的重要指標之一！如何讓 CPU 在多工的情況下以最快速的方式將所有的工作完成，這方面的演算法是目前各主要作業系統持續在進步的部份。
  • 磁碟存取與檔案系統
  • 作業系統則需要驅動磁碟(不論是傳統硬碟還是 SSD)，然後也需要了解該磁碟內的檔案系統格式， 之後透過檔案系統這個子系統來進行資料的處理。
  • 裝置的驅動程式
  • 作業系統必須要能夠接受硬體裝置的驅動，所以硬體製造商可以推出給各個不同作業系統使用的驅動程式 (dirver / modules)， 這樣作業系統直接將該驅動程式載入後，即可開始使用該硬體，而不需要重新編譯作業系統。
  • 網路子系統
  • 使用者界面
  • 現代 CPU 設計的主要思考依據，讓一個 CPU 封裝 (單一一顆 CPU 硬體) 裡面，整合多個 CPU 核心，也就是多核心 CPU 製造的思考方向。
  • 對於單執行緒的程式來說， 多核心的 CPU 不見得會跑得比單核的快！這是因為單執行緒只有一個程序在進行，所以 CPU 時脈越高，代表會越快執行完畢。
  • 軟體會將單一工作拆分成數個小工作，分別交給不同的核心去執行，這樣每個核心只要負責一小段任務， 當然 CPU 時脈不用高，只要數量夠大，效能就會提昇很明顯
  • 由於 CPU 是由作業系統控制的，因此，你要使用到多核心的硬體系統，你的作業系統、應用程式都需要設計程可以支援多核心才行！
  • 所謂的平行處理功能，讓一件工作可以拆分成數個部份，讓這些不同的部份丟給不同的 CPU 去運算， 然後再透過一支監控程式，將各別的計算在一定的時間內收回統整後，再次的細分小工作發派出去，持續這些動作後，直到程式執行完畢為止。
  • 對於 Linux 來說，大部分都可以支援到 4096 個 CPU 核心數。
  • 銀行商用大型主機 Unix 系統
張 旭

作業系統 - 維基百科，自由的百科全書 - 0 views

zh.wikipedia.org/...%8D%E4%BD%9C%E7%B3%BB%E7%BB%9F

system linux

shared by 張 旭 on 22 Jun 21 - No Cached
  • 作業系統位於底層硬體與使用者之間，是兩者溝通的橋樑。
  • 行程管理（Processing management）
  • 安全機制（Security）
  • ...20 more annotations...
  • 記憶體管理（Memory management）
  • 核心 - 作業系統之最核心部分，通常執行在最高特權級，負責提供基礎性、結構性的功能。
  • 驅動程式 - 最底層的、直接控制和監視各類硬體的部分，它們的職責是隱藏硬體的具體細節，並向其他部分提供一個抽象的、通用的介面。
  • 作業系統的分類沒有一個單一的標準，可以根據工作方式分為批次處理作業系統、分時作業系統、即時作業系統、網路作業系統和分散式作業系統等
  • 根據帕金森定律：「你給程式再多記憶體，程式也會想盡辦法耗光」
  • 大部分的現代電腦記憶體架構都是階層式的，最快且數量最少的暫存器為首，然後是快取、記憶體以及最慢的磁碟儲存裝置。
  • 虛擬記憶體管理的功能大幅增加每個行程可獲得的記憶空間
  • 當年運用馮·諾伊曼結構建造電腦時，每個中央處理器最多只能同時執行一個行程。
  • 現代的作業系統，即使只擁有一個CPU，也可以利用多行程（multitask）功能同時執行多個行程。行程管理指的是作業系統調整多個行程的功能。
  • 作業系統尚有擔負起行程間通訊（IPC）、行程異常終止處理以及死結（Dead Lock）偵測及處理等較為艱深的問題。
  • 檔案系統，通常指稱管理磁碟資料的系統，可將資料以目錄或檔案的型式儲存。每個檔案系統都有自己的特殊格式與功能，例如日誌管理或不需磁碟重整。
  • 現代的作業系統都具備操作主流網路通訊協定TCP/IP的能力。也就是說這樣的作業系統可以進入網路世界，並且與其他系統分享諸如檔案、印表機與掃描器等資源。
  • 作業系統提供外界直接或間接存取數種資源的管道
  • 作業系統有能力認證資源存取的請求
  • 通常是一個正在執行的程式發出的資源請求。在某些系統上，一個程式一旦可執行就可做任何事情（例如DOS時代的病毒），但通常作業系統會給程式一個識別代號，並且在此程式發出請求時，檢查其代號與所需資源的存取權限關係。
  • 一個高安全等級的系統也會提供記錄選項，允許記錄各種請求對資源存取的行為（例如「誰曾經讀了這個檔案？」）
  • 大部分的作業系統都包含圖形化使用者介面（GUI）。有幾類較舊的作業系統將圖形化使用者介面與核心緊密結合，例如最早的Windows與Mac OS實作產品。
  • 驅動程式（Device driver）是指某類設計來與硬體互動的電腦軟體。通常是一設計完善的裝置互動介面，利用與此硬體連接的電腦匯排流或通訊子系統，提供對此裝置下令與接收資訊的功能；以及最終目的，將訊息提供給作業系統或應用程式。
  • 驅動程式是針對特定硬體與特定作業系統設計的軟體，通常以作業系統核心模組、應用軟體包或普通電腦程式的形式在作業系統核心底下執行，以達到通透順暢地與硬體互動的效果
  • 適合的驅動程式一旦安裝，相對應的新裝置就可以無誤地執行。此新驅動程式可以讓此裝置完美地切合在作業系統中，讓使用者察覺不到這是作業系統原本沒有的功能。
  • 張 旭 on 22 Jun 21
     
    "作業系統位於底層硬體與使用者之間，是兩者溝通的橋樑。"
張 旭

The differences between Docker, containerd, CRI-O and runc - Tutorial Works - 0 views

www.tutorialworks.com/ocker-containerd-runc-crio-oci

docker container devops

shared by 張 旭 on 28 Jun 21 - No Cached
  • Docker isn’t the only container contender on the block.
  • Container Runtime Interface (CRI), which defines an API between Kubernetes and the container runtime
  • Open Container Initiative (OCI) which publishes specifications for images and containers.
  • ...20 more annotations...
  • for a lot of people, the name “Docker” itself is synonymous with the word “container”.
  • Docker created a very ergonomic (nice-to-use) tool for working with containers – also called docker.
  • docker is designed to be installed on a workstation or server and comes with a bunch of tools to make it easy to build and run containers as a developer, or DevOps person.
  • containerd: This is a daemon process that manages and runs containers.
  • runc: This is the low-level container runtime (the thing that actually creates and runs containers).
  • libcontainer, a native Go-based implementation for creating containers.
  • Kubernetes includes a component called dockershim, which allows it to support Docker.
  • Kubernetes prefers to run containers through any container runtime which supports its Container Runtime Interface (CRI).
  • Kubernetes will remove support for Docker directly, and prefer to use only container runtimes that implement its Container Runtime Interface.
  • Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.
  • Docker images, are actually images packaged in the Open Container Initiative (OCI) format.
  • CRI is the API that Kubernetes uses to control the different runtimes that create and manage containers.
  • CRI makes it easier for Kubernetes to use different container runtimes
  • containerd is a high-level container runtime that came from Docker, and implements the CRI spec
  • containerd was separated out of the Docker project, to make Docker more modular.
  • CRI-O is another high-level container runtime which implements the Container Runtime Interface (CRI).
  • The idea behind the OCI is that you can choose between different runtimes which conform to the spec.
  • runc is an OCI-compatible container runtime.
  • A reference implementation is a piece of software that has implemented all the requirements of a specification or standard.
  • runc provides all of the low-level functionality for containers, interacting with existing low-level Linux features, like namespaces and control groups.
張 旭

Logstash Alternatives: Pros & Cons of 5 Log Shippers [2019] - Sematext - 0 views

sematext.com/...logstash-alternatives

log monitor system logstash fluentd

shared by 張 旭 on 05 Nov 19 - No Cached
  • In this case, Elasticsearch. And because Elasticsearch can be down or struggling, or the network can be down, the shipper would ideally be able to buffer and retry
  • Logstash is typically used for collecting, parsing, and storing logs for future use as part of log management.
  • Logstash’s biggest con or “Achille’s heel” has always been performance and resource consumption (the default heap size is 1GB).
  • ...37 more annotations...
  • This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.
  • Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.
  • differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.
  • Filebeat is just a tiny binary with no dependencies.
  • For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.
  • For example, the apache module will point Filebeat to default access.log and error.log paths
  • Filebeat’s scope is very limited,
  • Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.
  • Filebeat can parse JSON
  • you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.
  • You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off
  • For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well
  • The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.
  • It can tail files, parse them, buffer (on disk and in memory) and ship to a number of destinations, including Elasticsearch.
  • rsyslog is the fastest shipper
  • Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).
  • use it as a simple router/shipper, any decent machine will be limited by network bandwidth
  • It’s also one of the lightest parsers you can find, depending on the configured memory buffers.
  • rsyslog requires more work to get the configuration right
  • the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.
  • rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).
  • rsyslog also works well when you need that ultimate performance.
  • syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).
  • a modular syslog daemon, that can do much more than just syslog
  • Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.
  • Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.
  • syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance
  • Fluentd was built on the idea of logging in JSON wherever possible (which is a practice we totally agree with) so that log shippers down the line don’t have to guess which substring is which field of which type.
  • Fluentd plugins are in Ruby and very easy to write.
  • structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).
  • Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.
  • Fluentd is a good fit when you have diverse or exotic sources and destinations for your logs, because of the number of plugins.
  • Splunk isn’t a log shipper, it’s a commercial logging solution
  • Graylog is another complete logging solution, an open-source alternative to Splunk.
  • everything goes through graylog-server, from authentication to queries.
  • Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.
  • it depends
張 旭

Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation - 0 views

docs.docker.com/...rootless

docker kubernetes system linux

shared by 張 旭 on 04 Aug 20 - No Cached
  • running the Docker daemon and containers as a non-root user
  • Rootless mode does not require root privileges even during the installation of the Docker daemon
  • Rootless mode executes the Docker daemon and containers inside a user namespace.
  • ...9 more annotations...
  • in rootless mode, both the daemon and the container are running without root privileges.
  • Rootless mode does not use binaries with SETUID bits or file capabilities, except newuidmap and newgidmap, which are needed to allow multiple UIDs/GIDs to be used in the user namespace.
  • expose privileged ports (< 1024)
  • add net.ipv4.ip_unprivileged_port_start=0 to /etc/sysctl.conf (or /etc/sysctl.d) and run sudo sysctl --system
  • dockerd-rootless.sh uses slirp4netns (if installed) or VPNKit as the network stack by default.
  • These network stacks run in userspace and might have performance overhead
  • This error occurs when the number of available entries in /etc/subuid or /etc/subgid is not sufficient.
  • This error occurs mostly when the host is running in cgroup v2. See the section Fedora 31 or later for information on switching the host to use cgroup v1.
  • --net=host doesn’t listen ports on the host network namespace This is an expected behavior, as the daemon is namespaced inside RootlessKit’s network namespace. Use docker run -p instead.
張 旭

Running rootless Podman as a non-root user | Enable Sysadmin - 0 views

www.redhat.com/...rootless-podman-makes-sense

docker podman linux system

shared by 張 旭 on 04 Aug 20 - No Cached
  • By default, rootless Podman runs as root within the container.
  • the processes in the container have the default list of namespaced capabilities which allow the processes to act like root inside of the user namespace
  • the directory is owned by UID 26, but UID 26 is not mapped into the container and is not the same UID that Postgres runs with while in the container.
  • ...8 more annotations...
  • Podman launches a container inside of the user namespace, which is mapped with the range of UIDs defined for the user in /etc/subuid and /etc/subgid
  • The easy solution to this problem is to chown the html directory to match the UID that Postgresql runs with inside of the container.
  • use the podman unshare command, which drops you into the same user namespace that rootless Podman uses
  • This setup also means that the processes inside of the container are running as the user’s UID. If the container process escaped the container, the process would have full access to files in your home directory based on UID separation.
  • SELinux would still block the access, but I have heard that some people disable SELinux.
  • If you run the processes within the container as a different non-root UID, however, then those processes will run as that UID. If they escape the container, they would only have world access to content in your home directory.
  • run a podman unshare command, or set up the directories' group ownership as owned by your UID (root inside of the container).
  • running containers as non-root should always be your top priority for security reasons.
張 旭

Understanding GitHub Actions - GitHub Docs - 0 views

docs.github.com/...understanding-github-actions

github cicd devops

shared by 張 旭 on 11 Sep 21 - No Cached
  • A job is a set of steps that execute on the same runner. By default, a workflow with multiple jobs will run those jobs in parallel.
  • Workflows are made up of one or more jobs and can be scheduled or triggered by an event
  • An event is a specific activity that triggers a workflow.
  • ...8 more annotations...
  • configure a workflow to run jobs sequentially.
  • A step is an individual task that can run commands in a job. A step can be either an action or a shell command.
  • Each step in a job executes on the same runner, allowing the actions in that job to share data with each other.
  • Actions are standalone commands that are combined into steps to create a job.
  • Actions are the smallest portable building block of a workflow.
  • To use an action in a workflow, you must include it as a step.
  • You can use a runner hosted by GitHub, or you can host your own.
  • GitHub-hosted runners are based on Ubuntu Linux, Microsoft Windows, and macOS, and each job in a workflow runs in a fresh virtual environment.
  • 張 旭 on 11 Sep 21
     
    "A job is a set of steps that execute on the same runner. By default, a workflow with multiple jobs will run those jobs in parallel. "
張 旭

Installing Addons | Kubernetes - 0 views

kubernetes.io/...addons

kubernetes network networking

shared by 張 旭 on 25 Jan 23 - No Cached
  • Calico is a networking and network policy provider. Calico supports a flexible set of networking options so you can choose the most efficient option for your situation, including non-overlay and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts, pods, and (if using Istio & Envoy) applications at the service mesh layer.
  • Cilium is a networking, observability, and security solution with an eBPF-based data plane. Cilium provides a simple flat Layer 3 network with the ability to span multiple clusters in either a native routing or overlay/encapsulation mode, and can enforce network policies on L3-L7 using an identity-based security model that is decoupled from network addressing. Cilium can act as a replacement for kube-proxy; it also offers additional, opt-in observability and security features.
  • CoreDNS is a flexible, extensible DNS server which can be installed as the in-cluster DNS for pods.
  • ...1 more annotation...
  • The node problem detector runs on Linux nodes and reports system issues as either Events or Node conditions.
張 旭

Installing kubeadm | Kubernetes - 0 views

kubernetes.io/...install-kubeadm

kubernetes

shared by 張 旭 on 02 Jun 21 - No Cached
  • Swap disabled. You MUST disable swap in order for the kubelet to work properly.
  • The product_uuid can be checked by using the command sudo cat /sys/class/dmi/id/product_uuid
  • some virtual machines may have identical values.
  • ...6 more annotations...
  • Kubernetes uses these values to uniquely identify the nodes in the cluster.
  • Make sure that the br_netfilter module is loaded.
  • you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config,
  • kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you.
  • one minor version skew between the kubelet and the control plane is supported, but the kubelet version may never exceed the API server version.
  • Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.
« First ‹ Previous 81 - 100 of 102 Next ›
Showing 20▼ items per page