Skip to main content

Home/ Larvata/ Group items tagged memory

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
張 旭

Memory inside Linux containers | Fabio Kung - 0 views

fabiokung.com/...memory-inside-linux-containers

linux docker system process

shared by 張 旭 on 05 Dec 19 - No Cached
  • /sys/fs/cgroup/ is the recommended location for cgroup hierarchies, but it is not a standard.
  • most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others.
  • cat /sys/fs/cgroup/memory/memory.stat
  • ...3 more annotations...
  • /sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location.
  • an userspace library that processes can use to query their memory usage and available memory.
  • we might need to encourage people to stop using those tools inside containers.
crazylion lee

The memory models that underlie programming languages - 0 views

canonical.org/...memory-models

ai memory

shared by crazylion lee on 30 Jan 17 - No Cached
  • crazylion lee on 30 Jan 17
     
    " There are about six major conceptualizations of memory, which I'm calling "memory models"², that dominate today's programming. Three of them derive from the three most historically important programming languages of the 1950s - COBOL, LISP, and FORTRAN - and the other three derive from the three historically important data storage systems: magnetic tape, Unix-style hierarchical filesystems, and relational databases."
張 旭

MongoDB Performance Tuning: Everything You Need to Know - Stackify - 0 views

stackify.com/mongodb-performance-tuning

database mongodb

shared by 張 旭 on 15 Apr 21 - No Cached
  • db.serverStatus().globalLock
  • db.serverStatus().locks
  • globalLock.currentQueue.total: This number can indicate a possible concurrency issue if it’s consistently high. This can happen if a lot of requests are waiting for a lock to be released.
  • ...35 more annotations...
  • globalLock.totalTime: If this is higher than the total database uptime, the database has been in a lock state for too long.
  • Unlike relational databases such as MySQL or PostgreSQL, MongoDB uses JSON-like documents for storing data.
  • Databases operate in an environment that consists of numerous reads, writes, and updates.
  • When a lock occurs, no other operation can read or modify the data until the operation that initiated the lock is finished.
  • locks.deadlockCount: Number of times the lock acquisitions have encountered deadlocks
  • Is the database frequently locking from queries? This might indicate issues with the schema design, query structure, or system architecture.
  • For version 3.2 on, WiredTiger is the default.
  • MMAPv1 locks whole collections, not individual documents.
  • WiredTiger performs locking at the document level.
  • When the MMAPv1 storage engine is in use, MongoDB will use memory-mapped files to store data.
  • All available memory will be allocated for this usage if the data set is large enough.
  • db.serverStatus().mem
  • mem.resident: Roughly equivalent to the amount of RAM in megabytes that the database process uses
  • If mem.resident exceeds the value of system memory and there’s a large amount of unmapped data on disk, we’ve most likely exceeded system capacity.
  • If the value of mem.mapped is greater than the amount of system memory, some operations will experience page faults.
  • The WiredTiger storage engine is a significant improvement over MMAPv1 in performance and concurrency.
  • By default, MongoDB will reserve 50 percent of the available memory for the WiredTiger data cache.
  • wiredTiger.cache.bytes currently in the cache – This is the size of the data currently in the cache.
  • wiredTiger.cache.tracked dirty bytes in the cache – This is the size of the dirty data in the cache.
  • we can look at the wiredTiger.cache.bytes read into cache value for read-heavy applications. If this value is consistently high, increasing the cache size may improve overall read performance.
  • check whether the application is read-heavy. If it is, increase the size of the replica set and distribute the read operations to secondary members of the set.
  • write-heavy, use sharding within a sharded cluster to distribute the load.
  • Replication is the propagation of data from one node to another
  • Replication sets handle this replication.
  • Sometimes, data isn’t replicated as quickly as we’d like.
  • a particularly thorny problem if the lag between a primary and secondary node is high and the secondary becomes the primary
  • use the db.printSlaveReplicationInfo() or the rs.printSlaveReplicationInfo() command to see the status of a replica set from the perspective of the secondary member of the set.
  • shows how far behind the secondary members are from the primary. This number should be as low as possible.
  • monitor this metric closely.
  • watch for any spikes in replication delay.
  • Always investigate these issues to understand the reasons for the lag.
  • One replica set is primary. All others are secondary.
  • it’s not normal for nodes to change back and forth between primary and secondary.
  • use the profiler to gain a deeper understanding of the database’s behavior.
  • Enabling the profiler can affect system performance, due to the additional activity.
  • 張 旭 on 15 Apr 21
     
    "globalLock.currentQueue.total: This number can indicate a possible concurrency issue if it's consistently high. This can happen if a lot of requests are waiting for a lock to be released."
張 旭

鳥哥的 Linux 私房菜 -- 第零章、計算機概論 - 0 views

linux.vbird.org/...0105computers.php

system

shared by 張 旭 on 22 Jun 21 - No Cached
  • 但因為 CPU 的運算速度比其他的設備都要來的快，又為了要滿足 FSB 的頻率，因此廠商就在 CPU 內部再進行加速， 於是就有所謂的外頻與倍頻了。
  • 中央處理器 (Central Processing Unit, CPU)，CPU 為一個具有特定功能的晶片， 裡頭含有微指令集，如果你想要讓主機進行什麼特異的功能，就得要參考這顆 CPU 是否有相關內建的微指令集才可以。
  • CPU 內又可分為兩個主要的單元，分別是： 算數邏輯單元與控制單元。
  • ...63 more annotations...
  • CPU 讀取的資料都是從主記憶體來的！ 主記憶體內的資料則是從輸入單元所傳輸進來！而 CPU 處理完畢的資料也必須要先寫回主記憶體中，最後資料才從主記憶體傳輸到輸出單元。
  • 重點在於 CPU 與主記憶體。 特別要看的是實線部分的傳輸方向，基本上資料都是流經過主記憶體再轉出去的！
  • CPU 實際要處理的資料則完全來自於主記憶體 (不管是程式還是一般文件資料)！這是個很重要的概念喔！ 這也是為什麼當你的記憶體不足時，系統的效能就很糟糕！
  • 常見到的兩種主要 CPU 架構， 分別是：精簡指令集 (RISC) 與複雜指令集 (CISC) 系統。
  • 微指令集較為精簡，每個指令的執行時間都很短，完成的動作也很單純，指令的執行效能較佳； 但是若要做複雜的事情，就要由多個指令來完成。
  • CISC在微指令集的每個小指令可以執行一些較低階的硬體操作，指令數目多而且複雜， 每條指令的長度並不相同。因為指令執行較為複雜所以每條指令花費的時間較長， 但每條個別指令可以處理的工作較為豐富。
  • 多媒體微指令集：MMX, SSE, SSE2, SSE3, SSE4, AMD-3DNow! 虛擬化微指令集：Intel-VT, AMD-SVM 省電功能：Intel-SpeedStep, AMD-PowerNow! 64/32位元相容技術：AMD-AMD64, Intel-EM64T
  • 若光以效能來說，目前的個人電腦效能已經夠快了，甚至已經比工作站等級以上的電腦運算速度還要快！ 但是工作站電腦強調的是穩定不當機，並且運算過程要完全正確，因此工作站以上等級的電腦在設計時的考量與個人電腦並不相同啦
  • 1 Byte = 8 bits
  • 檔案容量使用的是二進位的方式，所以 1 GBytes 的檔案大小實際上為：1024x1024x1024 Bytes 這麼大！ 速度單位則常使用十進位，例如 1GHz 就是 1000x1000x1000 Hz 的意思。
  • CPU的運算速度常使用 MHz 或者是 GHz 之類的單位，這個 Hz 其實就是秒分之一
  • 在網路傳輸方面，由於網路使用的是 bit 為單位，因此網路常使用的單位為 Mbps 是 Mbits per second，亦即是每秒多少 Mbit
  • (1)北橋：負責連結速度較快的CPU、主記憶體與顯示卡界面等元件
  • (2)南橋：負責連接速度較慢的裝置介面， 包括硬碟、USB、網路卡等等
  • CPU內部含有微指令集，不同的微指令集會導致CPU工作效率的優劣
  • 時脈就是CPU每秒鐘可以進行的工作次數。 所以時脈越高表示這顆CPU單位時間內可以作更多的事情。
  • 早期的 CPU 架構主要透過北橋來連結系統最重要的 CPU、主記憶體與顯示卡裝置。因為所有的設備都得掉透過北橋來連結，因此每個設備的工作頻率應該要相同。
  • 前端匯流排 (FSB)
  • 外頻指的是CPU與外部元件進行資料傳輸時的速度
  • 倍頻則是 CPU 內部用來加速工作效能的一個倍數
  • 新的 CPU 設計中， 已經將記憶體控制器整合到 CPU 內部，而連結 CPU 與記憶體、顯示卡的控制器的設計，在Intel部份使用 QPI (Quick Path Interconnect) 與 DMI 技術，而 AMD 部份則使用 Hyper Transport 了，這些技術都可以讓 CPU 直接與主記憶體、顯示卡等設備分別進行溝通，而不需要透過外部的連結晶片了。
  • 如何知道主記憶體能提供的資料量呢？此時還是得要藉由 CPU 內的記憶體控制晶片與主記憶體間的傳輸速度『前端匯流排速度(Front Side Bus, FSB)
  • 主記憶體也是有其工作的時脈，這個時脈限制還是來自於 CPU 內的記憶體控制器所決定的。
  • CPU每次能夠處理的資料量稱為字組大小(word size)， 字組大小依據CPU的設計而有32位元與64位元。我們現在所稱的電腦是32或64位元主要是依據這個 CPU解析的字組大小而來的
  • 早期的32位元CPU中，因為CPU每次能夠解析的資料量有限， 因此由主記憶體傳來的資料量就有所限制了。這也導致32位元的CPU最多只能支援最大到4GBytes的記憶體。
  • 在每一個 CPU 內部將重要的暫存器 (register) 分成兩群， 而讓程序分別使用這兩群暫存器。
  • 可以有兩個程序『同時競爭 CPU 的運算單元』，而非透過作業系統的多工切換！
  • 大多發現 HT 雖然可以提昇效能，不過，有些情況下卻可能導致效能降低喔！因為，實際上明明就僅有一個運算單元
  • 個人電腦的主記憶體主要元件為動態隨機存取記憶體(Dynamic Random Access Memory, DRAM)， 隨機存取記憶體只有在通電時才能記錄與使用，斷電後資料就消失了。因此我們也稱這種RAM為揮發性記憶體。
  • 要啟用雙通道的功能你必須要安插兩支(或四支)主記憶體，這兩支記憶體最好連型號都一模一樣比較好， 這是因為啟動雙通道記憶體功能時，資料是同步寫入/讀出這一對主記憶體中，如此才能夠提升整體的頻寬啊！
  • 第二層快取(L2 cache)整合到CPU內部，因此這個L2記憶體的速度必須要CPU時脈相同。 使用DRAM是無法達到這個時脈速度的，此時就需要靜態隨機存取記憶體(Static Random Access Memory, SRAM)的幫忙了。
  • BIOS(Basic Input Output System)是一套程式，這套程式是寫死到主機板上面的一個記憶體晶片中， 這個記憶體晶片在沒有通電時也能夠將資料記錄下來，那就是唯讀記憶體(Read Only Memory, ROM)。
  • BIOS對於個人電腦來說是非常重要的， 因為他是系統在開機的時候首先會去讀取的一個小程式
  • 由於磁碟盤是圓的，並且透過機器手臂去讀寫資料，磁碟盤要轉動才能夠讓機器手臂讀寫。因此，通常資料寫入當然就是以圓圈轉圈的方式讀寫囉！ 所以，當初設計就是在類似磁碟盤同心圓上面切出一個一個的小區塊，這些小區塊整合成一個圓形，讓機器手臂上的讀寫頭去存取。 這個小區塊就是磁碟的最小物理儲存單位，稱之為磁區 (sector)，那同一個同心圓的磁區組合成的圓就是所謂的磁軌(track)。 由於磁碟裡面可能會有多個磁碟盤，因此在所有磁碟盤上面的同一個磁軌可以組合成所謂的磁柱 (cylinder)。
  • 原本硬碟的磁區都是設計成 512byte 的容量，但因為近期以來硬碟的容量越來越大，為了減少資料量的拆解，所以新的高容量硬碟已經有 4Kbyte 的磁區設計
  • 拿快閃記憶體去製作成高容量的設備，這些設備的連接界面也是透過 SATA 或 SAS，而且外型還做的跟傳統磁碟一樣
  • 固態硬碟最大的好處是，它沒有馬達不需要轉動，而是透過記憶體直接讀寫的特性，因此除了沒資料延遲且快速之外，還很省電
  • 硬碟主要是利用主軸馬達轉動磁碟盤來存取，因此轉速的快慢會影響到效能
  • 使用作業系統的正常關機方式，才能夠有比較好的硬碟保養啊！因為他會讓硬碟的機械手臂歸回原位啊！
  • I/O位址有點類似每個裝置的門牌號碼，每個裝置都有他自己的位址，一般來說，不能有兩個裝置使用同一個I/O位址， 否則系統就會不曉得該如何運作這兩個裝置了。
  • IRQ就可以想成是各個門牌連接到郵件中心(CPU)的專門路徑囉！ 各裝置可以透過IRQ中斷通道來告知CPU該裝置的工作情況，以方便CPU進行工作分配的任務。
  • BIOS為寫入到主機板上某一塊 flash 或 EEPROM 的程式，他可以在開機的時候執行，以載入CMOS當中的參數， 並嘗試呼叫儲存裝置中的開機程式，進一步進入作業系統當中。
  • 電腦都只有記錄0/1而已，甚至記錄的資料都是使用byte/bit等單位來記錄的
  • 常用的英文編碼表為ASCII系統，這個編碼系統中， 每個符號(英文、數字或符號等)都會佔用1bytes的記錄， 因此總共會有28=256種變化
  • 中文字當中的編碼系統早期最常用的就是big5這個編碼表了。 每個中文字會佔用2bytes，理論上最多可以有216=65536，亦即最多可達6萬多個中文字。
  • 國際組織ISO/IEC跳出來制訂了所謂的Unicode編碼系統， 我們常常稱呼的UTF8或萬國碼的編碼
  • CPU其實是具有微指令集的。因此，我們需要CPU幫忙工作時，就得要參考微指令集的內容， 然後撰寫讓CPU讀的懂的指令碼給CPU執行，這樣就能夠讓CPU運作了。
  • 編譯器』來將這些人類能夠寫的程式語言轉譯成為機器能看懂得機器碼
  • 當你需要將運作的資料寫入記憶體中，你就得要自行分配一個記憶體區塊出來讓自己的資料能夠填上去， 所以你還得要瞭解到記憶體的位址是如何定位的，啊！眼淚還是不知不覺的流了下來... 怎麼寫程式這麼麻煩啊！
  • 作業系統(Operating System, OS)其實也是一組程式， 這組程式的重點在於管理電腦的所有活動以及驅動系統中的所有硬體。
  • 作業系統的功能就是讓CPU可以開始判斷邏輯與運算數值、 讓主記憶體可以開始載入/讀出資料與程式碼、讓硬碟可以開始被存取、讓網路卡可以開始傳輸資料、 讓所有周邊可以開始運轉等等。
  • 只有核心有提供的功能，你的電腦系統才能幫你完成！舉例來說，你的核心並不支援TCP/IP的網路協定， 那麼無論你購買了什麼樣的網卡，這個核心都無法提供網路能力的！
  • 核心程式所放置到記憶體當中的區塊是受保護的！ 並且開機後就一直常駐在記憶體當中。
  • 作業系統通常會提供一整組的開發介面給工程師來開發軟體！ 工程師只要遵守該開發介面那就很容易開發軟體了！
  • 系統呼叫介面(System call interface)
  • 程序管理(Process control)
  • 記憶體管理(Memory management)
  • 檔案系統管理(Filesystem management)
  • 通常核心會提供虛擬記憶體的功能，當記憶體不足時可以提供記憶體置換(swap)的功能
  • 裝置的驅動(Device drivers)
  • 『可載入模組』功能，可以將驅動程式編輯成模組，就不需要重新的編譯核心
  • 驅動程式可以說是作業系統裡面相當重要的一環
  • 作業系統通常會提供一個開發介面給硬體開發商， 讓他們可以根據這個介面設計可以驅動他們硬體的『驅動程式』，如此一來，只要使用者安裝驅動程式後， 自然就可以在他們的作業系統上面驅動這塊顯示卡了。
  • 張 旭 on 22 Jun 21
     
    "但因為 CPU 的運算速度比其他的設備都要來的快，又為了要滿足 FSB 的頻率，因此廠商就在 CPU 內部再進行加速， 於是就有所謂的外頻與倍頻了。"
張 旭

Introducing CNAME Flattening: RFC-Compliant CNAMEs at a Domain's Root - 0 views

blog.cloudflare.com/liant-cnames-at-a-domains-root

dns networking

shared by 張 旭 on 15 Mar 22 - No Cached
  • you can now safely use a CNAME record, as opposed to an A record that points to a fixed IP address, as your root record in CloudFlare DNS without triggering a number of edge case error conditions because you’re violating the DNS spec.
  • CNAME Flattening allowed us to use a root domain while still maintaining DNS fault-tolerance across multiple IP addresses.
  • Traditionally, the root record of a domain needed to point to an IP address (known as an A -- for "address" -- Record).
  • ...13 more annotations...
  • WordPlumblr allows its users to use custom domains that point to the WordPlumblr infrastructure
  • A CNAME is an alias. It allows one domain to point to another domain which, eventually if you follow the CNAME chain, will resolve to an A record and IP address.
  • For example, WordPlumblr might have assigned the CNAME 6equj5.wordplumblr.com for Foo.com. Foo.com and the other customers may have all initially resolved, at the end of the CNAME chain, to the same IP address.
  • you usually don't want to address memory directly but, instead, you set up a pointer to a block of memory where you're going to store something. If the operating system needs to move the memory around then it just updates the pointer to point to wherever the chunk of memory has been moved to.
  • CNAMEs work great for subdomains like www.foo.com or blog.foo.com. Unfortunately, they don't work for a naked domain like foo.com itself.
  • the DNS spec enshrined that the root record -- the naked domain without any subdomain -- could not be a CNAME.
  • Technically, the root could be a CNAME but the RFCs state that once a record has a CNAME it can't have any other entries associated with it
  • a way to support a CNAME at the root, but still follow the RFC and return an IP address for any query for the root record.
  • extended our authoritative DNS infrastructure to, in certain cases, act as a kind of DNS resolver.
  • if there's a CNAME at the root, rather than returning that record directly we recurse through the CNAME chain ourselves until we find an A Record.
  • allows the flexibility of having CNAMEs at the root without breaking the DNS specification.
  • We cache the CNAME responses -- respecting the DNS TTLs, just like a recursor should -- which means often we have the answer without having to traverse the chain.
  • CNAME flattening solved email resolution errors for us which was very key.
crazylion lee

Apache Geode (incubating) | Home - 0 views

geode.incubator.apache.org

apache in-memory db

shared by crazylion lee on 30 Nov 15 - No Cached
  • crazylion lee on 30 Nov 15
     
    "Geode is an open source, distributed, in-memory database for scale-out applications."
張 旭

The Twelve-Factor App - 0 views

12factor.net/processes

development application programming

shared by 張 旭 on 20 Feb 19 - No Cached
  • stateless processes
  • a production deploy of a sophisticated app may use many process types, instantiated into zero or more running processes.
  • Twelve-factor processes are stateless and share-nothing.
  • ...6 more annotations...
  • Any data that needs to persist must be stored in a stateful backing service, typically a database.
  • The memory space or filesystem of the process can be used as a brief, single-transaction cache.
  • wipe out all local (e.g., memory and filesystem) state
  • compiling during the build stage
  • “sticky sessions” – that is, caching user session data in memory of the app’s process and expecting future requests from the same visitor to be routed to the same process.
  • Sticky sessions are a violation of twelve-factor and should never be used or relied upon
crazylion lee

Riemann - A network monitoring system - 0 views

riemann.io/index.html

monitoring dashboard sysadmin

shared by crazylion lee on 17 Apr 16 - No Cached
  • crazylion lee on 17 Apr 16
     
    "Riemann aggregates events from your servers and applications with a powerful stream processing language. Send an email for every exception in your app. Track the latency distribution of your web app. See the top processes on any host, by memory and CPU. Combine statistics from every Riak node in your cluster and forward to Graphite. Track user activity from second to second."
crazylion lee

Writing a Bootloader | Blog of Osanda - 1 views

osandamalith.wordpress.com/...writing-a-bootloader

bootloader linux c

shared by crazylion lee on 06 Nov 15 - No Cached
  • crazylion lee on 06 Nov 15
     
    "A bootloader is a special program that is executed each time a bootable device is initialized by the computer during its power on or reset that will load the kernel image into the memory. This application is very close to hardware and to the architecture of the CPU. All x86 PCs boot in Real Mode. In this mode you have only 16-bit instructions. Our bootloader runs in Real Mode and our bootloader is a 16-bit program."
張 旭

2. Swoole Structure · swooletw/laravel-swoole Wiki - 0 views

github.com/...2.-Swoole-Structure

php swoole performance

shared by 張 旭 on 24 Apr 19 - No Cached
  • Laravel application will exist in Worker processes.
  • means Laravel can be stored and kept in memory.
  • Laravel application will exist in the memory and only initialize at the first time. Any changes you did to Laravel will be kept unless you reset them by yourself.
  • 張 旭 on 24 Apr 19
     
    "Laravel application will exist in Worker processes. "
張 旭

第 05 章 - 計算機概論 - 主機系統與 I/O 界面 - 0 views

dic.vbird.tw/...2020unit05.php

shared by 張 旭 on 21 Jun 21 - No Cached
  • 接受使用者輸入指令與資料，經由中央處理器的數學與邏輯單元運算處理後，以產生或儲存成有用的資訊
  • CPU 為一個具有特定功能的晶片， 裡頭含有微指令集，如果你想要讓主機進行什麼特異的功能，就得要參考這顆 CPU 是否有相關內建的微指令集才可以。
  • CPU 讀取的資料都是從主記憶體來的！ 主記憶體內的資料則是從輸入單元所傳輸進來！而 CPU 處理完畢的資料也必須要先寫回主記憶體中，最後資料才從主記憶體傳輸到輸出單元。
  • ...49 more annotations...
  • 算數邏輯單元與控制單元。其中算數邏輯單元主要負責程式運算與邏輯判斷，控制單元則主要在協調各周邊元件與各單元間的工作。
  • 資料會先寫入到主記憶體，然後 CPU 才能開始應用
  • CPU 其實內部已經含有一些微指令，我們所使用的軟體都要經過 CPU 內部的微指令集來達成才行。
  • 世界上常見到的兩種主要 CPU 架構， 分別是：精簡指令集 (RISC) 與複雜指令集 (CISC) 系統。
  • 微指令集較為精簡，每個指令的執行時間都很短，完成的動作也很單純，指令的執行效能較佳； 但是若要做複雜的事情，就要由多個指令來完成。
  • CISC在微指令集的每個小指令可以執行一些較低階的硬體操作，指令數目多而且複雜， 每條指令的長度並不相同。因為指令執行較為複雜所以每條指令花費的時間較長， 但每條個別指令可以處理的工作較為豐富。
  • 最早的那顆Intel發展出來的CPU代號稱為8086
  • AMD依此架構修改新一代的CPU為64位元
  • CPU 位元指的是CPU一次資料讀取的最大量！64位元CPU代表CPU一次可以讀寫64bits這麼多的資料
  • 因為CPU讀取資料量有限制，因此能夠從記憶體中讀寫的資料也就有所限制
  • ：(1)北橋：負責連結速度較快的CPU、主記憶體與顯示卡界面等元件；
  • (2)南橋：負責連接速度較慢的裝置介面， 包括硬碟、USB、網路卡等等。不過由於北橋最重要的就是 CPU 與主記憶體之間的橋接，因此目前的主流架構中， 大多將北橋記憶體控制器整合到 CPU 封裝當中
  • CPU 與可接受的晶片組是有搭配性的，尤其目前每種 CPU 的腳位都不一樣
  • 時脈越高，代表單位時間內可進行的工作越多
  • 大概都只看總頻寬或基準時脈
  • CPU 時脈越高的情況下，會產生很多諸如散熱、設計及與週邊元件溝通的問題。
  • 其他的程序還在等待 CPU ，但是 CPU 又在等待 I/O
  • Intel 在同一個運算核心底下安裝兩個暫存器來模擬出另一個核心，實際上是兩個暫存器 (可以想成是程式的執行器) 共用一個實體核心。 這就是超執行緒的簡易認知
  • 時脈的意思是每秒鐘能夠進行的工作次數，而每次工作能夠讀進的資料量就是位元數
  • 每個用戶端大多是短時間的大運算，所以通常不會有一隻程序一直佔用著系統資源。
  • CPU 的所有資料都是從記憶體讀寫來的，所以記憶體的容量與頻寬就相當的重要了
  • DRAM 根據技術的更新又分好幾代，而使用上較廣泛的有所謂的 SDRAM 與 DDR SDRAM 兩種。 這兩種記憶體的差別除了在於腳位與工作電壓上的不同之外，DDR 是所謂的雙倍資料傳送速度 (Double Data Rate)
  • 晶片組廠商就將兩個主記憶體彙整在一起，如果一支記憶體可達 64 位元，兩支記憶體就可以達到 128 位元了，這就是雙通道的設計理念。 在某些比較多核心的伺服器主機上面，甚至還使用了 3 通道到 4 通道的設計
  • CPU 內部的暫存器與少量記憶體被稱為第一、第二層快取 (L1, L2 cache)，而放在核心間的快取記憶體就被稱為第三層快取記憶體 (L3 cache)
  • CPU 的暫存器是直接設計在 CPU 核心內部，可以用來幫助 CPU 的運算。
  • 在多核心 CPU 的核心之間，會有另一個共享的快取記憶體存在，讓所有的 CPU 核心可以共享某些資源。
  • 靜態隨機存取記憶體 (Static Random Access Memory, SRAM)
  • 各項參數， 是被記錄到主機板上頭的一個稱為 CMOS 的晶片上，這個晶片需要藉著額外的電源來發揮記錄功能， 這也是為什麼你的主機板上面會有一顆電池的緣故。
  • BIOS(Basic Input Output System)是一套程式，這套程式是寫死到主機板上面的一個記憶體晶片中， 這個記憶體晶片在沒有通電時也能夠將資料記錄下來，那就是唯讀記憶體(Read Only Memory, ROM)。
  • BIOS對於個人電腦來說是非常重要的， 因為他是系統在開機的時候首先會去讀取的一個小程式喔
  • 韌體(firmware)很多也是使用ROM來進行軟體的寫入的。 韌體像軟體一樣也是一個被電腦所執行的程式，然而他是對於硬體內部而言更加重要的部分。例如BIOS就是一個韌體， BIOS雖然對於我們日常操作電腦系統沒有什麼太大的關係，但是他卻控制著開機時各項硬體參數的取得！
  • 現在的 BIOS 通常是寫入類似快閃記憶體 (flash) 或 EEPROM
  • 顯示卡能夠傳輸的資料量當然也是越大越好！這時就得要考慮到傳輸的界面了！當前 (2018) 主流的傳輸界面為 PCI-E，這個 PCI-E 又有三種版本， version 1, 2, 3 ，這三個版本的速度並不相同
  • PCI-E (PCI-Express) 使用的是類似管線的概念來處理，在 PCI-E 第一版中，每條管線可以具有 250MBytes/s 的頻寬效能，管線越多(通常設計到 x16 管線)則總頻寬越高
  • 通常 CPU 製造商會根據 CPU 來設計搭配的南橋晶片，由於現在只有一個晶片 (北橋整合到 CPU 內了)，所以目前只有一顆主機板晶片組。
  • 插槽上面的資料要傳輸到 CPU 就得要經過晶片組，然後透過 DMI 通道傳上去！所以，在某些情況下，系統的效能會被卡住在這條通道上喔！
  • 晶片組接的設備相當多喔！有 PCI-E 插槽、USB設備、網路設備、音效設備、硬碟設備等，這全部的裝置共用那一條 DMI 喔！ 所以，整個系統效能的瓶頸通常不在 CPU 啦！通常就是在南橋接的設備上面！所以，當你有非常複雜的程式要運作的時候，讓這些程式越少通過南橋， 他的系統效能就會比較好一點
  • 物理組成
  • 讀寫主要是透過在機械手臂上的讀取頭(head)來達成
  • 由於磁碟盤是圓的，並且透過機器手臂去讀寫資料，磁碟盤要轉動才能夠讓機器手臂讀寫。
  • 磁碟的最小物理儲存單位，稱之為磁區 (sector)，那同一個同心圓的磁區組合成的圓就是所謂的磁軌(track)。 由於磁碟裡面可能會有多個磁碟盤，因此在所有磁碟盤上面的同一個磁軌可以組合成所謂的磁柱 (cylinder)。
  • 通常資料的讀寫會由外圈開始往內寫
  • 目前主流的硬碟連接界面就是 SATA
  • 雖然 SATA III 界面理論傳輸可到達 600Mbytes/s，不過傳統硬碟由於物理設計的限制因素，通常存取速度效能大多在 150~200Mbytes/s 之間
  • 串列式 SCSI (Serial Attached SCSI, SAS)
  • 傳統硬碟有個很致命的問題，就是需要驅動馬達去轉動磁碟盤～這會造成很嚴重的磁碟讀取延遲
  • 快閃記憶體去製作成高容量的設備
  • 沒有讀寫頭與磁碟盤啊！都是記憶體！
  • 各個繪圖卡的運算晶片 (GPU) 設計的理念不同，加上驅動程式與軟體搭配的問題，並不是一張高效能繪圖卡就可以完勝其他的繪圖卡， 還得要注意相關的軟體才行。
張 旭

Logstash Alternatives: Pros & Cons of 5 Log Shippers [2019] - Sematext - 0 views

sematext.com/...logstash-alternatives

log monitor system logstash fluentd

shared by 張 旭 on 05 Nov 19 - No Cached
  • In this case, Elasticsearch. And because Elasticsearch can be down or struggling, or the network can be down, the shipper would ideally be able to buffer and retry
  • Logstash is typically used for collecting, parsing, and storing logs for future use as part of log management.
  • Logstash’s biggest con or “Achille’s heel” has always been performance and resource consumption (the default heap size is 1GB).
  • ...37 more annotations...
  • This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.
  • Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.
  • differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.
  • Filebeat is just a tiny binary with no dependencies.
  • For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.
  • For example, the apache module will point Filebeat to default access.log and error.log paths
  • Filebeat’s scope is very limited,
  • Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.
  • Filebeat can parse JSON
  • you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.
  • You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off
  • For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well
  • The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.
  • It can tail files, parse them, buffer (on disk and in memory) and ship to a number of destinations, including Elasticsearch.
  • rsyslog is the fastest shipper
  • Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).
  • use it as a simple router/shipper, any decent machine will be limited by network bandwidth
  • It’s also one of the lightest parsers you can find, depending on the configured memory buffers.
  • rsyslog requires more work to get the configuration right
  • the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.
  • rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).
  • rsyslog also works well when you need that ultimate performance.
  • syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).
  • a modular syslog daemon, that can do much more than just syslog
  • Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.
  • Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.
  • syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance
  • Fluentd was built on the idea of logging in JSON wherever possible (which is a practice we totally agree with) so that log shippers down the line don’t have to guess which substring is which field of which type.
  • Fluentd plugins are in Ruby and very easy to write.
  • structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).
  • Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.
  • Fluentd is a good fit when you have diverse or exotic sources and destinations for your logs, because of the number of plugins.
  • Splunk isn’t a log shipper, it’s a commercial logging solution
  • Graylog is another complete logging solution, an open-source alternative to Splunk.
  • everything goes through graylog-server, from authentication to queries.
  • Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.
  • it depends
張 旭

http - nginx upload client_max_body_size issue - Stack Overflow - 0 views

stackoverflow.com/...oad-client-max-body-size-issue

nginx upload file http server

shared by 張 旭 on 26 Sep 16 - No Cached
  • nginx "fails fast" when the client informs it that it's going to send a body larger than the client_max_body_size by sending a 413 response and closing the connection.
  • Because nginx closes the connection, the client sends data to the closed socket, causing a TCP RST.
  • Most clients don't read responses until the entire request body is sent.
  • ...2 more annotations...
  • Client body and buffers are key because nginx must buffer incoming data.
  • The clean setting frees up memory and consumption limits by instructing nginx to store incoming buffer in a file and then clean this file later from disk by deleting it.
張 旭

Rails Routing from the Outside In - Ruby on Rails Guides - 0 views

guides.rubyonrails.org/routing.html

rails router path url namespace

shared by 張 旭 on 09 Mar 14 - Cached
  • Resource routing allows you to quickly declare all of the common routes for a given resourceful controller.
  • Rails would dispatch that request to the destroy method on the photos controller with { id: '17' } in params.
  • a resourceful route provides a mapping between HTTP verbs and URLs to controller actions.
  • ...86 more annotations...
  • each action also maps to particular CRUD operations in a database
  • resource :photo and resources :photos creates both singular and plural routes that map to the same controller (PhotosController).
  • One way to avoid deep nesting (as recommended above) is to generate the collection actions scoped under the parent, so as to get a sense of the hierarchy, but to not nest the member actions.
  • to only build routes with the minimal amount of information to uniquely identify the resource
  • The shallow method of the DSL creates a scope inside of which every nesting is shallow
  • These concerns can be used in resources to avoid code duplication and share behavior across routes
  • add a member route, just add a member block into the resource block
  • You can leave out the :on option, this will create the same member route except that the resource id value will be available in params[:photo_id] instead of params[:id].
  • Singular Resources
  • use a singular resource to map /profile (rather than /profile/:id) to the show action
  • Passing a String to get will expect a controller#action format
  • workaround
  • organize groups of controllers under a namespace
  • route /articles (without the prefix /admin) to Admin::ArticlesController
  • route /admin/articles to ArticlesController (without the Admin:: module prefix)
  • Nested routes allow you to capture this relationship in your routing.
  • helpers take an instance of Magazine as the first parameter (magazine_ads_url(@magazine)).
  • Resources should never be nested more than 1 level deep.
  • via the :shallow option
  • a balance between descriptive routes and deep nesting
  • :shallow_path prefixes member paths with the specified parameter
  • Routing Concerns allows you to declare common routes that can be reused inside other resources and routes
  • Rails can also create paths and URLs from an array of parameters.
  • use url_for with a set of objects
  • In helpers like link_to, you can specify just the object in place of the full url_for call
  • insert the action name as the first element of the array
  • This will recognize /photos/1/preview with GET, and route to the preview action of PhotosController, with the resource id value passed in params[:id]. It will also create the preview_photo_url and preview_photo_path helpers.
  • pass :on to a route, eliminating the block:
  • Collection Routes
  • This will enable Rails to recognize paths such as /photos/search with GET, and route to the search action of PhotosController. It will also create the search_photos_url and search_photos_path route helpers.
  • simple routing makes it very easy to map legacy URLs to new Rails actions
  • add an alternate new action using the :on shortcut
  • When you set up a regular route, you supply a series of symbols that Rails maps to parts of an incoming HTTP request.
  • :controller maps to the name of a controller in your application
  • :action maps to the name of an action within that controller
  • optional parameters, denoted by parentheses
  • This route will also route the incoming request of /photos to PhotosController#index, since :action and :id are
  • use a constraint on :controller that matches the namespace you require
  • dynamic segments don't accept dots
  • The params will also include any parameters from the query string
  • :defaults option.
  • set params[:format] to "jpg"
  • cannot override defaults via query parameters
  • specify a name for any route using the :as option
  • create logout_path and logout_url as named helpers in your application.
  • Inside the show action of UsersController, params[:username] will contain the username for the user.
  • should use the get, post, put, patch and delete methods to constrain a route to a particular verb.
  • use the match method with the :via option to match multiple verbs at once
  • Routing both GET and POST requests to a single action has security implications
  • 'GET' in Rails won't check for CSRF token. You should never write to the database from 'GET' requests
  • use the :constraints option to enforce a format for a dynamic segment
  • constraints
  • don't need to use anchors
  • Request-Based Constraints
  • the same name as the hash key and then compare the return value with the hash value.
  • constraint values should match the corresponding Request object method return type
    • 張 旭
      張 旭 on 11 Apr 16
       
      應該就是檢查來源的 request， 如果是某個特定的 request 來訪問的，就通過。
  • blacklist
    • 張 旭
      張 旭 on 11 Apr 16
       
      這裡有點複雜 ...
  • redirect helper
  • reuse dynamic segments from the match in the path to redirect
  • this redirection is a 301 "Moved Permanently" redirect.
  • root method
  • put the root route at the top of the file
  • The root route only routes GET requests to the action.
  • root inside namespaces and scopes
  • For namespaced controllers you can use the directory notation
  • Only the directory notation is supported
  • use the :constraints option to specify a required format on the implicit id
  • specify a single constraint to apply to a number of routes by using the block
  • non-resourceful routes
  • :id parameter doesn't accept dots
  • :as option lets you override the normal naming for the named route helpers
  • use the :as option to prefix the named route helpers that Rails generates for a rout
  • prevent name collisions
  • prefix routes with a named parameter
  • This will provide you with URLs such as /bob/articles/1 and will allow you to reference the username part of the path as params[:username] in controllers, helpers and views
  • :only option
  • :except option
  • generate only the routes that you actually need can cut down on memory use and speed up the routing process.
  • alter path names
  • http://localhost:3000/rails/info/routes
  • rake routes
  • setting the CONTROLLER environment variable
  • Routes should be included in your testing strategy
  • assert_generates assert_recognizes assert_routing
張 旭

Sorting for Humans : Natural Sort Order - 0 views

blog.codinghorror.com/-for-humans-natural-sort-order

python programming sort file

shared by 張 旭 on 07 Dec 18 - No Cached
  • It's a completely nonsensical ordering to anyone who doesn't have the ASCII chart committed to memory (and by the way, uppercase A is decimal 65).
  • it's hard to find information on natural sorting
  • Implementing a natural sort is more complex than it seems
  • ...3 more annotations...
  • a basic natural sort shouldn't require the 40+ lines of code it takes in most languages.
  • ASCIIbetical does not equal alphabetical
  • a more human-friendly natural sort option should be built into mainstream programming languages
張 旭

Home · sysown/proxysql Wiki - 0 views

github.com/wiki

mysql database cluster mariadb proxy

shared by 張 旭 on 19 Nov 18 - No Cached
  • bear in mind that the best way to configure ProxySQL is through its admin interface.
  • llow you to control the list of the backend servers, how traffic is routed to them, and other important settings (such as caching, access control, etc)
  • Once you've made modifications to the in-memory data structure, you must load the new configuration to the runtime, or persist the new settings to disk
  • ...4 more annotations...
  • mysql_variables: contains global variables that control the functionality for handling the incoming MySQL traffic.
  • mysql_users: contains rows for the mysql_users table from the admin interface. Basically, these define the users which can connect to the proxy, and the users with which the proxy can connect to the backend servers.
  • mysql_servers: contains rows for the mysql_servers table from the admin interface. Basically, these define the backend servers towards which the incoming MySQL traffic is routed.
  • mysql_query_rules: contains rows for the mysql_query_rules table from the admin interface. Basically, these define the rules used to classify and route the incoming MySQL traffic, according to various criteria (patterns matched, user used to run the query, etc.).
張 旭

How services work | Docker Documentation - 0 views

docs.docker.com/...services

docker swarm cluster system

shared by 張 旭 on 07 Sep 18 - No Cached
  • a service is the image for a microservice within the context of some larger application.
  • When you create a service, you specify which container image to use and which commands to execute inside running containers.
  • an overlay network for the service to connect to other services in the swarm
  • ...13 more annotations...
  • In the swarm mode model, each task invokes exactly one container
  • A task is analogous to a “slot” where the scheduler places a container.
  • A task is the atomic unit of scheduling within a swarm.
  • A task is a one-directional mechanism. It progresses monotonically through a series of states: assigned, prepared, running, etc.
  • Docker swarm mode is a general purpose scheduler and orchestrator.
  • Hypothetically, you could implement other types of tasks such as virtual machine tasks or non-containerized process tasks.
  • If all nodes are paused or drained, and you create a service, it is pending until a node becomes available.
  • reserve a specific amount of memory for a service.
  • impose placement constraints on the service
  • As the administrator of a swarm, you declare the desired state of your swarm, and the manager works with the nodes in the swarm to create that state.
  • two types of service deployments, replicated and global.
  • A global service is a service that runs one task on every node.
  • Good candidates for global services are monitoring agents, an anti-virus scanners or other types of containers that you want to run on every node in the swarm.
張 旭

What are Docker : images? - Project Atomic - 0 views

www.projectatomic.io/...at-are-docker-none-none-images

docker system

shared by 張 旭 on 18 Sep 18 - No Cached
  • Now we understand what these <none>:<none> images stand for. They stand for intermediate images and can be seen using docker images -a
  • They don’t result into a disk space problem but it is definitely a screen real estate problem
  • dangling images
  • ...7 more annotations...
  • Another style of <none>:<none> images are the dangling images which can cause disk space problems.
  • In programming languages like Java or Golang a dangling block of memory is a block that is not referenced by any piece of code.
  • a dangling file system layer in Docker is something that is unused and is not being referenced by any images.
  • intermediate images
  • do docker images and see <none>:<none> images in the list, these are dangling images and needs to be pruned.
  • These dangling images are produced as a result of docker build or pull command.
  • docker rmi $(docker images -f "dangling=true" -q)
1 - 20 of 32 Next ›
Showing 20▼ items per page