Skip to main content

Home/ Larvata/ Group items tagged serialization

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
crazylion lee

The MessagePack Project - 0 views

msgpack.org

json serialization binary

shared by crazylion lee on 07 Sep 16 - Cached
  • crazylion lee on 07 Sep 16
     
    "MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves."
crazylion lee

amznlabs/ion-java: Java streaming parser/serializer for Ion. - 0 views

github.com/ion-java

java json ion

shared by crazylion lee on 24 Apr 16 - No Cached
  • crazylion lee on 24 Apr 16
     
    "Java streaming parser/serializer for Ion."
crazylion lee

Amazon Ion - 0 views

amznlabs.github.io/...index.html

amazon protocolbuf xml format data

shared by crazylion lee on 25 Apr 16 - No Cached
  • crazylion lee on 25 Apr 16
     
    "Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The text format (a superset of JSON) is easy to read and author, supporting rapid prototyping. The binary representation is efficient to store, transmit, and skip-scan parse. The rich type system provides unambiguous semantics for long-term preservation of business data which can survive multiple generations of software evolution. Ion was built to solve the rapid development, decoupling, and efficiency challenges faced every day while engineering large-scale, service-oriented architectures. Ion has been addressing these challenges within Amazon for nearly a decade, and we believe others will benefit as well. "
張 旭

Queues - Laravel - The PHP Framework For Web Artisans - 0 views

laravel.com/queues

php programming laravel queue worker framework

shared by 張 旭 on 02 Mar 19 - No Cached
  • Laravel queues provide a unified API across a variety of different queue backends, such as Beanstalk, Amazon SQS, Redis, or even a relational database.
  • The queue configuration file is stored in config/queue.php
  • a synchronous driver that will execute jobs immediately (for local use)
  • ...56 more annotations...
  • A null queue driver is also included which discards queued jobs.
  • In your config/queue.php configuration file, there is a connections configuration option.
  • any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.
  • each connection configuration example in the queue configuration file contains a queue attribute.
  • if you dispatch a job without explicitly defining which queue it should be dispatched to, the job will be placed on the queue that is defined in the queue attribute of the connection configuration
  • pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed
  • specify which queues it should process by priority.
  • If your Redis queue connection uses a Redis Cluster, your queue names must contain a key hash tag.
  • ensure all of the Redis keys for a given queue are placed into the same hash slot
  • all of the queueable jobs for your application are stored in the app/Jobs directory.
  • Job classes are very simple, normally containing only a handle method which is called when the job is processed by the queue.
  • we were able to pass an Eloquent model directly into the queued job's constructor. Because of the SerializesModels trait that the job is using, Eloquent models will be gracefully serialized and unserialized when the job is processing.
  • When the job is actually handled, the queue system will automatically re-retrieve the full model instance from the database.
  • The handle method is called when the job is processed by the queue
  • The arguments passed to the dispatch method will be given to the job's constructor
  • delay the execution of a queued job, you may use the delay method when dispatching a job.
  • dispatch a job immediately (synchronously), you may use the dispatchNow method.
  • When using this method, the job will not be queued and will be run immediately within the current process
  • specify a list of queued jobs that should be run in sequence.
  • Deleting jobs using the $this->delete() method will not prevent chained jobs from being processed. The chain will only stop executing if a job in the chain fails.
  • this does not push jobs to different queue "connections" as defined by your queue configuration file, but only to specific queues within a single connection.
  • To specify the queue, use the onQueue method when dispatching the job
  • To specify the connection, use the onConnection method when dispatching the job
  • defining the maximum number of attempts on the job class itself.
  • to defining how many times a job may be attempted before it fails, you may define a time at which the job should timeout.
  • using the funnel method, you may limit jobs of a given type to only be processed by one worker at a time
  • using the throttle method, you may throttle a given type of job to only run 10 times every 60 seconds.
  • If an exception is thrown while the job is being processed, the job will automatically be released back onto the queue so it may be attempted again.
  • dispatch a Closure. This is great for quick, simple tasks that need to be executed outside of the current request cycle
  • When dispatching Closures to the queue, the Closure's code contents is cryptographically signed so it can not be modified in transit.
  • Laravel includes a queue worker that will process new jobs as they are pushed onto the queue.
  • once the queue:work command has started, it will continue to run until it is manually stopped or you close your terminal
  • queue workers are long-lived processes and store the booted application state in memory.
  • they will not notice changes in your code base after they have been started.
  • during your deployment process, be sure to restart your queue workers.
  • customize your queue worker even further by only processing particular queues for a given connection
  • The --once option may be used to instruct the worker to only process a single job from the queue
  • The --stop-when-empty option may be used to instruct the worker to process all jobs and then exit gracefully.
  • Daemon queue workers do not "reboot" the framework before processing each job.
  • you should free any heavy resources after each job completes.
  • Since queue workers are long-lived processes, they will not pick up changes to your code without being restarted.
  • restart the workers during your deployment process.
  • php artisan queue:restart
  • The queue uses the cache to store restart signals
  • the queue workers will die when the queue:restart command is executed, you should be running a process manager such as Supervisor to automatically restart the queue workers.
  • each queue connection defines a retry_after option. This option specifies how many seconds the queue connection should wait before retrying a job that is being processed.
  • The --timeout option specifies how long the Laravel queue master process will wait before killing off a child queue worker that is processing a job.
  • When jobs are available on the queue, the worker will keep processing jobs with no delay in between them.
  • While sleeping, the worker will not process any new jobs - the jobs will be processed after the worker wakes up again
  • the numprocs directive will instruct Supervisor to run 8 queue:work processes and monitor all of them, automatically restarting them if they fail.
  • Laravel includes a convenient way to specify the maximum number of times a job should be attempted.
  • define a failed method directly on your job class, allowing you to perform job specific clean-up when a failure occurs.
  • a great opportunity to notify your team via email or Slack.
  • php artisan queue:retry all
  • php artisan queue:flush
  • When injecting an Eloquent model into a job, it is automatically serialized before being placed on the queue and restored when the job is processed
張 旭

rails/activeresource - 0 views

github.com/activeresource

rails database MVC api

shared by 張 旭 on 09 Mar 14 - No Cached
    • 張 旭
      張 旭 on 09 Mar 14
       
      所以執行 Person.find 時，會發送一個 GET 到 api.people.com:3000
    • 張 旭
      張 旭 on 09 Mar 14
       
      所以執行 Person.find 時，會發送一個 GET 到 api.people.com:3000
  • Active Resource is built on a standard JSON or XML format for requesting and submitting resources over HTTP
  • REST uses HTTP, but unlike “typical” web applications, it makes use of all the verbs available in the HTTP specification
  • ...2 more annotations...
  • When a request is made to a remote resource, a REST JSON request is generated, transmitted, and the result received and serialized into a usable Ruby object.
  • Relationships between resources can be declared using the standard association syntax that should be familiar to anyone who uses activerecord
張 旭

Backends: State Storage and Locking - Terraform by HashiCorp - 0 views

www.terraform.io/...state.html

terraform devops system storage status

shared by 張 旭 on 07 Apr 19 - No Cached
  • Backends determine where state is stored.
  • backends happen to provide locking: local via system APIs and Consul via locking APIs.
  • manually retrieve the state from the remote state using the terraform state pull command
  • ...3 more annotations...
  • manually write state with terraform state push. This is extremely dangerous and should be avoided if possible. This will overwrite the remote state.
  • The "lineage" is a unique ID assigned to a state when it is created.
  • Every state has a monotonically increasing "serial" number.
  • 張 旭 on 07 Apr 19
     
    "Backends determine where state is stored."
張 旭

Storing Sessions in a Database, by Chris Shiflett - 0 views

shiflett.org/...storing-sessions-in-a-database

php session database

shared by 張 旭 on 14 Jan 19 - Cached
  • to store sessions in a database rather than the filesystem
  • server affinity (methods that direct requests from the same client to the same server)
  • store sessions in a central database that is common to all servers
  • ...6 more annotations...
  • security concerns
  • PHP provides a function that lets you override the default session mechanism by specifying the names of your own functions for taking care of the distinct tasks
  • The handler PHP uses to handle data serialization is defined by the session.serialize_handler configuration directive. It is set to php by default.
  • REPLACE
  • REPLACE, which behaves exactly like INSERT, except that it handles cases where a record already exists with the same session identifier by first deleting that record.
  • the _write() function keeps the timestamp of the last access in the access column for each record, this can be used to determine which records to delete.
張 旭

第 05 章 - 計算機概論 - 主機系統與 I/O 界面 - 0 views

dic.vbird.tw/...2020unit05.php

shared by 張 旭 on 21 Jun 21 - No Cached
  • 接受使用者輸入指令與資料，經由中央處理器的數學與邏輯單元運算處理後，以產生或儲存成有用的資訊
  • CPU 為一個具有特定功能的晶片， 裡頭含有微指令集，如果你想要讓主機進行什麼特異的功能，就得要參考這顆 CPU 是否有相關內建的微指令集才可以。
  • CPU 讀取的資料都是從主記憶體來的！ 主記憶體內的資料則是從輸入單元所傳輸進來！而 CPU 處理完畢的資料也必須要先寫回主記憶體中，最後資料才從主記憶體傳輸到輸出單元。
  • ...49 more annotations...
  • 算數邏輯單元與控制單元。其中算數邏輯單元主要負責程式運算與邏輯判斷，控制單元則主要在協調各周邊元件與各單元間的工作。
  • 資料會先寫入到主記憶體，然後 CPU 才能開始應用
  • CPU 其實內部已經含有一些微指令，我們所使用的軟體都要經過 CPU 內部的微指令集來達成才行。
  • 世界上常見到的兩種主要 CPU 架構， 分別是：精簡指令集 (RISC) 與複雜指令集 (CISC) 系統。
  • 微指令集較為精簡，每個指令的執行時間都很短，完成的動作也很單純，指令的執行效能較佳； 但是若要做複雜的事情，就要由多個指令來完成。
  • CISC在微指令集的每個小指令可以執行一些較低階的硬體操作，指令數目多而且複雜， 每條指令的長度並不相同。因為指令執行較為複雜所以每條指令花費的時間較長， 但每條個別指令可以處理的工作較為豐富。
  • 最早的那顆Intel發展出來的CPU代號稱為8086
  • AMD依此架構修改新一代的CPU為64位元
  • CPU 位元指的是CPU一次資料讀取的最大量！64位元CPU代表CPU一次可以讀寫64bits這麼多的資料
  • 因為CPU讀取資料量有限制，因此能夠從記憶體中讀寫的資料也就有所限制
  • ：(1)北橋：負責連結速度較快的CPU、主記憶體與顯示卡界面等元件；
  • (2)南橋：負責連接速度較慢的裝置介面， 包括硬碟、USB、網路卡等等。不過由於北橋最重要的就是 CPU 與主記憶體之間的橋接，因此目前的主流架構中， 大多將北橋記憶體控制器整合到 CPU 封裝當中
  • CPU 與可接受的晶片組是有搭配性的，尤其目前每種 CPU 的腳位都不一樣
  • 時脈越高，代表單位時間內可進行的工作越多
  • 大概都只看總頻寬或基準時脈
  • CPU 時脈越高的情況下，會產生很多諸如散熱、設計及與週邊元件溝通的問題。
  • 其他的程序還在等待 CPU ，但是 CPU 又在等待 I/O
  • Intel 在同一個運算核心底下安裝兩個暫存器來模擬出另一個核心，實際上是兩個暫存器 (可以想成是程式的執行器) 共用一個實體核心。 這就是超執行緒的簡易認知
  • 時脈的意思是每秒鐘能夠進行的工作次數，而每次工作能夠讀進的資料量就是位元數
  • 每個用戶端大多是短時間的大運算，所以通常不會有一隻程序一直佔用著系統資源。
  • CPU 的所有資料都是從記憶體讀寫來的，所以記憶體的容量與頻寬就相當的重要了
  • DRAM 根據技術的更新又分好幾代，而使用上較廣泛的有所謂的 SDRAM 與 DDR SDRAM 兩種。 這兩種記憶體的差別除了在於腳位與工作電壓上的不同之外，DDR 是所謂的雙倍資料傳送速度 (Double Data Rate)
  • 晶片組廠商就將兩個主記憶體彙整在一起，如果一支記憶體可達 64 位元，兩支記憶體就可以達到 128 位元了，這就是雙通道的設計理念。 在某些比較多核心的伺服器主機上面，甚至還使用了 3 通道到 4 通道的設計
  • CPU 內部的暫存器與少量記憶體被稱為第一、第二層快取 (L1, L2 cache)，而放在核心間的快取記憶體就被稱為第三層快取記憶體 (L3 cache)
  • CPU 的暫存器是直接設計在 CPU 核心內部，可以用來幫助 CPU 的運算。
  • 在多核心 CPU 的核心之間，會有另一個共享的快取記憶體存在，讓所有的 CPU 核心可以共享某些資源。
  • 靜態隨機存取記憶體 (Static Random Access Memory, SRAM)
  • 各項參數， 是被記錄到主機板上頭的一個稱為 CMOS 的晶片上，這個晶片需要藉著額外的電源來發揮記錄功能， 這也是為什麼你的主機板上面會有一顆電池的緣故。
  • BIOS(Basic Input Output System)是一套程式，這套程式是寫死到主機板上面的一個記憶體晶片中， 這個記憶體晶片在沒有通電時也能夠將資料記錄下來，那就是唯讀記憶體(Read Only Memory, ROM)。
  • BIOS對於個人電腦來說是非常重要的， 因為他是系統在開機的時候首先會去讀取的一個小程式喔
  • 韌體(firmware)很多也是使用ROM來進行軟體的寫入的。 韌體像軟體一樣也是一個被電腦所執行的程式，然而他是對於硬體內部而言更加重要的部分。例如BIOS就是一個韌體， BIOS雖然對於我們日常操作電腦系統沒有什麼太大的關係，但是他卻控制著開機時各項硬體參數的取得！
  • 現在的 BIOS 通常是寫入類似快閃記憶體 (flash) 或 EEPROM
  • 顯示卡能夠傳輸的資料量當然也是越大越好！這時就得要考慮到傳輸的界面了！當前 (2018) 主流的傳輸界面為 PCI-E，這個 PCI-E 又有三種版本， version 1, 2, 3 ，這三個版本的速度並不相同
  • PCI-E (PCI-Express) 使用的是類似管線的概念來處理，在 PCI-E 第一版中，每條管線可以具有 250MBytes/s 的頻寬效能，管線越多(通常設計到 x16 管線)則總頻寬越高
  • 通常 CPU 製造商會根據 CPU 來設計搭配的南橋晶片，由於現在只有一個晶片 (北橋整合到 CPU 內了)，所以目前只有一顆主機板晶片組。
  • 插槽上面的資料要傳輸到 CPU 就得要經過晶片組，然後透過 DMI 通道傳上去！所以，在某些情況下，系統的效能會被卡住在這條通道上喔！
  • 晶片組接的設備相當多喔！有 PCI-E 插槽、USB設備、網路設備、音效設備、硬碟設備等，這全部的裝置共用那一條 DMI 喔！ 所以，整個系統效能的瓶頸通常不在 CPU 啦！通常就是在南橋接的設備上面！所以，當你有非常複雜的程式要運作的時候，讓這些程式越少通過南橋， 他的系統效能就會比較好一點
  • 物理組成
  • 讀寫主要是透過在機械手臂上的讀取頭(head)來達成
  • 由於磁碟盤是圓的，並且透過機器手臂去讀寫資料，磁碟盤要轉動才能夠讓機器手臂讀寫。
  • 磁碟的最小物理儲存單位，稱之為磁區 (sector)，那同一個同心圓的磁區組合成的圓就是所謂的磁軌(track)。 由於磁碟裡面可能會有多個磁碟盤，因此在所有磁碟盤上面的同一個磁軌可以組合成所謂的磁柱 (cylinder)。
  • 通常資料的讀寫會由外圈開始往內寫
  • 目前主流的硬碟連接界面就是 SATA
  • 雖然 SATA III 界面理論傳輸可到達 600Mbytes/s，不過傳統硬碟由於物理設計的限制因素，通常存取速度效能大多在 150~200Mbytes/s 之間
  • 串列式 SCSI (Serial Attached SCSI, SAS)
  • 傳統硬碟有個很致命的問題，就是需要驅動馬達去轉動磁碟盤～這會造成很嚴重的磁碟讀取延遲
  • 快閃記憶體去製作成高容量的設備
  • 沒有讀寫頭與磁碟盤啊！都是記憶體！
  • 各個繪圖卡的運算晶片 (GPU) 設計的理念不同，加上驅動程式與軟體搭配的問題，並不是一張高效能繪圖卡就可以完勝其他的繪圖卡， 還得要注意相關的軟體才行。
1 - 8 of 8
Showing 20▼ items per page