Skip to main content

Home/ Larvata/ Group items tagged classes

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
張 旭

Scalable architecture without magic (and how to build it if you're not Google) - DEV Co... - 0 views

dev.to/...ld-it-if-youre-not-google-336a

development architecture system google

shared by 張 旭 on 24 Feb 19 - No Cached
  • Don’t mess up write-first and read-first databases.
  • keep them stateless.
  • you should know how to make a scalable setup on bare metal.
  • ...29 more annotations...
  • Different programming languages are for different tasks.
  • Go or C which are compiled to run on bare metal.
  • To run NodeJS on multiple cores, you have to use something like PM2, but since this you have to keep your code stateless.
  • Python have very rich and sugary syntax that’s great for working with data while keeping your code small and expressive.
  • SQL is almost always slower than NoSQL
  • databases are often read-first or write-first
  • write-first, just like Cassandra.
  • store all of your data to your databases and leave nothing at backend
  • Functional code is stateless by default
  • It’s better to go for stateless right from the very beginning.
  • deliver exactly the same responses for same requests.
  • Sessions? Store them at Redis and allow all of your servers to access it.
  • Only the first user will trigger a data query, and all others will be receiving exactly the same data straight from the RAM
  • never, never cache user input
  • Only the server output should be cached
  • Varnish is a great cache option that works with HTTP responses, so it may work with any backend.
  • a rate limiter – if there’s not enough time have passed since last request, the ongoing request will be denied.
  • different requests blasting every 10ms can bring your server down
  • Just set up entry relations and allow your database to calculate external keys for you
  • the query planner will always be faster than your backend.
  • Backend should have different responsibilities: hashing, building web pages from data and templates, managing sessions and so on.
  • For anything related to data management or data models, move it to your database as procedures or queries.
  • a distributed database.
  • your code has to be stateless
  • Move anything related to the data to the database.
  • For load-balancing a database, go for cluster.
  • DB is balancing requests, as well as your backend.
  • Users from different continents are separated with DNS.
  • Keep is scalable, keep is stateless.
  • 張 旭 on 24 Feb 19
     
    "Don't mess up write-first and read-first databases."
張 旭

Introduction To The Queue System - Diving Laravel - 0 views

divinglaravel.com/troduction-to-the-queue-system

php programming laravel

shared by 張 旭 on 02 Mar 19 - No Cached
  • Laravel is shipped with a built-in queue system that helps you run tasks in the background
  • The QueueManager is registered into the container and it knows how to connect to the different built-in queue drivers
  • for example when we called the Queue::push() method, what happened is that the manager selected the desired queue driver, connected to it, and called the push method on that driver.
  • ...2 more annotations...
  • All calls to methods that don't exist in the QueueManager class will be sent to the loaded driver
  • when you do Queue::push() you're actually calling the push method on the queue driver you're using
  • 張 旭 on 02 Mar 19
     
    "Laravel is shipped with a built-in queue system that helps you run tasks in the background "
張 旭

DevOps - 0 views

mp.weixin.qq.com/...oELHwosd26dS4_qh5HEZRw

devops system

shared by 張 旭 on 27 Feb 19 - No Cached
  • 对于运维来说，知识的传承非常重要，非常有必要建立运维的知识库。一方面 有利于对事件的复盘回顾，另一方面也有助于日后参加运维的人员尽快熟悉与掌握系统的运维技能
  • 云平台主要从以下3个方面对DevOps提供支撑（括号内为承载此能力的软件工具）： 1. 基于IaaS的自服务与环境编排能力（VMWare） 2. 基于PaaS的弹性伸缩能力（K8s） 3. 基于SaaS的软件服务能力
  • 考虑自建私有云，至少是混合云。
  • ...11 more annotations...
  • 内网建立所谓的私库，作为代理与外网的公共库同步。
  • 很难做到真正意义上的DevOps to Production
  • 可视化是为了实时展现持续交付流水线执行情况与单元测试的执行报告
  • 通过持续交付流水线串联自动化测试，在测试环境部署成功后触发自动化测试。
  • 测试阶段也需要测试报告的可视化与结果通知
  • 企业的持续交付流水线往往都打不通到生产环境
  • Service Desk不是某一款软件的名字，而是ITIL（信息技术基础架构库，可认为是ITSM的落地实现）里面承载变更管理与事件管理的工具统称。
  • 构建底层的云平台，是整个DevOps基础架构的基石
  • 架构不是一成不变的，而是应该随着实际需求变化而持续演化，能力也要跟着持续提升。
  • 并行测试的执行环境通过PaaS平台按需自动生成，测试执行完毕后自动销毁。
  • 即使是雷同的项目，在对编译构建上的一些细枝末节的差别也很可能导致它们的持续交付流水线设计非常不一样。
  • 張 旭 on 27 Feb 19
     
    "对于运维来说，知识的传承非常重要，非常有必要建立运维的知识库。一方面 有利于对事件的复盘回顾，另一方面也有助于日后参加运维的人员尽快熟悉与掌握系统的运维技能。"
張 旭

The Twelve-Factor App - 0 views

12factor.net/config

development programming system software

shared by 張 旭 on 01 Jan 19 - No Cached
  • An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc)
  • Resource handles
  • Credentials
  • ...8 more annotations...
  • Per-deploy values
  • trict separation of config from code.
  • Config varies substantially across deploys, code does not.
  • he codebase could be made open source at any moment, without compromising any credentials.
  • “config” does not include internal application config
  • stores config in environment variables (often shortened to env vars or env).
  • env vars are granular controls, each fully orthogonal to other env vars
  • They are never grouped together as “environments”
張 旭

HTTPS 升级指南 - 阮一峰的网络日志 - 0 views

www.ruanyifeng.com/...igrate-from-http-to-https.html

ssh security network https

shared by 張 旭 on 19 Jan 19 - No Cached
  • 域名认证（Domain Validation）：最低级别认证，可以确认申请人拥有这个域名。
  • 公司认证（Company Validation）：确认域名所有人是哪一家公司，证书里面会包含公司信息。
  • 扩展认证（Extended Validation）：最高级别的认证，浏览器地址栏会显示公司名。
  • ...8 more annotations...
  • 多域名
  • 单域名
  • 通配符
  • 网站的响应头里面，加入一个强制性声明
  • Strict-Transport-Security: max-age=31536000; includeSubDomains
  • 确保浏览器只在使用 HTTPS 时，才发送Cookie。
  • Set-Cookie:
  • ; Secure
張 旭

MySQL 到底能不能放到 Docker 里跑？ - 0 views

mp.weixin.qq.com/s

mysql server docker

shared by 張 旭 on 07 Aug 17 - No Cached
  • 忙碌又容易出错的工作其实是无意义的
  • 单机多实例运行 MySQL
  • MySQL 运行的就是个进程而且对 IO 要求比较高
  • ...12 more annotations...
  • Docker 的资源限制用的就是 cgroups
  • Percona：我们的备份、慢日志分析、过载保护等功能都是基于 pt-tools 工具包来实现的。
  • Consul：分布式的服务发现和配置共享软件
  • 容器调度的开源产品主要有 Kubernetes 和 mesos
  • 适合自己现状的需求才是最好的
  • 有机会做到计算调度和存储调度分离的情况下我们可能会转向 Kubernetes 的方案
  • 根据这个需求按照我们的资源筛选规则 (比如主从不能在同一台机器、内存配置不允许超卖等等)，从现有的资源池中匹配出可用资源，然后依次创建主从关系、创建高可用管理、检查集群复制状态、推送集群信息到中间件 (选用了中间件的情况下) 控制中心、最后将以上相关信息都同步到 CMDB。
    • 張 旭
      張 旭 on 07 Aug 17
       
      感覺用 K8S 就不用那麼麻煩了。
  • 每一个工作都是通过服务端发送消息到 agent，然后由 agent 执行对应的脚本，脚本会返回指定格式的执行结果
  • 备份工具我们是用 percona-xtrabackup
  • zabbix 来实现监控告警
  • grafana 是监控画图界的扛把子，功能齐全的度量仪表盘和图形编辑器，经过简单配置就能完成各种监控图形的展示。
  • (MariaDB 不支持写 table，只能写 file)，极大减少了从库复制带来的 IOPS。
張 旭

The Asset Pipeline - Ruby on Rails Guides - 0 views

guides.rubyonrails.org/asset_pipeline.html

rails image javascript font CSS file

shared by 張 旭 on 29 Dec 14 - No Cached
  • provides a framework to concatenate and minify or compress JavaScript and CSS assets
  • adds the ability to write these assets in other languages and pre-processors such as CoffeeScript, Sass and ERB
  • invalidate the cache by altering this fingerprint
  • ...80 more annotations...
  • Rails 4 automatically adds the sass-rails, coffee-rails and uglifier gems to your Gemfile
  • reduce the number of requests that a browser makes to render a web page
  • Starting with version 3.1, Rails defaults to concatenating all JavaScript files into one master .js file and all CSS files into one master .css file
  • In production, Rails inserts an MD5 fingerprint into each filename so that the file is cached by the web browser
  • The technique sprockets uses for fingerprinting is to insert a hash of the content into the name, usually at the end.
  • asset minification or compression
  • The sass-rails gem is automatically used for CSS compression if included in Gemfile and no config.assets.css_compressor option is set.
  • Supported languages include Sass for CSS, CoffeeScript for JavaScript, and ERB for both by default.
  • When a filename is unique and based on its content, HTTP headers can be set to encourage caches everywhere (whether at CDNs, at ISPs, in networking equipment, or in web browsers) to keep their own copy of the content
  • asset pipeline is technically no longer a core feature of Rails 4
  • Rails uses for fingerprinting is to insert a hash of the content into the name, usually at the end
  • With the asset pipeline, the preferred location for these assets is now the app/assets directory.
  • Fingerprinting is enabled by default for production and disabled for all other environments
  • The files in app/assets are never served directly in production.
  • Paths are traversed in the order that they occur in the search path
  • You should use app/assets for files that must undergo some pre-processing before they are served.
  • By default .coffee and .scss files will not be precompiled on their own
  • app/assets is for assets that are owned by the application, such as custom images, JavaScript files or stylesheets.
  • lib/assets is for your own libraries' code that doesn't really fit into the scope of the application or those libraries which are shared across applications.
  • vendor/assets is for assets that are owned by outside entities, such as code for JavaScript plugins and CSS frameworks.
  • Any path under assets/* will be searched
  • By default these files will be ready to use by your application immediately using the require_tree directive.
  • By default, this means the files in app/assets take precedence, and will mask corresponding paths in lib and vendor
  • Sprockets uses files named index (with the relevant extensions) for a special purpose
  • Rails.application.config.assets.paths
  • causes turbolinks to check if an asset has been updated and if so loads it into the page
  • if you add an erb extension to a CSS asset (for example, application.css.erb), then helpers like asset_path are available in your CSS rules
  • If you add an erb extension to a JavaScript asset, making it something such as application.js.erb, then you can use the asset_path helper in your JavaScript code
  • The asset pipeline automatically evaluates ERB
  • data URI — a method of embedding the image data directly into the CSS file — you can use the asset_data_uri helper.
  • Sprockets will also look through the paths specified in config.assets.paths, which includes the standard application paths and any paths added by Rails engines.
  • image_tag
  • the closing tag cannot be of the style -%>
  • asset_data_uri
  • app/assets/javascripts/application.js
  • sass-rails provides -url and -path helpers (hyphenated in Sass, underscored in Ruby) for the following asset classes: image, font, video, audio, JavaScript and stylesheet.
  • Rails.application.config.assets.compress
  • In JavaScript files, the directives begin with //=
  • The require_tree directive tells Sprockets to recursively include all JavaScript files in the specified directory into the output.
  • manifest files contain directives — instructions that tell Sprockets which files to require in order to build a single CSS or JavaScript file.
  • You should not rely on any particular order among those
  • Sprockets uses manifest files to determine which assets to include and serve.
  • the family of require directives prevents files from being included twice in the output
  • which files to require in order to build a single CSS or JavaScript file
  • Directives are processed top to bottom, but the order in which files are included by require_tree is unspecified.
  • In JavaScript files, Sprockets directives begin with //=
  • If require_self is called more than once, only the last call is respected.
  • require directive is used to tell Sprockets the files you wish to require.
  • You need not supply the extensions explicitly. Sprockets assumes you are requiring a .js file when done from within a .js file
  • paths must be specified relative to the manifest file
  • require_directory
  • Rails 4 creates both app/assets/javascripts/application.js and app/assets/stylesheets/application.css regardless of whether the --skip-sprockets option is used when creating a new rails application.
  • The file extensions used on an asset determine what preprocessing is applied.
  • app/assets/stylesheets/application.css
  • Additional layers of preprocessing can be requested by adding other extensions, where each extension is processed in a right-to-left manner
  • require_self
  • use the Sass @import rule instead of these Sprockets directives.
  • Keep in mind that the order of these preprocessors is important
  • In development mode, assets are served as separate files in the order they are specified in the manifest file.
  • when these files are requested they are processed by the processors provided by the coffee-script and sass gems and then sent back to the browser as JavaScript and CSS respectively.
  • css.scss.erb
  • js.coffee.erb
  • Keep in mind the order of these preprocessors is important.
  • By default Rails assumes that assets have been precompiled and will be served as static assets by your web server
  • with the Asset Pipeline the :cache and :concat options aren't used anymore
  • Assets are compiled and cached on the first request after the server is started
  • RAILS_ENV=production bundle exec rake assets:precompile
  • Debug mode can also be enabled in Rails helper methods
  • If you set config.assets.initialize_on_precompile to false, be sure to test rake assets:precompile locally before deploying
  • By default Rails assumes assets have been precompiled and will be served as static assets by your web server.
  • a rake task to compile the asset manifests and other files in the pipeline
  • RAILS_ENV=production bin/rake assets:precompile
  • a recipe to handle this in deployment
  • links the folder specified in config.assets.prefix to shared/assets
  • config/initializers/assets.rb
  • The initialize_on_precompile change tells the precompile task to run without invoking Rails
  • The X-Sendfile header is a directive to the web server to ignore the response from the application, and instead serve a specified file from disk
  • the jquery-rails gem which comes with Rails as the standard JavaScript library gem.
  • Possible options for JavaScript compression are :closure, :uglifier and :yui
  • concatenate assets
張 旭

Keycloak and FreeIPA Intro - scott poore's blog - 0 views

spoore.wordpress.com/...keycloak-and-freeipa-intro

authorization authentication oauth system security freeipa user ldap

shared by 張 旭 on 12 Jul 19 - No Cached
  • Keycloak is an “Open source identity and access management” solution.
  • setup a central Identity Provider (IdP) that applications acting as Service Providers (SP) use to authenticate or authorize user access.
  • FreeIPA does a LOT more than just provide user info though.  It can manage user groups, service lists, hosts, DNS, certificates, and much, much, more.
  • ...5 more annotations...
  • IPA – refers to the FreeIPA Master Server.
  • IdP – as mentioned earlier, IdP stands for Identity Provider.
  • SP – stands for Service Provider.   This can be a java application, jboss, etc.  It can also be a simple Apache web server
  • SAML – stands for Security Assertion Markup Language and refers to mod_auth_mellon here.  This provides the SSO functionality.
  • Openidc – stands for OpenID Connect.
張 旭

How to create reusable infrastructure with Terraform modules - 0 views

blog.gruntwork.io/terraform-modules-25526d65f73d

terraform devops programming module

shared by 張 旭 on 05 Sep 19 - No Cached
  • auto scaling schedule
  • The easiest way to create a versioned module is to put the code for the module in a separate Git repository and to set the source parameter to that repository’s URL.
張 旭

Understanding Nginx HTTP Proxying, Load Balancing, Buffering, and Caching | DigitalOcean - 0 views

www.digitalocean.com/...alancing-buffering-and-caching

nginx proxy networking HA haproxy

shared by 張 旭 on 14 Mar 17 - No Cached
  • allow Nginx to pass requests off to backend http servers for further processing
  • Nginx is often set up as a reverse proxy solution to help scale out infrastructure or to pass requests to other servers that are not designed to handle large client loads
  • explore buffering and caching to improve the performance of proxying operations for clients
  • ...48 more annotations...
  • Nginx is built to handle many concurrent connections at the same time.
  • provides you with flexibility in easily adding backend servers or taking them down as needed for maintenance
  • Proxying in Nginx is accomplished by manipulating a request aimed at the Nginx server and passing it to other servers for the actual processing
  • The servers that Nginx proxies requests to are known as upstream servers.
  • Nginx can proxy requests to servers that communicate using the http(s), FastCGI, SCGI, and uwsgi, or memcached protocols through separate sets of directives for each type of proxy
  • When a request matches a location with a proxy_pass directive inside, the request is forwarded to the URL given by the directive
  • For example, when a request for /match/here/please is handled by this block, the request URI will be sent to the example.com server as http://example.com/match/here/please
  • The request coming from Nginx on behalf of a client will look different than a request coming directly from a client
  • Nginx gets rid of any empty headers
  • Nginx, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request
    • 張 旭
      張 旭 on 16 Mar 17
       
      這裡要注意一下，header 欄位名稱有設定底線的，要設定 Nginx 讓它可以通過。
  • The "Host" header is re-written to the value defined by the $proxy_host variable.
  • The upstream should not expect this connection to be persistent
  • Headers with empty values are completely removed from the passed request.
  • if your backend application will be processing non-standard headers, you must make sure that they do not have underscores
  • by default, this will be set to the value of $proxy_host, a variable that will contain the domain name or IP address and port taken directly from the proxy_pass definition
  • This is selected by default as it is the only address Nginx can be sure the upstream server responds to
  • (as it is pulled directly from the connection info)
  • $http_host: Sets the "Host" header to the "Host" header from the client request.
  • The headers sent by the client are always available in Nginx as variables. The variables will start with an $http_ prefix, followed by the header name in lowercase, with any dashes replaced by underscores.
  • preference to: the host name from the request line itself
  • set the "Host" header to the $host variable. It is the most flexible and will usually provide the proxied servers with a "Host" header filled in as accurately as possible
  • sets the "Host" header to the $host variable, which should contain information about the original host being requested
  • This variable takes the value of the original X-Forwarded-For header retrieved from the client and adds the Nginx server's IP address to the end.
  • The upstream directive must be set in the http context of your Nginx configuration.
  • http context
  • Once defined, this name will be available for use within proxy passes as if it were a regular domain name
  • By default, this is just a simple round-robin selection process (each request will be routed to a different host in turn)
  • Specifies that new connections should always be given to the backend that has the least number of active connections.
  • distributes requests to different servers based on the client's IP address.
  • mainly used with memcached proxying
  • As for the hash method, you must provide the key to hash against
  • Server Weight
  • Nginx's buffering and caching capabilities
  • Without buffers, data is sent from the proxied server and immediately begins to be transmitted to the client.
  • With buffers, the Nginx proxy will temporarily store the backend's response and then feed this data to the client
  • Nginx defaults to a buffering design
  • can be set in the http, server, or location contexts.
  • the sizing directives are configured per request, so increasing them beyond your need can affect your performance
  • When buffering is "off" only the buffer defined by the proxy_buffer_size directive will be used
  • A high availability (HA) setup is an infrastructure without a single point of failure, and your load balancers are a part of this configuration.
  • multiple load balancers (one active and one or more passive) behind a static IP address that can be remapped from one server to another.
  • Nginx also provides a way to cache content from backend servers
  • The proxy_cache_path directive must be set in the http context.
  • proxy_cache backcache;
    • 張 旭
      張 旭 on 16 Mar 17
       
      這裡的 backcache 是前文設定的 backcache 變數，看起來每個 location 都可以有自己的 cache 目錄。
  • The proxy_cache_bypass directive is set to the $http_cache_control variable. This will contain an indicator as to whether the client is explicitly requesting a fresh, non-cached version of the resource
  • any user-related data should not be cached
  • For private content, you should set the Cache-Control header to "no-cache", "no-store", or "private" depending on the nature of the data
張 旭

Rails Environment Variables · RailsApps - 1 views

railsapps.github.io/ils-environment-variables.html

rails environment system variable

shared by 張 旭 on 13 Mar 17 - No Cached
  • You can pass local configuration settings to an application using environment variables.
  • Operating systems (Linux, Mac OS X, Windows) provide mechanisms to set local environment variables, as does Heroku and other deployment platforms.
  • In general, you shouldn’t save email account credentials or private API keys to a shared git repository.
  • ...10 more annotations...
  • You could “hardcode” your Gmail username and password into the file but that would expose it to everyone who has access to your git repository.
  • It’s important to learn to use the Unix shell if you’re commited to improving your skills as a developer.
  • The gem reads a config/application.yml file and sets environment variables before anything else is configured in the Rails application.
  • make sure this file is listed in the .gitignore file so it isn’t checked into the git repository
  • Rails provides a config.before_configuration
  • YAML.load(File.open(env_file)).each do |key, value| ENV[key.to_s] = value end if File.exists?(env_file)
  • Heroku is a popular choice for low cost, easily configured Rails application hosting.
  • heroku config:add
  • the dotenv Ruby gem
  • Foreman is a tool for starting and configuring multiple processes in a complex application
張 旭

plataformatec/devise: Flexible authentication solution for Rails with Warden. - 0 views

github.com/devise

rails user devise system login authorization authentication

shared by 張 旭 on 22 Jan 14 - No Cached
  • we advise you to start a simple authentication system from scratch
  • If you are building your first Rails application, we recommend you do not use Devise. Devise requires a good understanding of the Rails Framework
  • The generator will install an initializer which describes ALL of Devise's configuration options
  • ...6 more annotations...
  • Replace MODEL with the class name used for the application’s users (it’s frequently User but could also be Admin)
  • If you add an option, be sure to inspect the migration file (created by the generator if your ORM supports them) and uncomment the appropriate section
  • set up the default URL options for the Devise mailer in each environment
  • should restart your application after changing Devise's configuration options
  • set up a controller with user authentication, just add this before_action
  • when using a :user resource, the user_root_path will be used if it exists; otherwise, the default root_path will be used
張 旭

MetalLB, bare metal load-balancer for Kubernetes - 0 views

metallb.universe.tf

kubernetes network

shared by 張 旭 on 14 Apr 21 - No Cached
  • Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters
  • If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.
  • Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services.
張 旭

Introducing Infrastructure as Code | Linode - 0 views

www.linode.com/...tion-to-infrastructure-as-code

devops server infrastructure

shared by 張 旭 on 14 Apr 21 - No Cached
  • Infrastructure as Code (IaC) is a technique for deploying and managing infrastructure using software, configuration files, and automated tools.
  • With the older methods, technicians must configure a device manually, perhaps with the aid of an interactive tool. Information is added to configuration files by hand or through the use of ad-hoc scripts. Configuration wizards and similar utilities are helpful, but they still require hands-on management. A small group of experts owns the expertise, the process is typically poorly defined, and errors are common.
  • The development of the continuous integration and continuous delivery (CI/CD) pipeline made the idea of treating infrastructure as software much more attractive.
  • ...20 more annotations...
  • Infrastructure as Code takes advantage of the software development process, making use of quality assurance and test automation techniques.
  • Consistency/Standardization
  • Each node in the network becomes what is known as a snowflake, with its own unique settings. This leads to a system state that cannot easily be reproduced and is difficult to debug.
  • With standard configuration files and software-based configuration, there is greater consistency between all equipment of the same type. A key IaC concept is idempotence.
  • Idempotence makes it easy to troubleshoot, test, stabilize, and upgrade all the equipment.
  • Infrastructure as Code is central to the culture of DevOps, which is a mix of development and operations
  • edits are always made to the source configuration files, never on the target.
  • A declarative approach describes the final state of a device, but does not mandate how it should get there. The specific IaC tool makes all the procedural decisions. The end state is typically defined through a configuration file, a JSON specification, or a similar encoding.
  • An imperative approach defines specific functions or procedures that must be used to configure the device. It focuses on what must happen, but does not necessarily describe the final state. Imperative techniques typically use scripts for the implementation.
  • With a push configuration, the central server pushes the configuration to the destination device.
  • If a device is mutable, its configuration can be changed while it is active
  • Immutable devices cannot be changed. They must be decommissioned or rebooted and then completely rebuilt.
  • an immutable approach ensures consistency and avoids drift. However, it usually takes more time to remove or rebuild a configuration than it does to change it.
  • System administrators should consider security issues as part of the development process.
  • Ansible is a very popular open source IaC application from Red Hat
  • Ansible is often used in conjunction with Kubernetes and Docker.
  • Linode offers a collection of several Ansible guides for a more comprehensive overview.
  • Pulumi permits the use of a variety of programming languages to deploy and manage infrastructure within a cloud environment.
  • Terraform allows users to provision data center infrastructure using either JSON or Terraform’s own declarative language.
  • Terraform manages resources through the use of providers, which are similar to APIs.
張 旭

[Elasticsearch] 分散式特性 & 分散式搜尋的機制 | 小信豬的原始部落 - 0 views

godleon.github.io/...icsearch-distributed-mechanism

elasticsearch database

shared by 張 旭 on 17 Apr 21 - No Cached
  • 水平擴展儲存空間
  • Data HA：若有 node 掛掉，資料不會遺失
  • 若是要查詢 cluster 中的 node 狀態，可以使用 GET /_cat/nodes API
  • ...39 more annotations...
  • 決定每個 shard 要被分配到哪個 data node 上
  • 為 cluster 設置多個 master node
  • 一旦發現被選中的 master node 出現問題，就會選出新的 master node
  • 每個 node 啟動時就預設是一個 master eligible node，可以透過設定 node.master: false 取消此預設設定
  • 處理 request 的 node 稱為 Coordinating Node，其功能是將 request 轉發到合適的 node 上
  • 所有的 node 都預設是 Coordinating Node
  • coordinating node 可以直接接收 search request 並處理，不需要透過 master node 轉過來
  • 可以保存資料的 node，每個 node 啟動後都會預設是 data node，可以透過設定 node.data: false 停用 data node 功能
  • 由 master node 決定如何把分片分發到不同的 data node 上
  • 每個 node 上都保存了 cluster state
  • 只有 master 才可以修改 cluster state 並負責同步給其他 node
  • 每個 node 都會詳細紀錄本身的狀態資訊
  • shard 是 Elasticsearch 分散式儲存的基礎，包含 primary shard & replica shard
  • 每一個 shard 就是一個 Lucene instance
  • primary shard 功能是將一份被索引後的資料，分散到多個 data node 上存放，實現儲存方面的水平擴展
  • primary shard 的數量在建立 index 時就會指定，後續是無法修改的，若要修改就必須要進行 reindex
  • 當 primary shard 遺失時，replica shard 就可以被 promote 成 primary shard 來保持資料完整性
  • replica shard 數量可以動態調整，讓每個 data node 上都有完整的資料
  • ES 7.0 開始，primary shard 預設為 1，replica shard 預設為 0
  • replica shard 若設定過多，會降低 cluster 整體的寫入效能
  • replica shard 必須和 primary shard 被分配在不同的 data node 上
  • 所有的 primary shard 可以在同一個 data node 上
  • 透過 GET _cluster/health/<target> 可以取得目前 cluster 的健康狀態
  • Yellow：表示 primary shard 可以正常分配，但 replica shard 分配有問題
  • 透過 GET /_cat/shards/<target> 可以取得目前的 shard 狀態
  • replica shard 無法被分配，因此 cluster 健康狀態為黃色
  • 若是擔心 reboot 機器造成 failover 動作開始執行，可以設定將 replication 延遲一段時間後再執行(透過調整 settings 中的 index.unassigned.node_left.delayed_timeout 參數)，避免無謂的 data copy 動作 (此功能稱為 delay allocation)
  • 集群變紅，代表有 primary shard 丟失，這個時候會影響讀寫。
  • 如果 node 重新回來，會從 translog 中恢復沒有寫入的資料
  • 設定 index settings 之後，primary shard 數量無法隨意變更
  • 不建議直接發送請求到master節點，雖然也會工作，但是大量請求發送到 master，會有潛在的性能問題
  • shard 是 ES 中最小的工作單元
  • shard 是一個 Lucene 的 index
  • 將 Index Buffer 中的內容寫入 Segment，而這寫入的過程就稱為 Refresh
  • 當 document 被 refresh 進入到 segment 之後，就可以被搜尋到了
  • 在進行 refresh 時先將 segment 寫入 cache 以開放查詢
  • 將 document 進行索引時，同時也會寫入 transaction log，且預設都會寫入磁碟中
  • 每個 shard 都會有對應的 transaction log
  • 由於 transaction log 都會寫入磁碟中，因此當 node 從故障中恢復時，就會優先讀取 transaction log 來恢復資料
張 旭

Extend the Kubernetes API with CustomResourceDefinitions | Kubernetes - 0 views

kubernetes.io/...custom-resource-definitions

kubernetes

shared by 張 旭 on 12 May 21 - No Cached
  • When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server creates a new RESTful resource path for each version you specify.
  • The CRD can be either namespaced or cluster-scoped, as specified in the CRD's scope field
  • deleting a namespace deletes all custom objects in that namespace.
  • ...7 more annotations...
  • CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.
  • Custom objects can contain custom fields. These fields can contain arbitrary JSON.
  • When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint and delete all custom objects stored in it
  • CustomResourceDefinitions store validated resource data in the cluster's persistence store, etcd.
  • By default, all unspecified fields for a custom resource, across all versions, are pruned.
  • The field json can store any JSON value, without anything being pruned.
  • Finalizers allow controllers to implement asynchronous pre-delete hooks.
張 旭

第 06 章 - 計算機概論 - 作業系統概論 - 0 views

dic.vbird.tw/...2020unit06.php

system

shared by 張 旭 on 22 Jun 21 - No Cached
  • 自行參考電腦硬體來設計出運算的軟體，當時的系統並沒有『作業系統』的概念，因為應用程式與作業系統是同時設計的。
  • 電腦裡面有儲存設備 (不論是硬碟還是記憶體)， 所以電腦硬體裡面會執行一隻監督程式 (monitor)，使用者可以預先將自己的程式讀進系統，系統先儲存該程式到佇列 (queue)，等到輪到該程式運作後， 就將該程式讀入讓 CPU 開始運作，直到運作結束輸出到印表機之後，將該工作丟棄，然後開始讀入在 queue 裡面的新的程式，依序執行。
  • 將 CPU 與 I/O 分離開
  • ...38 more annotations...
  • 透過卡片與讀卡機，將程式碼一次性的讀進大機器，然後就是等待大機器的運作， 結果再交由印表機印出。如果打卡紙打洞錯誤呢？只好重新打洞，重新排隊去運作程式了。
  • 允許兩個以上的程序在記憶體中等待被 CPU 執行，當 CPU 執行完其中一隻程式後， 第二隻程式就可以立刻被執行，因此效能會比較好。
  • 程序的狀態進入中斷狀態，CPU 不會理會該程序
  • CPU 的排程 (cpu scheduling)
  • 早期單核 CPU 的運作中，CPU 一次只能運作一個工作，因此，若有多個工作要同時進行， 那麼 CPU 就得要安排一個 CPU 運作時間給所有的工作，當該程序達到最大工作時間後，CPU 就會將該工作排回佇列，讓下一隻程序接著運作。
  • 你會覺得 CPU 是同時運作所有的程序，其實不是的！而是 CPU 在各個程序之間切換工作而已。
  • 分時系統其實與多元程式處理系統有點類似， 只是工作的輸入改為透過終端機操作輸入，CPU 可以在各個用戶操作間切換工作，於是每個使用者感覺似乎都是在同步操作電腦系統一般， 這就是分時系統。
  • 早期的程式設計師要設計程式是件苦差事，因為得要了解電腦硬體，並根據該電腦硬體來選擇程式語言，然後根據程式語言來設計運算工作、記憶體讀寫工作、 磁碟與影像輸入輸出工作、檔案存取工作等。等於從硬體、軟體、輸入輸出行為都得要在自己的程式碼裡面一口氣完成才行。
  • 在 1971 年開始的 unix 系統開發後，後續的系統大多使用 unix 的概念
  • 將硬體管理的工作統一交給一組程式碼去進行，而且這組程式碼還提供了一個開發界面
  • 軟體工程師只要依據這組程式碼規範的開發界面後，該軟體開發完成就能夠在這組程式碼上面運作了
  • 程式的執行
  • 作業系統需要將使用者交付的軟體程序分配到記憶體中， 然後透過 CPU 排程持續的交錯的完成各項任務才行。
  • CPU 中斷 (interrupt) 的功能
  • CPU 根據硬體擁有許多與週邊硬體的中斷通道， 當接收到中斷訊號時，CPU 就會嘗試將該程序列入等待的狀態下，讓該硬體自行完成相關的任務後，然後再接管系統。
  • 記憶體管理模組
  • 舊的環境底下，程式設計師需要自己判斷自己的程式會用到多少記憶體，然後自行指定記憶體使用位址的任務。
  • 系統會自動去偵測與管理主記憶體的使用狀態，避免同一個記憶體位址同時被兩個程序所使用而讓程序工作損毀
  • 作業系統核心也在記憶體中， 因此核心也會被這個子系統放入受保護的記憶體區段，一般用戶是無法直接操作該受保護的記憶區段的。
  • 虛擬記憶體 (virtual memory)
  • 主記憶體當中的資料並不是連續的，主記憶體的資料就像磁碟一樣，重複讀、刪、寫之後， 記憶區段是不會連續的
  • CPU 主要讀出虛擬記憶體，記憶體管理模組就會主動讀出資料
  • 一隻程序的資料是連續的 (左側)，但是實際上對應的是在主記憶體或其他位置上
  • CPU 排程
  • 作業系統好不好的重要指標之一！如何讓 CPU 在多工的情況下以最快速的方式將所有的工作完成，這方面的演算法是目前各主要作業系統持續在進步的部份。
  • 磁碟存取與檔案系統
  • 作業系統則需要驅動磁碟(不論是傳統硬碟還是 SSD)，然後也需要了解該磁碟內的檔案系統格式， 之後透過檔案系統這個子系統來進行資料的處理。
  • 裝置的驅動程式
  • 作業系統必須要能夠接受硬體裝置的驅動，所以硬體製造商可以推出給各個不同作業系統使用的驅動程式 (dirver / modules)， 這樣作業系統直接將該驅動程式載入後，即可開始使用該硬體，而不需要重新編譯作業系統。
  • 網路子系統
  • 使用者界面
  • 現代 CPU 設計的主要思考依據，讓一個 CPU 封裝 (單一一顆 CPU 硬體) 裡面，整合多個 CPU 核心，也就是多核心 CPU 製造的思考方向。
  • 對於單執行緒的程式來說， 多核心的 CPU 不見得會跑得比單核的快！這是因為單執行緒只有一個程序在進行，所以 CPU 時脈越高，代表會越快執行完畢。
  • 軟體會將單一工作拆分成數個小工作，分別交給不同的核心去執行，這樣每個核心只要負責一小段任務， 當然 CPU 時脈不用高，只要數量夠大，效能就會提昇很明顯
  • 由於 CPU 是由作業系統控制的，因此，你要使用到多核心的硬體系統，你的作業系統、應用程式都需要設計程可以支援多核心才行！
  • 所謂的平行處理功能，讓一件工作可以拆分成數個部份，讓這些不同的部份丟給不同的 CPU 去運算， 然後再透過一支監控程式，將各別的計算在一定的時間內收回統整後，再次的細分小工作發派出去，持續這些動作後，直到程式執行完畢為止。
  • 對於 Linux 來說，大部分都可以支援到 4096 個 CPU 核心數。
  • 銀行商用大型主機 Unix 系統
張 旭

LXC vs Docker: Why Docker is Better | UpGuard - 0 views

www.upguard.com/...docker-vs-lxc

docker container devops

shared by 張 旭 on 28 Jun 21 - No Cached
  • LXC (LinuX Containers) is a OS-level virtualization technology that allows creation and running of multiple isolated Linux virtual environments (VE) on a single control host.
  • Docker, previously called dotCloud, was started as a side project and only open-sourced in 2013. It is really an extension of LXC’s capabilities.
  • run processes in isolation.
  • ...35 more annotations...
  • Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure.
  • Docker acts as a portable container engine, packaging the application and all its dependencies in a virtual container that can run on any Linux server.
  • a VE there is no preloaded emulation manager software as in a VM.
  • In a VE, the application (or OS) is spawned in a container and runs with no added overhead, except for a usually minuscule VE initialization process.
  • LXC will boast bare metal performance characteristics because it only packages the needed applications.
  • the OS is also just another application that can be packaged too.
  • a VM, which packages the entire OS and machine setup, including hard drive, virtual processors and network interfaces. The resulting bloated mass usually takes a long time to boot and consumes a lot of CPU and RAM.
  • don’t offer some other neat features of VM’s such as IaaS setups and live migration.
  • LXC as supercharged chroot on Linux. It allows you to not only isolate applications, but even the entire OS.
  • Libvirt, which allows the use of containers through the LXC driver by connecting to 'lxc:///'.
  • 'LXC', is not compatible with libvirt, but is more flexible with more userspace tools.
  • Portable deployment across machines
  • Versioning: Docker includes git-like capabilities for tracking successive versions of a container
  • Component reuse: Docker allows building or stacking of already created packages.
  • Shared libraries: There is already a public registry (http://index.docker.io/ ) where thousands have already uploaded the useful containers they have created.
  • Docker taking the devops world by storm since its launch back in 2013.
  • LXC, while older, has not been as popular with developers as Docker has proven to be
  • LXC having a focus on sys admins that’s similar to what solutions like the Solaris operating system, with its Solaris Zones, Linux OpenVZ, and FreeBSD, with its BSD Jails virtualization system
  • it started out being built on top of LXC, Docker later moved beyond LXC containers to its own execution environment called libcontainer.
  • Unlike LXC, which launches an operating system init for each container, Docker provides one OS environment, supplied by the Docker Engine
  • LXC tooling sticks close to what system administrators running bare metal servers are used to
  • The LXC command line provides essential commands that cover routine management tasks, including the creation, launch, and deletion of LXC containers.
  • Docker containers aim to be even lighter weight in order to support the fast, highly scalable, deployment of applications with microservice architecture.
  • With backing from Canonical, LXC and LXD have an ecosystem tightly bound to the rest of the open source Linux community.
  • Docker Swarm
  • Docker Trusted Registry
  • Docker Compose
  • Docker Machine
  • Kubernetes facilitates the deployment of containers in your data center by representing a cluster of servers as a single system.
  • Swarm is Docker’s clustering, scheduling and orchestration tool for managing a cluster of Docker hosts. 
  • rkt is a security minded container engine that uses KVM for VM-based isolation and packs other enhanced security features. 
  • Apache Mesos can run different kinds of distributed jobs, including containers. 
  • Elastic Container Service is Amazon’s service for running and orchestrating containerized applications on AWS
  • LXC offers the advantages of a VE on Linux, mainly the ability to isolate your own private workloads from one another. It is a cheaper and faster solution to implement than a VM, but doing so requires a bit of extra learning and expertise.
  • Docker is a significant improvement of LXC’s capabilities.
張 旭

Full Cycle Developers at Netflix - Operate What You Build - 1 views

netflixtechblog.com/lopers-at-netflix-a08c31f83249

devops system

shared by 張 旭 on 27 May 20 - No Cached
  • Researching issues felt like bouncing a rubber ball between teams, hard to catch the root cause and harder yet to stop from bouncing between one another.
  • In the past, Edge Engineering had ops-focused teams and SRE specialists who owned the deploy+operate+support parts of the software life cycle
  • hearing about those problems second-hand
  • ...17 more annotations...
  • devs could push code themselves when needed, and also were responsible for off-hours production issues and support requests
  • What were we trying to accomplish and why weren’t we being successful?
  • These specialized roles create efficiencies within each segment while potentially creating inefficiencies across the entire life cycle.
  • Grouping differing specialists together into one team can reduce silos, but having different people do each role adds communication overhead, introduces bottlenecks, and inhibits the effectiveness of feedback loops.
  • devops principles
  • develops a system also be responsible for operating and supporting that system
  • Each development team owns deployment issues, performance bugs, capacity planning, alerting gaps, partner support, and so on.
  • Those centralized teams act as force multipliers by turning their specialized knowledge into reusable building blocks.
  • Communication and alignment are the keys to success.
  • Full cycle developers are expected to be knowledgeable and effective in all areas of the software life cycle.
  • ramping up on areas they haven’t focused on before
  • We run dev bootcamps and other forms of ongoing training to impart this knowledge and build up these skills
  • “how can I automate what is needed to operate this system?”
  • “what self-service tool will enable my partners to answer their questions without needing me to be involved?”
  • A full cycle developer thinks and acts like an SWE, SDET, and SRE. At times they create software that solves business problems, at other times they write test cases for that, and still other times they automate operational aspects of that system.
  • the need for continuous delivery pipelines, monitoring/observability, and so on.
  • Tooling and automation help to scale expertise, but no tool will solve every problem in the developer productivity and operations space
張 旭

Helm | Template Functions and Pipelines - 0 views

helm.sh/...functions_and_pipelines

helm

shared by 張 旭 on 02 Oct 21 - No Cached
  • When injecting strings from the .Values object into the template, we ought to quote these strings.
  • Helm has over 60 available functions. Some of them are defined by the Go template language itself. Most of the others are part of the Sprig template library
  • the "Helm template language" as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates.
  • ...10 more annotations...
  • Drawing on a concept from UNIX, pipelines are a tool for chaining together a series of template commands to compactly express a series of transformations.
  • the default function: default DEFAULT_VALUE GIVEN_VALUE
  • all static default values should live in the values.yaml, and should not be repeated using the default command (otherwise they would be redundant).
  • the default command is perfect for computed values, which can not be declared inside values.yaml.
  • When lookup returns an object, it will return a dictionary.
  • The synopsis of the lookup function is lookup apiVersion, kind, namespace, name -> resource or resource list
  • When no object is found, an empty value is returned. This can be used to check for the existence of an object.
  • The lookup function uses Helm's existing Kubernetes connection configuration to query Kubernetes.
  • Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run, so the lookup function will return an empty list (i.e. dict) in such a case.
  • the operators (eq, ne, lt, gt, and, or and so on) are all implemented as functions. In pipelines, operations can be grouped with parentheses ((, and )).
  • 張 旭 on 02 Oct 21
     
    "When injecting strings from the .Values object into the template, we ought to quote these strings. "
« First ‹ Previous 181 - 200 of 203 Next ›
Showing 20▼ items per page