Skip to main content

Home/ Larvata/ Group items tagged system

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
張 旭

Best practices for building Kubernetes Operators and stateful apps | Google Cloud Blog - 0 views

cloud.google.com/...es-operators-and-stateful-apps

kubernetes system operator

shared by 張 旭 on 17 May 21 - No Cached
  • use the StatefulSet workload controller to maintain identity for each of the pods, and to use Persistent Volumes to persist data so it can survive a service restart.
  • a way to extend Kubernetes functionality with application specific logic using custom resources and custom controllers.
  • An Operator can automate various features of an application, but it should be specific to a single application
  • ...12 more annotations...
  • Kubebuilder is a comprehensive development kit for building and publishing Kubernetes APIs and Controllers using CRDs
  • Design declarative APIs for operators, not imperative APIs. This aligns well with Kubernetes APIs that are declarative in nature.
  • With declarative APIs, users only need to express their desired cluster state, while letting the operator perform all necessary steps to achieve it.
  • scaling, backup, restore, and monitoring. An operator should be made up of multiple controllers that specifically handle each of the those features.
  • the operator can have a main controller to spawn and manage application instances, a backup controller to handle backup operations, and a restore controller to handle restore operations.
  • each controller should correspond to a specific CRD so that the domain of each controller's responsibility is clear.
  • If you keep a log for every container, you will likely end up with unmanageable amount of logs.
  • integrate application-specific details to the log messages such as adding a prefix for the application name.
  • you may have to use external logging tools such as Google Stackdriver, Elasticsearch, Fluentd, or Kibana to perform the aggregations.
  • adding labels to metrics to facilitate aggregation and analysis by monitoring systems.
  • a more viable option is for application pods to expose a metrics HTTP endpoint for monitoring tools to scrape.
  • A good way to achieve this is to use open-source application-specific exporters for exposing Prometheus-style metrics.
張 旭

The Backup Cycle - Full Backups - 0 views

www.percona.com/...full_backup.html

database mysql backup percona xtradb

shared by 張 旭 on 02 Dec 19 - No Cached
  • xtrabackup will not overwrite existing files, it will fail with operating system error 17, file exists.
  • Log copying thread checks the transactional log every second to see if there were any new log records written that need to be copied, but there is a chance that the log copying thread might not be able to keep up with the amount of writes that go to the transactional logs, and will hit an error when the log records are overwritten before they could be read.
  • It is safe to cancel at any time, because xtrabackup does not modify the database.
  • ...15 more annotations...
  • need to prepare it in order to restore it.
  • Data files are not point-in-time consistent until they are prepared, because they were copied at different times as the program ran, and they might have been changed while this was happening.
  • You can run the prepare operation on any machine; it does not need to be on the originating server or the server to which you intend to restore.
  • you simply run xtrabackup with the --prepare option and tell it which directory to prepare,
  • All following prepares will not change the already prepared data files
  • It is not recommended to interrupt xtrabackup process while preparing backup
  • Backup validity is not guaranteed if prepare process was interrupted.
  • If you intend the backup to be the basis for further incremental backups, you should use the --apply-log-only option when preparing the backup, or you will not be able to apply incremental backups to it.
  • Backup needs to be prepared before it can be restored.
  • xtrabackup --copy-back --target-dir=/data/backups/
  • The datadir must be empty before restoring the backup.
  • MySQL server needs to be shut down before restore is performed.
  • You cannot restore to a datadir of a running mysqld instance (except when importing a partial backup).
  • rsync -avrP /data/backup/ /var/lib/mysql/
  • chown -R mysql:mysql /var/lib/mysql
張 旭

Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation - 0 views

docs.docker.com/...rootless

docker kubernetes system linux

shared by 張 旭 on 04 Aug 20 - No Cached
  • running the Docker daemon and containers as a non-root user
  • Rootless mode does not require root privileges even during the installation of the Docker daemon
  • Rootless mode executes the Docker daemon and containers inside a user namespace.
  • ...9 more annotations...
  • in rootless mode, both the daemon and the container are running without root privileges.
  • Rootless mode does not use binaries with SETUID bits or file capabilities, except newuidmap and newgidmap, which are needed to allow multiple UIDs/GIDs to be used in the user namespace.
  • expose privileged ports (< 1024)
  • add net.ipv4.ip_unprivileged_port_start=0 to /etc/sysctl.conf (or /etc/sysctl.d) and run sudo sysctl --system
  • dockerd-rootless.sh uses slirp4netns (if installed) or VPNKit as the network stack by default.
  • These network stacks run in userspace and might have performance overhead
  • This error occurs when the number of available entries in /etc/subuid or /etc/subgid is not sufficient.
  • This error occurs mostly when the host is running in cgroup v2. See the section Fedora 31 or later for information on switching the host to use cgroup v1.
  • --net=host doesn’t listen ports on the host network namespace This is an expected behavior, as the daemon is namespaced inside RootlessKit’s network namespace. Use docker run -p instead.
crazylion lee

GitHub - fonoster/fonos: - 0 views

github.com/fonos

voip

shared by crazylion lee on 29 Oct 20 - No Cached
  • crazylion lee on 29 Oct 20
     
    "Project Fonos is open-source telecommunications for the cloud. It helps VoIP integrators quickly deploy new networks and benefit from value-added services such as Programmable Voice, Messaging, and Video. This repository assembles the various components needed to deploy a telephony system at scale."
張 旭

phusion/passenger-docker: Docker base images for Ruby, Python, Node.js and Meteor web apps - 0 views

github.com/...passenger-docker

docker ruby rails

shared by 張 旭 on 14 Jan 22 - No Cached
  • Ubuntu 20.04 LTS as base system
  • 2.7.5 is configured as the default.
  • Python 3.8
  • ...23 more annotations...
  • A build system, git, and development headers for many popular libraries, so that the most popular Ruby, Python and Node.js native extensions can be compiled without problems.
  • Nginx 1.18. Disabled by default
  • production-grade features, such as process monitoring, administration and status inspection.
  • Redis 5.0. Not installed by default.
  • The image has an app user with UID 9999 and home directory /home/app. Your application is supposed to run as this user.
  • running applications without root privileges is good security practice.
  • Your application should be placed inside /home/app.
  • COPY --chown=app:app
  • Passenger works like a mod_ruby, mod_nodejs, etc. It changes Nginx into an application server and runs your app from Nginx.
  • placing a .conf file in the directory /etc/nginx/sites-enabled
  • The best way to configure Nginx is by adding .conf files to /etc/nginx/main.d and /etc/nginx/conf.d
  • files in conf.d are included in the Nginx configuration's http context.
  • any environment variables you set with docker run -e, Docker linking and /etc/container_environment, won't reach Nginx.
  • To preserve these variables, place an Nginx config file ending with *.conf in the directory /etc/nginx/main.d, in which you tell Nginx to preserve these variables.
  • By default, Phusion Passenger sets all of the following environment variables to the value production
  • Setting these environment variables yourself (e.g. using docker run -e RAILS_ENV=...) will not have any effect, because Phusion Passenger overrides all of these environment variables.
  • PASSENGER_APP_ENV environment variable
  • passenger-docker autogenerates an Nginx configuration file (/etc/nginx/conf.d/00_app_env.conf) during container boot.
  • The configuration file is in /etc/redis/redis.conf. Modify it as you see fit, but make sure daemonize no is set.
  • You can add additional daemons to the image by creating runit entries.
  • The shell script must be called run, must be executable
  • the shell script must run the daemon without letting it daemonize/fork it.
  • We use RVM to install and to manage Ruby interpreters.
snow9816

CentOS 出現 connect: Network is unreachable 解決方法 | IT達人 - 0 views

www.itmaster.tw/?p=10837

network linux system

shared by snow9816 on 01 Sep 16 - No Cached
  • /etc/sysconfig/network-scripts/ifcfg-eth0
  • service network restart
張 旭

Larry Cai - Travis CI会替代Jenkins吗？ - 0 views

www.larrycaiyu.com/...travis-ci-is-evolution.html

travis jenkins system CI test

shared by 張 旭 on 06 Sep 16 - No Cached
  • Jenkins能够让通过主从模式（master/slave）多台机器一起构建。
  • 一切都可以在Web界面中运行。
  • 当然你可有使用虚拟机的技术vagrant/virtualbox，参见使用vagrant+jenkins来管理虚拟机的技巧。可以工作，不太优雅。因为它不是原生的，有点复杂。
    • 張 旭
      張 旭 on 06 Sep 16
       
      現在應該有 Docker Container 跑測試的整合了。
  • ...5 more annotations...
  • 在CI服务器创建任务（记住：这些配置文件不是有版本控制的）
  • 构建的配置文件直接就和源码放在一起，而且配置文件使用DSL写的，可读性更高。
  • 要求在两个Ruby环境中运行，它就帮我做到了，我并不关心它是怎么切换的
  • Travis CI使用的Ruby语言，一开始考虑的就是分布式构建
  • 它的虚拟机部分只是Vagrant/Virtualbox，但是这一块是很容易迁移到其他的技术的。
    • 張 旭
      張 旭 on 06 Sep 16
       
      Docker! Docker! Docker!
張 旭

Choosing an HTTP Status Code - Stop Making It Hard - Racksburg - 0 views

racksburg.com/choosing-an-http-status-code

http status code system server backend

shared by 張 旭 on 08 Sep 16 - No Cached
  • crazylion lee on 08 Sep 16
     
    絕佳的解說
張 旭

Kubernetes 架构浅析 - 0 views

jolestar.com/kubernetes-architecture

docker kubernetes google system development

shared by 張 旭 on 08 Aug 16 - No Cached
  • 将Loadbalancer改造成Smart Loadbalancer，通过服务发现机制，应用实例启动或者销毁时自动注册到一个配置中心（etcd/zookeeper），Loadbalancer监听应用配置的变化自动修改自己的配置。
  • Mysql计划该成域名访问方式，而不是ip。为了避免dns变更时的延迟问题，需要在内网架设私有dns。
  • 配合服务发现机制自动修改dns
  • ...23 more annotations...
  • 通过增加一层代理的机制实现
  • 操作系统和基础库的依赖允许应用自定义
  • 对磁盘路径以及端口的依赖通过Docker运行参数动态注入
  • Docker的自定义变量以及参数，需要提供标准化的配置文件
  • 每个服务器节点上要有个agent来执行具体的操作，监控该节点上的应用
  • 还要提供接口以及工具去操作。
  • 应用进程和资源（包括 cpu，内存，磁盘，网络）的解耦
  • 服务依赖关系的解耦
  • scheduler在Kubernetes中是一个plugin，可以用其他的实现替换（比如mesos）
  • 大多数接口都是直接读写etcd中的数据。
  • etcd 作为配置中心和存储服务
  • kubelet 主要包含容器管理，镜像管理，Volume管理等。同时kubelet也是一个rest服务，和pod相关的命令操作都是通过调用接口实现的。
  • kube-proxy 主要用于实现Kubernetes的service机制。提供一部分SDN功能以及集群内部的智能LoadBalancer。
  • Pods Kubernetes将应用的具体实例抽象为pod。每个pod首先会启动一个google_containers/pause docker容器，然后再启动应用真正的docker容器。这样做的目的是为了可以将多个docker容器封装到一个pod中，共享网络地址。
  • Replication Controller 控制pod的副本数量
  • Services service是对一组pods的抽象，通过kube-proxy的智能LoadBalancer机制，pods的销毁迁移不会影响services的功能以及上层的调用方。
  • Namespace Kubernetes中的namespace主要用来避免pod，service的名称冲突。同一个namespace内的pod，service的名称必须是唯一的。
  • Kubernetes的理念里，pod之间是可以直接通讯的
  • 需要用户自己选择解决方案: Flannel,OpenVSwitch,Weave 等。
  • Hypernetes就是一个实现了多租户的Kubernetes版本。
  • 如果运维系统跟不上，服务拆太细，很容易出现某个服务器的角落里部署着一个很古老的不常更新的服务，后来大家竟然忘记了，最后服务器迁移的时候给丢了，用户投诉才发现。
  • 在Kubernetes上的微服务治理框架可以一揽子解决微服务的rpc，监控，容灾问题
  • 同一个pod的多个容器定义中没有优先级，启动顺序不能保证
crazylion lee

Open Whisper Systems >> Blog >> Advanced cryptographic ratcheting - 0 views

whispersystems.org/...advanced-ratcheting

chat protocol security p2p

shared by crazylion lee on 10 Jul 16 - No Cached
  • crazylion lee on 10 Jul 16
     
    "At Open WhisperSystems, we've been working on improving our encrypted asynchronous chat protocol for TextSecure. The TextSecure protocol was originally a derivative of OTR, with minor changes to accommodate it for transports with constraints like SMS or Push. Some of the recent changes we've made include simplifying and improving OTR's deniability, as well as creating a key exchange mechanism for asynchronous transports. Our most recent change incorporates what we believe to be substantial improvements to OTR's forward secrecy "ratchet.""
張 旭

How to Deploy Software - 0 views

zachholman.com/...deploying-software

development devops deploy system

shared by 張 旭 on 03 Apr 16 - No Cached
  • crazylion lee on 03 Apr 16
     
    "Make your team's deploys as boring as hell and stop stressing about it."
crazylion lee

Riemann - A network monitoring system - 0 views

riemann.io/index.html

monitoring dashboard sysadmin

shared by crazylion lee on 17 Apr 16 - No Cached
  • crazylion lee on 17 Apr 16
     
    "Riemann aggregates events from your servers and applications with a powerful stream processing language. Send an email for every exception in your app. Track the latency distribution of your web app. See the top processes on any host, by memory and CPU. Combine statistics from every Riak node in your cluster and forward to Graphite. Track user activity from second to second."
張 旭

从达标到卓越 -- API 设计之道 | Taobao FED | 淘宝前端团队 - 0 views

taobaofed.org/...index.html

api system development programming taobao

shared by 張 旭 on 21 Feb 17 - No Cached
  • 高级语言和自然语言（英语）其实相差无几，因此正确地使用（英语的）词法和语法是程序员最基本的素养。
  • 只要能够足够接近人类的日常语言和思维，并且不需要引发额外的大脑思考，那就是易用
  • 词法和语法
  • ...53 more annotations...
  • 语义
  • 版本控制
  • 正确地拼写一个单词是底线
  • 认真地注意 IDE 的 typo 提示（单词拼写错误提示）
  • state 是整个 Component 状态机中的某一个特定状态，既然描述为了状态机，那么状态和状态之间是互相切换的关系。所以对于初始状态，用 initial 来修饰。
  • props 是指 Element 的属性，要么是不存在某个属性值后来为它赋值，要么是存在属性的默认值后来将其覆盖。所以这种行为，default 是合理的修饰词。
  • 成对出现的词应该是：show & hide、open & close。
  • 成对出现的正反义词不可混用
  • 在复数的风格上保持一致，要么所有都是 -s，要么所有都是 -list。
  • 涉及到诸如字典（Dictionary）、表（Map）的时候，不要使用复数！
  • 「map」本身已经包含了这层意思，不需要再用复数去修饰它
  • 最好遵从惯例，使用名词组合 success 和 failure
  • 方法命名用动词
  • 属性命名用名词
  • 布尔值类型用形容词
  • 首字母缩写词的所有字母均大写。（如果某个语言环境有明确的业界惯例，则遵循惯例。）
  • 如果什么都没定，也没业界惯例，那么把单词写全了总是不会错的。
  • React 采用了 componentDidMount 这种过去时风格，而没有使用 componentMounted，从而跟 componentWillMount 形成对照组，方便记忆。
  • 尽量避免使用被动语态。因为被动语态看起来会比较绕，不够直观，因此我们要将被动语态的 API 转换为主动语态。
  • 无论是友好的参数设置，还是让人甜蜜蜜的语法糖，都体现了程序员的人文关怀。
  • 在文件（file）层面同样如此，一个文件只编写一个类，保证文件的职责单一（当然这对很多语言来说是天然的规则）。
  • 将混杂在一个大坨函数中的两件独立事情拆分出去，保证函数（function）级别的职责单一。
  • 现实中的 OOP 编程场景难免触及副作用。
  • 函数本身的运行稳定可预期
  • 函数的运行不对外部环境造成意料外的污染
  • SPM 是阿里通用的埋点统计方案
  • 对外部造成污染一般是两种途径：一是在函数体内部直接修改外部作用域的变量，甚至全局变量；二是通过修改实参间接影响到外部环境，如果实参是引用类型的数据结构。
  • 控制读写权限
  • 优化参数顺序。相关性越高的参数越要前置
  • 可省略的参数后置，以及 为可省略的参数设定缺省值
  • 将可省参数后置同样是最佳实践。
  • 重载（overload）
  • 如果入口参数无法进行有效区分，不要选择重载
  • 根本不明白某个 Boolean 标记位是用来干嘛的，这大大降低了用户的开发体验，以及代码可读性。
    • 張 旭
      張 旭 on 21 Feb 17
       
      我在 ADATA 的 message saver 犯了一樣的錯誤。
  • 同时支持单个和批量的处理，可以降低用户的认知负担。
  • 让 setter 型 API 始终返回 this。这是 jQuery 为我们带来的经典启示 —— 通过返回 this，来产生一种「链式调用（chaining）」的风格
  • 对异步操作都返回一个 Promise
  • 对于一些创造出来的、业务特色的词汇，如果不能用英语简明地翻译，就直接用拼音
  • 一致性可以最大程度降低信息熵
  • 打 log 要么都用中文，要么都用英文。
  • 所有的 setter 操作必须返回 this
  • 「大版本号」即「语义化版本命名」<major>.<minor>.<patch> 中的第一位 <major> 位
  • 接口的扩展方式有很多，比如：继承（extend）、组合（mixin）、装饰（decorate）
  • 在逻辑上确实存在派生关系，并且需要沿用基类行为同时自定义行为的，采用重量级的继承
  • 仅仅是扩充一些行为功能，但是逻辑上压根不存在父子关系的，使用组合
  • 装饰手法更多应用于给定一个接口，将其包装成多种适用于不同场景新接口的情况
  • $.fn.customMethod = function() {};
  • 合理的做法是新增一个 subType 字段
  • 抽象级别一般来说越高越好，将 API 设计成业务无关的，更通用，而且方便扩展
  • 利用多态性（Polymorphism）构建 Consistent APIs。
  • 作为 API 的开发者，一定要提供足够场景适用的 API，来引导我们的用户，不要让他们做出一些出人意料的「妙用」之举
  • 写代码，就像写作，而设计 API 好比列提纲。
  • Think about future, design with flexibility, but only implement for production.
張 旭

certificate - What is a Pem file and how does it differ from other OpenSSL Generated Ke... - 0 views

serverfault.com/...her-openssl-generated-key-file

https security server apache nginx system

shared by 張 旭 on 22 Feb 17 - No Cached
  • Certificate Signing Request.
  • Some applications can generate these for submission to certificate-authorities
  • These get signed by the CA and a certificate is returned
  • ...15 more annotations...
  • The returned certificate is the public certificate (which includes the public key but not the private key), which itself can be in a couple of formats.
  • this is a container format that may include just the public certificate (such as with Apache installs, and CA certificate files /etc/ssl/certs), or may include an entire certificate chain including public key, private key, and root certificates
  • Privacy Enhanced Mail (PEM), a failed method for secure email but the container format it used lives on
  • This is a PEM formatted file containing just the private-key of a specific certificate and is merely a conventional name and not a standardized one.
  • The rights on these files are very important
  • /etc/ssl/private
  • OpenSSL can convert these to .pem
  • .cert .cer .crt A .pem (or rarely .der) formatted file with a different extension
  • there are four different ways to present certificates and their components
  • used preferentially by open-source software
  • It can have a variety of extensions (.pem, .key, .cer, .cert, more)
  • The parent format of PEM
  • a binary version of the base64-encoded PEM file.
  • PEM on it's own isn't a certificate, it's just a way of encoding data
  • X.509 certificates are one type of data that is commonly encoded using PEM.
張 旭

Rails Database Best Practices - 0 views

blog.carbonfive.com/...rails-database-best-practices

rails database schema system

shared by 張 旭 on 01 Mar 17 - No Cached
  • Databases are extremely feature rich and are really freakin fast when used properly
  • create succinct helpers for accessing subsets of data that are relevant in specific situations
  • Relations are chainable
  • ...24 more annotations...
  • Return an ActiveRecord::Relation
  • Filtering in Ruby is slower
  • Please don't do this
  • trigger the query and therefore, we lose our Relation
  • leaving trivial ordering out of scopes all together.
  • where
  • where
  • .merge() makes it easy to use scopes from other models that have been joined into the query, reducing potential duplication.
  • ActiveRecord provides an easy API for doing many things with our database, but it also makes it pretty easy to do things inefficiently. The layer of abstraction hides what’s really happening.
  • first pure SQL, then ActiveRecord
  • Databases can only do fast lookups for columns with indexes, otherwise it’s doing a sequential scan
  • Add an index on every id column as well as any column that is used in a where clause.
  • use a Query class to encapsulate the potentially gnarly query.
  • subqueries
  • this Query returns an ActiveRecord::Relation
  • where
  • where
  • Single Responsibility Principle
  • Avoid ad-hoc queries outside of Scopes and Query Objects
  • encapsulate data access into scopes and Query objects
  • An ad-hoc query embedded in a controller (or view, task, etc) is harder to test in isolation and cannot be reused
  • to scopes and Query objects
    • 張 旭
      張 旭 on 01 Mar 17
       
      將查詢方式都封裝成 scope 或 query 物件。
  • Every databases provides more datatypes than your ORM might have you believe
  • Both Postgres and MySQL have full-text search capabilities
張 旭

mqtt - 0 views

mosquitto.org/mqtt-7.html

mqtt server queue message system

shared by 張 旭 on 15 Feb 17 - No Cached
  • MQTT is a lightweight publish/subscribe messaging protocol. It is useful for use with low power sensors
  • The MQTT protocol is based on the principle of publishing messages and subscribing to topics, or "pub/sub".
  • Multiple clients connect to a broker and subscribe to topics that they are interested in
  • ...22 more annotations...
  • Many clients may subscribe to the same topics
  • The broker and MQTT act as a simple, common interface for everything to connect to
  • Messages in MQTT are published on topics
  • no need to configure a topic, publishing on it is enough
  • Topics are treated as a hierarchy, using a slash (/) as a separator.
  • Clients can receive messages by creating subscriptions
  • A subscription may be to an explicit topic
  • Two wildcards are available, + or #.
  • # can be used as a wildcard for all remaining levels of hierarchy
  • + can be used as a wildcard for a single level of hierarchy
  • Zero length topic levels are valid, which can lead to some slightly non-obvious behaviour.
  • The QoS defines how hard the broker/client will try to ensure that a message is received.
  • Messages may be sent at any QoS level, and clients may attempt to subscribe to topics at any QoS level
  • the client chooses the maximum QoS it will receive
  • if a client is subscribed with QoS 2 and a message is published on QoS 0, the client will receive it on QoS 0.
  • 1: The broker/client will deliver the message at least once, with confirmation required.
  • All messages may be set to be retained.
  • the broker will keep the message even after sending it to all current subscribers
  • useful as a "last known good" mechanism
  • If clean session is set to false, then the connection is treated as durable
  • when the client disconnects, any subscriptions it has will remain and any subsequent QoS 1 or 2 messages will be stored until it connects again in the future
  • If clean session is true, then all subscriptions will be removed for the client when it disconnects
« First ‹ Previous 101 - 120 of 257 Next › Last »
Showing 20▼ items per page