Group items tagged

Different programming languages are for different tasks.

Go or C which are compiled to run on bare metal.

To run NodeJS on multiple cores, you have to use something like PM2, but since this you have to keep your code stateless.

Python have very rich and sugary syntax that’s great for working with data while keeping your code small and expressive.

SQL is almost always slower than NoSQL

databases are often read-first or write-first

write-first, just like Cassandra.

store all of your data to your databases and leave nothing at backend

Functional code is stateless by default

It’s better to go for stateless right from the very beginning.

deliver exactly the same responses for same requests.

Sessions? Store them at Redis and allow all of your servers to access it.

Only the first user will trigger a data query, and all others will be receiving exactly the same data straight from the RAM

Only the server output should be cached

Varnish is a great cache option that works with HTTP responses, so it may work with any backend.

a rate limiter – if there’s not enough time have passed since last request, the ongoing request will be denied.

Just set up entry relations and allow your database to calculate external keys for you

Backend should have different responsibilities: hashing, building web pages from data and templates, managing sessions and so on.

a distributed database.

your code has to be stateless

Move anything related to the data to the database.

For load-balancing a database, go for cluster.

DB is balancing requests, as well as your backend.

Users from different continents are separated with DNS.

Keep is scalable, keep is stateless.

Auto DevOps works with any Kubernetes cluster;

using the Docker or Kubernetes executor, with privileged mode enabled.

Base domain (needed for Auto Review Apps and Auto Deploy)

Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)

Prometheus (needed for Auto Monitoring)

scrape your Kubernetes cluster.

project level as a variable: KUBE_INGRESS_BASE_DOMAIN

A wildcard DNS A record matching the base domain(s) is required

Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).

review/ (every environment starting with review/)

staging

production

need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.

Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.

Continuous deployment to production using timed incremental rollout

Automatic deployment to staging, manual deployment to production

Auto Build creates a build of the application using an existing Dockerfile or Heroku buildpacks.

If a project’s repository contains a Dockerfile, Auto Build will use docker build to create a Docker image.

Each buildpack requires certain files to be in your project’s repository for Auto Build to successfully build your application.

Auto Test automatically runs the appropriate tests for your application using Herokuish and Heroku buildpacks by analyzing your project to detect the language and framework.

Auto Code Quality uses the Code Quality image to run static analysis and other code checks on the current code.

Static Application Security Testing (SAST) uses the SAST Docker image to run static analysis on the current code and checks for potential security issues.

Dependency Scanning uses the Dependency Scanning Docker image to run analysis on the project dependencies and checks for potential security issues.

License Management uses the License Management Docker image to search the project dependencies for their license.

Vulnerability Static Analysis for containers uses Clair to run static analysis on a Docker image and checks for potential security issues.

Review Apps are temporary application environments based on the branch’s code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur.

The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain.

Your apps should not be manipulated outside of Helm (using Kubernetes directly).

Dynamic Application Security Testing (DAST) uses the popular open source tool OWASP ZAProxy to perform an analysis on the current code and checks for potential security issues.

Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.

add the paths to a file named .gitlab-urls.txt in the root directory, one per line.

After a branch or merge request is merged into the project’s default branch (usually master), Auto Deploy deploys the application to a production environment in the Kubernetes cluster, with a namespace based on the project name and unique project ID

Auto Deploy doesn’t include deployments to staging or canary by default, but the Auto DevOps template contains job definitions for these tasks if you want to enable them.

For internal and private projects a GitLab Deploy Token will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.

If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.

If present, DB_INITIALIZE will be run as a shell command within an application pod as a helm post-install hook.

a post-install hook means that if any deploy succeeds, DB_INITIALIZE will not be processed thereafter.

DB_MIGRATE will be run as a shell command within an application pod as a helm pre-upgrade hook.

Once your application is deployed, Auto Monitoring makes it possible to monitor your application’s server and response metrics right out of the box.

annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254"

While Auto DevOps provides great defaults to get you started, you can customize almost everything to fit your needs; from custom buildpacks, to Dockerfiles, Helm charts, or even copying the complete CI/CD configuration into your project to enable staging and canary deployments, and more.

Auto DevOps uses Helm to deploy your application to Kubernetes.

make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.

specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment.

Your additions will be merged with the Auto DevOps template using the behaviour described for include

copy and paste the contents of the Auto DevOps template into your project and edit this as needed.

In order to support applications that require a database, PostgreSQL is provisioned by default.

Set up the replica variables using a project variable and scale your application by just redeploying it!

You should not scale your application using Kubernetes directly.

Some applications need to define secret variables that are accessible by the deployed application.

Auto DevOps pipelines will take your application secret variables to populate a Kubernetes secret.

Environment variables are generally considered immutable in a Kubernetes pod.

If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to 1 as a CI/CD variable), then the application will be automatically deployed to a staging environment, and a production_manual job will be created for you when you’re ready to manually deploy to production.

If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to 1 as a CI/CD variable) then two manual jobs will be created: canary which will deploy the application to the canary environment production_manual which is to be used by you when you’re ready to manually deploy to production.

If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead of the standard production job, 4 different manual jobs will be created: rollout 10% rollout 25% rollout 50% rollout 100%

To start a job, click on the play icon next to the job’s name.

not all buildpacks support Auto Test yet

When a project has been marked as private, GitLab’s Container Registry requires authentication when downloading containers.

Authentication credentials will be valid while the pipeline is running, allowing for a successful initial deployment.

We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.

each action also maps to particular CRUD operations in a database

One way to avoid deep nesting (as recommended above) is to generate the collection actions scoped under the parent, so as to get a sense of the hierarchy, but to not nest the member actions.

to only build routes with the minimal amount of information to uniquely identify the resource

The shallow method of the DSL creates a scope inside of which every nesting is shallow

These concerns can be used in resources to avoid code duplication and share behavior across routes

add a member route, just add a member block into the resource block

You can leave out the :on option, this will create the same member route except that the resource id value will be available in params[:photo_id] instead of params[:id].

Singular Resources

use a singular resource to map /profile (rather than /profile/:id) to the show action

Passing a String to get will expect a controller#action format

workaround

organize groups of controllers under a namespace

route /articles (without the prefix /admin) to Admin::ArticlesController

route /admin/articles to ArticlesController (without the Admin:: module prefix)

Nested routes allow you to capture this relationship in your routing.

Resources should never be nested more than 1 level deep.

via the :shallow option

a balance between descriptive routes and deep nesting

:shallow_path prefixes member paths with the specified parameter

Routing Concerns allows you to declare common routes that can be reused inside other resources and routes

Rails can also create paths and URLs from an array of parameters.

use url_for with a set of objects

In helpers like link_to, you can specify just the object in place of the full url_for call

insert the action name as the first element of the array

pass :on to a route, eliminating the block:

Collection Routes

simple routing makes it very easy to map legacy URLs to new Rails actions

add an alternate new action using the :on shortcut

When you set up a regular route, you supply a series of symbols that Rails maps to parts of an incoming HTTP request.

:controller maps to the name of a controller in your application

:action maps to the name of an action within that controller

This route will also route the incoming request of /photos to PhotosController#index, since :action and :id are

use a constraint on :controller that matches the namespace you require

dynamic segments don't accept dots

The params will also include any parameters from the query string

:defaults option.

set params[:format] to "jpg"

specify a name for any route using the :as option

create logout_path and logout_url as named helpers in your application.

Inside the show action of UsersController, params[:username] will contain the username for the user.

should use the get, post, put, patch and delete methods to constrain a route to a particular verb.

use the match method with the :via option to match multiple verbs at once

use the :constraints option to enforce a format for a dynamic segment

constraints

don't need to use anchors

the same name as the hash key and then compare the return value with the hash value.

constraint values should match the corresponding Request object method return type

blacklist

redirect helper

reuse dynamic segments from the match in the path to redirect

this redirection is a 301 "Moved Permanently" redirect.

root method

put the root route at the top of the file

root inside namespaces and scopes

For namespaced controllers you can use the directory notation

use the :constraints option to specify a required format on the implicit id

specify a single constraint to apply to a number of routes by using the block

non-resourceful routes

:id parameter doesn't accept dots

:as option lets you override the normal naming for the named route helpers

use the :as option to prefix the named route helpers that Rails generates for a rout

prevent name collisions

prefix routes with a named parameter

:only option

:except option

alter path names

http://localhost:3000/rails/info/routes

rake routes

setting the CONTROLLER environment variable

Routes should be included in your testing strategy

assert_generates assert_recognizes assert_routing

combining existing solutions and existing blocks: KeepAlived + ProxySQl + MySQL.

Keepalived implements a set of checkers to dynamically and adaptively maintain and manage load-balanced server pool according to their health.

Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions.

A sidecar proxy is a proxy instance that’s dedicated to a specific service instance.

The container management framework keeps a list of instances that are ready to receive requests.

The service mesh can encrypt and decrypt requests and responses

a service mesh application also includes a monitoring and management layer, called the control plane.

Service mesh architectures are not ever likely to be the answer to all application development and delivery problems

Docker under the hood creates IPTable rules so containers can't reach other containers unless you'd want to

Træfik can listen to Docker events and reconfigure its own internal configuration when containers are created (or shut down).

Enable the Docker provider and listen for container events on the Docker unix socket we've mounted earlier.

Enable automatic request and configuration of SSL certificates using Let's Encrypt. These certificates will be stored in the acme.json file, which you can back-up yourself and store off-premises.

there isn't a single container that has any published ports to the host -- everything is routed through Docker networks.

Thanks to Docker labels, we can tell Træfik how to create its internal routing configuration.

container labels and service labels

With the traefik.enable label, we tell Træfik to include this container in its internal configuration.

Service labels allow managing many routes for the same container.

When both container labels and service labels are defined, container labels are just used as default values for missing service labels but no frontend/backend are going to be defined only with these labels.

Always specify the correct port where the container expects HTTP traffic using traefik.port label.

With the traefik.frontend.auth.basic label, it's possible for Træfik to provide a HTTP basic-auth challenge for the endpoints you provide the label for.

Communication between pods is more complicated, however, and requires a separate networking component that can transparently route traffic from a pod on one node to a pod on another.

pod network plugins. For this cluster, you will use Flannel, a stable and performant option.

Passing the argument --pod-network-cidr=10.244.0.0/16 specifies the private subnet that the pod IPs will be assigned from.

kubectl apply -f descriptor.[yml|json] is the syntax for telling kubectl to create the objects described in the descriptor.[yml|json] file.

deploy Nginx using Deployments and Services

NodePort, a scheme that will make the pod accessible through an arbitrary port opened on each node of the cluster

load balancing requests to multiple pods

Pods are ubiquitous in Kubernetes, so understanding them will facilitate your work

how controllers such as deployments work since they are used frequently in stateless applications for scaling and the automated healing of unhealthy applications.

Understanding the types of services and the options they have is essential for running both stateless and stateful applications.

validate the template by running packer validate example.json. This command checks the syntax as well as the configuration values to verify they look valid.

At the end of running packer build, Packer outputs the artifacts that were created as part of the build.

Packer only builds images. It does not attempt to manage them in any way.

如果查询实在是太大，服务端会拒绝接收更多数据并抛出异常

服务器响应给用户的数据通常会很多，由多个数据包组成

减小通信间数据包的大小和数量是一个非常好的习惯

查询中尽量避免使用SELECT *以及加上LIMIT限制

在解析一个查询语句前，如果查询缓存是打开的，那么MySQL会检查这个查询语句是否命中查询缓存中的数据。

两个查询在任何字符上的不同（例如：空格、注释），都会导致缓存不会命中。

MySQL将缓存存放在一个引用表

如果查询中包含任何用户自定义函数、存储函数、用户变量、临时表、mysql库中的系统表，其查询结果 都不会被缓存。

MySQL的查询缓存系统会跟踪查询中涉及的每个表，如果这些表（数据或结构）发生变化，那么和这张表相关的所有缓存数据都将失效。

在任何的写操作时，MySQL必须将对应表的所有缓存都设置为失效。

查询缓存对系统的额外消耗也不仅仅在写操作，读操作也不例外

任何的查询语句在开始之前都必须经过检查，即使这条SQL语句永远不会命中缓存

如果查询结果可以被缓存，那么执行完成后，会将结果存入缓存，也会带来额外的系统消耗

并不是什么情况下查询缓存都会提高系统性能，缓存和失效都会带来额外消耗

用多个小表代替一个大表

批量插入代替循环单条插入

合理控制缓存空间大小

可以通过SQL_CACHE和SQL_NO_CACHE来控制某个查询语句是否需要进行缓存

不要轻易打开查询缓存，特别是写密集型应用

将query_cache_type设置为DEMAND，这时只有加入SQL_CACHE的查询才会走缓存，其他查询则不会，这样可以非常自由地控制哪些查询需要被缓存。

预处理则会根据MySQL规则进一步检查解析树是否合法。比如检查要查询的数据表和数据列是否存在等等。

一条查询可以有很多种执行方式，最后都返回相应的结果。优化器的作用就是找到这其中最好的执行计划。

查询当前会话的last_query_cost的值来得到其计算当前查询的成本。

尝试预测一个查询使用某种执行计划时的成本

有非常多的原因会导致MySQL选择错误的执行计划

MySQL值选择它认为成本小的，但成本小并不意味着执行时间短

提前终止查询（比如：使用Limit时，查找到满足数量的结果集后会立即终止查询）

找某列的最小值，如果该列有索引，只需要查找B+Tree索引最左端，反之则可以找到最大值

在完成解析和优化阶段以后，MySQL会生成对应的执行计划，查询执行引擎根据执行计划给出的指令逐步执行得出结果。

查询过程中的每一张表由一个handler实例表示。

MySQL在查询优化阶段就为每一张表创建了一个handler实例，优化器可以根据这些实例的接口来获取表的相关信息

即使查询不到数据，MySQL仍然会返回这个查询的相关信息，比如改查询影响到的行数以及执行时间等等。

结果集返回客户端是一个增量且逐步返回的过程

整型就比字符操作代价低，因而会使用整型来存储ip地址，使用DATETIME来存储时间，而不是使用字符串。

如果计划在列上创建索引，就应该将该列设置为NOT NULL

UNSIGNED表示不允许负值，大致可以使正数的上限提高一倍。

没有太大的必要使用DECIMAL数据类型。即使是在需要存储财务数据时，仍然可以使用BIGINT

大表ALTER TABLE非常耗时

MySQL执行大部分修改表结果操作的方法是用新的结构创建一个张空表，从旧表中查出所有的数据插入新表，然后再删除旧表

索引是提高MySQL查询性能的一个重要途径，但过多的索引可能会导致过高的磁盘使用率以及过高的内存占用，从而影响应用程序的整体性能。

索引是指B-Tree索引，它是目前关系型数据库中查找数据最为常用和有效的索引，大多数存储引擎都支持这种索引

B+Tree中的B是指balance，意为平衡。

平衡二叉树首先需要符合二叉查找树的定义，其次必须满足任何节点的两个子树的高度差不能大于1。

索引往往以索引文件的形式存储的磁盘上

如何减少查找过程中的I/O存取次数？

减少树的深度，将二叉树变为m叉树（多路搜索树），而B+Tree就是一种多路搜索树。

页是计算机管理存储器的逻辑块，硬件及OS往往将主存和磁盘存储区分割为连续的大小相等的块

B+Tree为了保持平衡，对于新插入的值需要做大量的拆分页操作，而页的拆分需要I/O操作，为了尽可能的减少页的拆分操作，B+Tree也提供了类似于平衡二叉树的旋转功能。

在多数情况下，在多个列上建立独立的索引并不能提高查询性能。理由非常简单，MySQL不知道选择哪个索引的查询效率更好

当出现多个索引做联合操作时（多个OR条件），对结果集的合并、排序等操作需要耗费大量的CPU和内存资源，特别是当其中的某些索引的选择性不高，需要返回合并大量数据时，查询成本更高。

explain时如果发现有索引合并（Extra字段出现Using union），应该好好检查一下查询和表结构是不是已经是最优的

索引选择性是指不重复的索引值和数据表的总记录数的比值

唯一索引的选择性是1，这时最好的索引选择性，性能也是最好的。

如果一个索引包含或者说覆盖所有需要查询的字段的值，那么就没有必要再回表查询，这就称为覆盖索引。

对结果集进行排序的操作

按照索引顺序扫描得出的结果自然是有序的

如果explain的结果中type列的值为index表示使用了索引扫描来做排序。

在设计索引时，如果一个索引既能够满足排序，有满足查询，是最好的。

比如有一个索引(A,B)，再创建索引(A)就是冗余索引。

对于非常小的表，简单的全表扫描更高效。对于中到大型的表，索引就非常有效。对于超大型的表，建立和维护索引的代价随之增长，这时候其他技术也许更有效，比如分区表。

explain后再提测是一种美德。

如果要统计行数，直接使用COUNT(*)，意义清晰，且性能更好。

执行EXPLAIN并不需要真正地去执行查询，所以成本非常低。

确保ON和USING字句中的列上有索引。在创建索引的时候就要考虑到关联的顺序

确保任何的GROUP BY和ORDER BY中的表达式只涉及到一个表中的列，这样MySQL才有可能使用索引来优化

MySQL关联执行的策略非常简单，它对任何的关联都执行嵌套循环关联操作，即先在一个表中循环取出单条数据，然后在嵌套循环到下一个表中寻找匹配的行，依次下去，直到找到所有表中匹配的行为为止。然后根据各个表匹配的行，返回查询中需要的各个列。

最外层的查询是根据A.xx列来查询的，A.c上如果有索引的话，整个关联查询也不会使用。再看内层的查询，很明显B.c上如果有索引的话，能够加速查询，因此只需要在关联顺序中的第二张表的相应列上创建索引即可。

MySQL处理UNION的策略是先创建临时表，然后再把各个查询结果插入到临时表中，最后再来做查询。

要使用UNION ALL，如果没有ALL关键字，MySQL会给临时表加上DISTINCT选项，这会导致整个临时表的数据做唯一性检查，这样做的代价非常高。

尽可能不要使用存储过程

VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

A vSphere datastore is an abstraction which hides storage details (such as LUNs) and provides a uniform interface for storing persistent data.

the datastores can be of the type vSAN, VMFS, NFS & VVol.

VMFS (Virtual Machine File System) is a cluster file system that allows virtualization to scale beyond a single node for multiple VMware ESX servers.

NFS (Network File System) is a distributed file protocol to access storage over network like local storage.

vSphere Cloud Provider supports every storage primitive exposed by Kubernetes

Kubernetes PVs are defined in Pod specifications.

we like HTTPS/non-www since HTTPS is needed for current browsers and non-www is short.

visit the Mozilla TLS Generator to get the latest cipher suites and TLS versions

add the necessary headers for GeoIP and proper logging.

HTML5 SSE simpler than websockets

nginx -t

Scan your site with SSL Labs scan

When you run Docker without using swarm mode, you execute container commands.

You can run swarm services and standalone containers on the same Docker instances.

A node is an instance of the Docker engine participating in the swarm

You can run one or more nodes on a single physical computer or cloud server

To deploy your application to a swarm, you submit a service definition to a manager node.

Manager nodes also perform the orchestration and cluster management functions required to maintain the desired state of the swarm.

Worker nodes receive and execute tasks dispatched from manager nodes.

service is the definition of the tasks to execute on the worker nodes

When you create a service, you specify which container image to use and which commands to execute inside running containers.

replicated services model, the swarm manager distributes a specific number of replica tasks among the nodes based upon the scale you set in the desired state.

global services, the swarm runs one task for the service on every available node in the cluster.

A task carries a Docker container and the commands to run inside the container

Manager nodes assign tasks to worker nodes according to the number of replicas set in the service scale.

If you do not specify a port, the swarm manager assigns the service a port in the 30000-32767 range.

Swarm mode has an internal DNS component that automatically assigns each service in the swarm a DNS entry.

The generate attribute is used to inform Terragrunt to generate the Terraform code for configuring the backend.

The find_in_parent_folders() helper will automatically search up the directory tree to find the root terragrunt.hcl and inherit the remote_state configuration from it.

Unlike the backend configurations, provider configurations support variables,

if you needed to modify the configuration to expose another parameter (e.g session_name), you would have to then go through each of your modules to make this change.

instructs Terragrunt to create the file provider.tf in the working directory (where Terragrunt calls terraform) before it calls any of the Terraform commands

large modules should be considered harmful.

Large modules are slow, insecure, hard to update, hard to code review, hard to test, and brittle (i.e., you have all your eggs in one basket).

Terragrunt allows you to define your Terraform code once and to promote a versioned, immutable “artifact” of that exact same code from environment to environment.

The control plane's components make global decisions about the cluster

Control plane components can be run on any machine in the cluster.

for simplicity, set up scripts typically start all control plane components on the same machine, and do not run user containers on this machine

The API server is the front end for the Kubernetes control plane.

kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances.

Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for those data.