Group items tagged

In full backups, two types of operations are performed to make the database consistent: committed transactions are replayed from the log file against the data files, and uncommitted transactions are rolled back.

You should use the --apply-log-only option to prevent the rollback phase.

An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s LSN.

Incremental backups do not actually compare the data files to the previous backup’s data files.

Incremental backups simply read the pages and compare their LSN to the last backup’s LSN.

The xtrabackup binary writes a file called xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn, which is the database’s LSN at the end of the backup.

from_lsn is the starting LSN of the backup and for incremental it has to be the same as to_lsn (if it is the last checkpoint) of the previous/base backup.

If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups will be useless.

If you restore it and start MySQL, InnoDB will detect that the rollback phase was not performed, and it will do that in the background, as it usually does for a crash recovery upon start.

xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc1

The final data is in /data/backups/base, not in the incremental directory.

xtrabackup --prepare --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc2

Even if the --apply-log-only was used on the last step, backup would still be consistent but in that case server would perform the rollback phase.

Auto DevOps works with any Kubernetes cluster;

using the Docker or Kubernetes executor, with privileged mode enabled.

Base domain (needed for Auto Review Apps and Auto Deploy)

Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)

Prometheus (needed for Auto Monitoring)

scrape your Kubernetes cluster.

project level as a variable: KUBE_INGRESS_BASE_DOMAIN

A wildcard DNS A record matching the base domain(s) is required

Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).

review/ (every environment starting with review/)

staging

production

need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.

Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.

Continuous deployment to production using timed incremental rollout

Automatic deployment to staging, manual deployment to production

Auto Build creates a build of the application using an existing Dockerfile or Heroku buildpacks.

If a project’s repository contains a Dockerfile, Auto Build will use docker build to create a Docker image.

Each buildpack requires certain files to be in your project’s repository for Auto Build to successfully build your application.

Auto Test automatically runs the appropriate tests for your application using Herokuish and Heroku buildpacks by analyzing your project to detect the language and framework.

Auto Code Quality uses the Code Quality image to run static analysis and other code checks on the current code.

Static Application Security Testing (SAST) uses the SAST Docker image to run static analysis on the current code and checks for potential security issues.

Dependency Scanning uses the Dependency Scanning Docker image to run analysis on the project dependencies and checks for potential security issues.

License Management uses the License Management Docker image to search the project dependencies for their license.

Vulnerability Static Analysis for containers uses Clair to run static analysis on a Docker image and checks for potential security issues.

Review Apps are temporary application environments based on the branch’s code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur.

The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain.

Your apps should not be manipulated outside of Helm (using Kubernetes directly).

Dynamic Application Security Testing (DAST) uses the popular open source tool OWASP ZAProxy to perform an analysis on the current code and checks for potential security issues.

Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.

add the paths to a file named .gitlab-urls.txt in the root directory, one per line.

After a branch or merge request is merged into the project’s default branch (usually master), Auto Deploy deploys the application to a production environment in the Kubernetes cluster, with a namespace based on the project name and unique project ID

Auto Deploy doesn’t include deployments to staging or canary by default, but the Auto DevOps template contains job definitions for these tasks if you want to enable them.

For internal and private projects a GitLab Deploy Token will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.

If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.

If present, DB_INITIALIZE will be run as a shell command within an application pod as a helm post-install hook.

a post-install hook means that if any deploy succeeds, DB_INITIALIZE will not be processed thereafter.

DB_MIGRATE will be run as a shell command within an application pod as a helm pre-upgrade hook.

Once your application is deployed, Auto Monitoring makes it possible to monitor your application’s server and response metrics right out of the box.

annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254"

While Auto DevOps provides great defaults to get you started, you can customize almost everything to fit your needs; from custom buildpacks, to Dockerfiles, Helm charts, or even copying the complete CI/CD configuration into your project to enable staging and canary deployments, and more.

Auto DevOps uses Helm to deploy your application to Kubernetes.

make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.

specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment.

Your additions will be merged with the Auto DevOps template using the behaviour described for include

copy and paste the contents of the Auto DevOps template into your project and edit this as needed.

In order to support applications that require a database, PostgreSQL is provisioned by default.

Set up the replica variables using a project variable and scale your application by just redeploying it!

You should not scale your application using Kubernetes directly.

Some applications need to define secret variables that are accessible by the deployed application.

Auto DevOps pipelines will take your application secret variables to populate a Kubernetes secret.

Environment variables are generally considered immutable in a Kubernetes pod.

If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to 1 as a CI/CD variable), then the application will be automatically deployed to a staging environment, and a production_manual job will be created for you when you’re ready to manually deploy to production.

If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to 1 as a CI/CD variable) then two manual jobs will be created: canary which will deploy the application to the canary environment production_manual which is to be used by you when you’re ready to manually deploy to production.

If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead of the standard production job, 4 different manual jobs will be created: rollout 10% rollout 25% rollout 50% rollout 100%

To start a job, click on the play icon next to the job’s name.

not all buildpacks support Auto Test yet

When a project has been marked as private, GitLab’s Container Registry requires authentication when downloading containers.

Authentication credentials will be valid while the pipeline is running, allowing for a successful initial deployment.

We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.

can overwrite the default label by passing it to the input method

configure the html of any of them

disable labels, hints or error

add a hint, an error, or even a placeholder

add an inline label

pass any html attribute straight to the input, by using the :input_html option

use the :defaults option in simple_form_fo

Simple Form generates a wrapper div around your label and input by default, you can pass any html attribute to that wrapper as well using the :wrapper_html option,

By default all inputs are required

the required property of any input can be overwritten

lets you overwrite the default input type it creates

can also render boolean attributes using as: :select to show a dropdown.

give the :disabled option to Simple Form, and it'll automatically mark the wrapper as disabled with a CSS class

Any extra option passed to these methods will be rendered as html option.

use label, hint, input_field, error and full_error helpers

to strip away all the div wrappers around the <input> field

is to use f.input_field

changing boolean_style from default value :nested to :inline

overriding the :collection option

Collections can be arrays or ranges, and when a :collection is given the :select input will be rendered by default

label_method

value_method

Both of these options also accept lambda/procs

define a to_label method on your model as Simple Form will search for and use :to_label as a :label_method first if it is found

create grouped collection selects, that will use the html optgroup tags

Grouped collection inputs accept the same :label_method and :value_method options

group_method

group_label_method

configured with a default value to be used on the site through the SimpleForm.country_priority and SimpleForm.time_zone_priority helpers.

association

association

render a :select input for choosing the :company, and another :select input with :multiple option for the :roles

all options available to :select, :radio_buttons and :check_boxes are also available to association

f.input

f.select

create a <button> element

simple_fields_for

Creates a collection of radio inputs with labels associated

Creates a collection of checkboxes with labels associated

collection_radio_buttons

collection_check_boxes

associations in your model

Role.all

the html element you will get for each attribute according to its database definition

redefine existing Simple Form inputs by creating a new class with the same name

Simple Form uses all power of I18n API to lookup labels, hints, prompts and placeholders

specify defaults for all models under the 'defaults' key

Simple Form will always look for a default attribute translation under the "defaults" key if no specific is found inside the model key

Simple Form will fallback to default human_attribute_name from Rails when no other translation is found for labels.

Simple Form will only do the lookup for options if you give a collection composed of symbols only.

"Add %{model}"

translations for labels, hints and placeholders for a namespaced model, e.g. Admin::User, should be placed under admin_user, not under admin/user

configure how your components will be rendered using the wrappers API

optional

unless_blank

By default, Simple Form will generate input field types and attributes that are supported in HTML5

The HTML5 extensions include the new field types such as email, number, search, url, tel, and the new attributes such as required, autofocus, maxlength, min, max, step.

If you want to have all other HTML 5 features, such as the new field types, you can disable only the browser validation

add novalidate to a specific form by setting the option on the form itself

the Simple Form configuration file

passing the html5 option

as: :date, html5: true

use all rules keywords, like if, changes, and exists, in the same rule. The rule evaluates to true only when all included keywords evaluate to true.

use parentheses with && and || to build more complicated variable expressions.

Use workflow to specify which types of pipelines can run.

every push to an open merge request’s source branch causes duplicated pipelines.

avoid duplicate pipelines by changing the job rules to avoid either push (branch) pipelines or merge request pipelines.

For behavior similar to the only/except keywords, you can check the value of the $CI_PIPELINE_SOURCE variable

commonly used variables for if clauses

rules:changes expressions to determine when to add jobs to a pipeline

Use !reference tags to reuse rules in different jobs.

Use except to define when a job does not run.

only or except used without refs is the same as only:refs / except/refs

If you change multiple files, but only one file ends in .md, the build job is still skipped.

If you use multiple keywords with only or except, the keywords are evaluated as a single conjoined expression.

only includes the job if all of the keys have at least one condition that matches.

except excludes the job if any of the keys have at least one condition that matches.

To specify a job as manual, add when: manual to the job in the .gitlab-ci.yml file.

Use protected environments to define a list of users authorized to run a manual job.

Use when: delayed to execute scripts after a waiting period, or if you want to avoid jobs immediately entering the pending state.

To split a large job into multiple smaller jobs that run in parallel, use the parallel keyword

run a trigger job multiple times in parallel in a single pipeline, but with different variable values for each instance of the job.

The @ symbol denotes the beginning of a ref’s repository path. To match a ref name that contains the @ character in a regular expression, you must use the hex character code match \x40.

Compare a variable to a string

Check if a variable is undefined

Check if a variable is empty

Check if a variable exists

Check if a variable is empty

Matches are found when using =~.

Matches are not found when using !~.

Join variable expressions together with && or ||

凡是使用 JSX 的地方，都要加上 type="text/babel"

Browser.js 的作用是将 JSX 语法转为 JavaScript 语法，这一步很消耗时间，实际上线的时候，应该将它放到服务器完成。

HTML 语言直接写在 JavaScript 语言之中，不加任何引号，这就是 JSX 的语法，它允许 HTML 与 JavaScript 的混写

JSX 的基本语法规则：遇到 HTML 标签（以 < 开头），就用 HTML 规则解析；遇到代码块（以 { 开头），就用 JavaScript 规则解析。

JSX 允许直接在模板插入 JavaScript 变量。如果这个变量是一个数组，则会展开这个数组的所有成员

所有组件类都必须有自己的 render 方法，用于输出组件

组件类的第一个字母必须大写，否则会报错

组件类只能包含一个顶层标签

添加组件属性，有一个地方需要注意，就是 class 属性需要写成 className ，for 属性需要写成 htmlFor ，这是因为 class 和 for 是 JavaScript 的保留字。

Percona XtraBackup works by remembering the log sequence number (LSN) when it starts, and then copying away the data files.

Percona XtraBackup runs a background process that watches the transaction log files, and copies changes from it.

Percona XtraBackup uses Backup locks where available as a lightweight alternative to FLUSH TABLES WITH READ LOCK.

Locking is only done for MyISAM and other non-InnoDB tables after Percona XtraBackup finishes backing up all InnoDB/XtraDB data and logs.

When backup locks are supported by the server, xtrabackup first copies InnoDB data, runs the LOCK TABLES FOR BACKUP and then copies the MyISAM tables.

the STDERR of xtrabackup is not written in any file. You will have to redirect it to a file, e.g., xtrabackup OPTIONS 2> backupout.log

During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files, using the copied transaction log file. After this is done, the database is ready to restore and use.

the tools enable you to do operations such as streaming and incremental backups with various combinations of copying the data files, copying the log files, and applying the logs to the data.

To restore a backup with xtrabackup you can use the --copy-back or --move-back options.

DI means that you can declare components very freely and then from any other component, just ask for an instance of it and it will be granted

do test-driven development iteratively in AngularJS!

only do DOM manipulation in a directive

with ngClass we can dynamically update the class;

ngBind allows two-way data binding;

ngShow and ngHide programmatically show or hide an element;

The less DOM manipulation, the easier directives are to test, the easier they are to style, the easier they are to change in the future, and the more re-usable and distributable they are.

still wrong.

Before doing DOM manipulation anywhere in your application, ask yourself if you really need to.

a few things wrong with this

jQuery was never necessary

use angular.element and our component will still work when dropped into a project that doesn't have jQuery.

just use angular.element

the element that is passed to the link function would already be a jQuery element!

directives aren't just collections of jQuery-like functions

Directives are actually extensions of HTML

If HTML doesn't do something you need it to do, you write a directive to do it for you, and then use it just as if it was part of HTML.

think how the team would accomplish it to fit right in with ngClick, ngClass, et al.

Don't even use jQuery. Don't even include it.

ry to think about how to do it within the confines the AngularJS.

In jQuery, selectors are used to find DOM elements and then bind/register event handlers to them.

Views are (declarative) HTML that contain AngularJS directives

Directives set up the event handlers behind the scenes for us and give us dynamic databinding.

Views are tied to models (via scopes). Views are a projection of the model

In AngularJS, think about models, rather than jQuery-selected DOM elements that hold your data.

AngularJS uses controllers and directives (each of which can have their own controller, and/or compile and linking functions) to remove behavior from the view/structure (HTML). Angular also has services and filters to help separate/organize your application.

Think about your models

Think about how you want to present your models -- your views.

Attach a controller to each view (using ng-view and routing, or ng-controller)

You can do a lot with jQuery without knowing about how JavaScript prototypal inheritance works.

jQuery is a library

AngularJS is a beautiful client-side framework

view templates are written in a language called ERB (Embedded Ruby) which is converted by the request cycle in Rails before being sent to the user.

Each action's purpose is to collect information to provide it to a view.

A view's purpose is to display this information in a human readable format.

routing file which holds entries in a special DSL (domain-specific language) that tells Rails how to connect incoming requests to controllers and actions.

You can create, read, update and destroy items for a resource and these operations are referred to as CRUD operations

A controller is simply a class that is defined to inherit from ApplicationController.

If not found, then it will attempt to load a template called application/new. It looks for one here because the PostsController inherits from ApplicationController

:formats specifies the format of template to be served in response. The default format is :html, and so Rails is looking for an HTML template.

:handlers, is telling us what template handlers could be used to render our template.

When you call form_for, you pass it an identifying object for this form. In this case, it's the symbol :post. This tells the form_for helper what this form is for.

that the action attribute for the form is pointing at /posts/new

When a form is submitted, the fields of the form are sent to Rails as parameters.

parameters can then be referenced inside the controller actions, typically to perform a particular task

params method is the object which represents the parameters (or fields) coming in from the form.

Active Record is smart enough to automatically map column names to model attributes,

Rails uses rake commands to run migrations, and it's possible to undo a migration after it's been applied to your database

every Rails model can be initialized with its respective attributes, which are automatically mapped to the respective database columns.

migration creates a method named change which will be called when you run this migration.

The action defined in this method is also reversible, which means Rails knows how to reverse the change made by this migration, in case you want to reverse it later

Migration filenames include a timestamp to ensure that they're processed in the order that they were created.

@post.save returns a boolean indicating whether the model was saved or not.

prevents an attacker from setting the model's attributes by manipulating the hash passed to the model.

If you want to link to an action in the same controller, you don't need to specify the :controller option, as Rails will use the current controller by default.

inherits from ActiveRecord::Base

Rails includes methods to help you validate the data that you send to models

Rails can validate a variety of conditions in a model, including the presence or uniqueness of columns, their format, and the existence of associated objects.

redirect_to will tell the browser to issue another request.

rendering is done within the same request as the form submission

Each request for a comment has to keep track of the post to which the comment is attached, thus the initial call to the find method of the Post model to get the post in question.

pluralize is a rails helper that takes a number and a string as its arguments. If the number is greater than one, the string will be automatically pluralized.

The render method is used so that the @post object is passed back to the new template when it is rendered.

it accepts a hash containing the attributes that you want to update.

field_with_errors. You can define a css rule to make them standout

belongs_to :post, which sets up an Active Record association

creates comments as a nested resource within posts

call destroy on Active Record objects when you want to delete them from the database.

Rails allows you to use the dependent option of an association to achieve this.

store all external data as UTF-8

you're better off ensuring that all external data is UTF-8

use UTF-8 as the internal storage of your database

Rails defaults to converting data from your database into UTF-8 at the boundary.

:patch

By default forms built with the form_for helper are sent via POST

The :method and :'data-confirm' options are used as HTML5 attributes so that when the link is clicked, Rails will first show a confirm dialog to the user, and then submit the link with method delete. This is done via the JavaScript file jquery_ujs which is automatically included into your application's layout (app/views/layouts/application.html.erb) when you generated the application.

Without this file, the confirmation dialog box wouldn't appear.

just defines the partial template we want to render

As the render method iterates over the @post.comments collection, it assigns each comment to

use the authentication system

require and permit

the method is often made private to make sure it can't be called outside its intended context.

need to prepare it in order to restore it.

You can run the prepare operation on any machine; it does not need to be on the originating server or the server to which you intend to restore.

you simply run xtrabackup with the --prepare option and tell it which directory to prepare,

All following prepares will not change the already prepared data files

Backup validity is not guaranteed if prepare process was interrupted.

If you intend the backup to be the basis for further incremental backups, you should use the --apply-log-only option when preparing the backup, or you will not be able to apply incremental backups to it.

xtrabackup --copy-back --target-dir=/data/backups/

You cannot restore to a datadir of a running mysqld instance (except when importing a partial backup).

rsync -avrP /data/backup/ /var/lib/mysql/

chown -R mysql:mysql /var/lib/mysql

运维又会分成基础运维和应用运维，开发则会分成基础核心开发和业务开发。

基础运维和开发的同学更多的只是关注资源的利用率和性能，而应用运维和业务开发则更多关注的是应用和服务上的东西。

试想一下城市交通的优化，当城市规模到一定程度的时候，整体的性能你是无法通过优化几条路或是几条街区来完成的，你需要对整个城市做整体的功能体的规划才可能达到整体效率的提升

当系统越来越复杂的时候，用户把他们的  PHP，Python, .NET，或 Node.js 的架构完全都迁移到 Java + Go 的架构上来的案例不断的发生。

更为工业化的技术

使用更为成熟更为工业化的技术栈，而不是自己熟悉的技术栈

不要自己发明轮子，更不要魔改

完全没有必要。不重新发明轮子，不魔改，不是因为自己技术不能，而是因为，这个世界早已不是自己干所有事的年代了

好些公司的架构都被技术负责人个人的喜好、擅长和个人经验给绑架了，完全不是从一个客观的角度来进行技术选型

全中国所有的电商平台，几百家银行，三大电信运营商，所有的保险公司，劵商的系统，医院里的系统，电子政府系统，等等，基本都是用 Java 开发的，包括 AWS 的主流语言也是 Java

NoSQL 的数据库在 Join 上都表现的太差

永远使用完备支持 ACID 的关系型数据库

性能上的事，总是有解的，手段也是最多的，这个比起架构的完备性和扩展性来说真的不必太过担心。

很多公司的系统既没有服从业界标准，也没有形成自己公司的标准，感觉就像一群乌合之众一样。

Restful API 的规范。我觉得是非常重要的，这里给两个我觉得写得最好的参考：Paypal 和 Microsoft 。

监控系统宁可自己死了也不能干扰实际应用。

一个公司至少一年要有一次软件版本升级的review，然后形成软件版本的统一和一致

架构和软件不是写好就完的，是需要不断修改不断维护的，80%的软件成本都是在维护上。

通过服务发现或服务网关来降低服务依赖所带来的运维复杂度

一定要使用各种软件设计的原则。比如：像SOLID这样的原则（参看《一些软件设计的原则》），IoC/DIP，SOA 或 Spring Cloud 等 架构的最佳实践（参看《SteveY对Amazon和Google平台的吐槽》中的 Service Interface 的那几条军规），分布式系统架构的相关实践（参看：《分布式系统的事务处理》，或微软件的 《Cloud Design Patterns》）……等等

没有自动化测试，没有好的软件文档，没有质量好的代码，没有标准和规范

以前欠下的技术债，都得要还，没打好的地基要重新打，没建配套设施都要建。这些基础设施如果不按照正确科学的方式建立的话，你是不可能有一个好的的系统

与其花大力气迁就技术债务，不如直接还技术债

建设没有技术债的“新城区”，并通过“防腐层 ”的架构模型，不要让技术债侵入“新城区”。

如果有一天你在做技术决定的时候，开始凭自己以往的经验，那么你就已经不可能再成长了。

做任何决定之前，最好花上一点时间，上网查一下相关的资料，技术博客，文章，论文等 ，同时，也看看各个公司，或是各个开源软件他们是怎么做的？然后，比较多种方案的 Pros/Cons，最终形成自己的决定

我很喜欢追问为什么 ，这种追问，会让客户也跟着来一起重新思考。

激进并不是瞎搞，也不是见新技术就上，而是积极拥抱会改变未来的新技术

不是不喜欢的就不学了，我对区块链和 Rust 我一样学习，我也知道这些技术的优势，但我不会大规模使用它们。

进步永远来自于探索，探索是要付出代价的，但是收益更大。

不敢冒险才是最大的冒险，不敢犯错才是最大的错误，害怕失去会让你失去的更多

Within your Terraform configuration, you may include the name of the current workspace using the ${terraform.workspace} interpolation sequence.

changing behavior based on the workspace.

Named workspaces allow conveniently switching between multiple instances of a single configuration within its single backend.

A common use for multiple workspaces is to create a parallel, distinct copy of a set of infrastructure in order to test a set of changes before modifying the main production infrastructure.

Non-default workspaces are often related to feature branches in version control.

In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production) or different internal teams.

use one or more re-usable modules to represent the common elements, and then represent each instance as a separate configuration that instantiates those common elements in the context of a different backend.

If a Terraform state for one configuration is stored in a remote backend that is accessible to other configurations then terraform_remote_state can be used to directly consume its root module outputs from those other configurations.

For server addresses, use a provider-specific resource to create a DNS record with a predictable name and then either use that name directly or use the dns provider to retrieve the published addresses in other configurations.