Group items tagged

docker build --cache-from "$CI_REGISTRY_IMAGE:latest" -t "$CI_REGISTRY_IMAGE:new-tag"

But if you maintain a CHANGELOG in this format, and/or your Git tags are also your Docker tags, you can get the previous version and use cache the this image version.

script: - export PREVIOUS_VERSION=$(perl -lne 'print "v${1}" if /^##\s\[(\d\.\d\.\d)\]\s-\s\d{4}(?:-\d{2}){2}\s*$/' CHANGELOG.md | sed -n '2 p') - docker build --cache-from "$CI_REGISTRY_IMAGE:$PREVIOUS_VERSION" -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG" -f ./prod.Dockerfile .

« Docker layer caching » is enough to optimize the build time.

We're building a Docker image, dependencies are installed inside a container.We can't cache a dependencies directory if it doesn't exists in the job workspace.

Dependencies will always be installed from a container but will be extracted by the GitLab Runner in the job workspace. Our goal is to send the cached version in the build context.

- docker cp app:/var/www/html/vendor/ ./vendor

after_script

- docker cp app:/var/www/html/node_modules/ ./node_modules

To avoid old dependencies to be mixed with the new ones, at the risk of keeping unused dependencies in cache, which would make cache and images heavier.

If you need to cache directories in testing jobs, it's easier: use volumes !

version your cache keys !

sharing Docker image between jobs

In every job, we automatically get artifacts from previous stages.

docker save $DOCKER_CI_IMAGE | gzip > app.tar.gz

I personally use the « push / pull » technique,

Nginx is built to handle many concurrent connections at the same time.

provides you with flexibility in easily adding backend servers or taking them down as needed for maintenance

Proxying in Nginx is accomplished by manipulating a request aimed at the Nginx server and passing it to other servers for the actual processing

When a request matches a location with a proxy_pass directive inside, the request is forwarded to the URL given by the directive

The request coming from Nginx on behalf of a client will look different than a request coming directly from a client

Nginx gets rid of any empty headers

Nginx, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request

The "Host" header is re-written to the value defined by the $proxy_host variable.

The upstream should not expect this connection to be persistent

Headers with empty values are completely removed from the passed request.

by default, this will be set to the value of $proxy_host, a variable that will contain the domain name or IP address and port taken directly from the proxy_pass definition

This is selected by default as it is the only address Nginx can be sure the upstream server responds to

$http_host: Sets the "Host" header to the "Host" header from the client request.

preference to: the host name from the request line itself

set the "Host" header to the $host variable. It is the most flexible and will usually provide the proxied servers with a "Host" header filled in as accurately as possible

This variable takes the value of the original X-Forwarded-For header retrieved from the client and adds the Nginx server's IP address to the end.

Once defined, this name will be available for use within proxy passes as if it were a regular domain name

By default, this is just a simple round-robin selection process (each request will be routed to a different host in turn)

Specifies that new connections should always be given to the backend that has the least number of active connections.

distributes requests to different servers based on the client's IP address.

mainly used with memcached proxying

As for the hash method, you must provide the key to hash against

Server Weight

Nginx's buffering and caching capabilities

Without buffers, data is sent from the proxied server and immediately begins to be transmitted to the client.

With buffers, the Nginx proxy will temporarily store the backend's response and then feed this data to the client

Nginx defaults to a buffering design

can be set in the http, server, or location contexts.

the sizing directives are configured per request, so increasing them beyond your need can affect your performance

When buffering is "off" only the buffer defined by the proxy_buffer_size directive will be used

A high availability (HA) setup is an infrastructure without a single point of failure, and your load balancers are a part of this configuration.

multiple load balancers (one active and one or more passive) behind a static IP address that can be remapped from one server to another.

Nginx also provides a way to cache content from backend servers

proxy_cache backcache;

The proxy_cache_bypass directive is set to the $http_cache_control variable. This will contain an indicator as to whether the client is explicitly requesting a fresh, non-cached version of the resource

For private content, you should set the Cache-Control header to "no-cache", "no-store", or "private" depending on the nature of the data

globalLock.totalTime: If this is higher than the total database uptime, the database has been in a lock state for too long.

Unlike relational databases such as MySQL or PostgreSQL, MongoDB uses JSON-like documents for storing data.

Databases operate in an environment that consists of numerous reads, writes, and updates.

When a lock occurs, no other operation can read or modify the data until the operation that initiated the lock is finished.

locks.deadlockCount: Number of times the lock acquisitions have encountered deadlocks

Is the database frequently locking from queries? This might indicate issues with the schema design, query structure, or system architecture.

For version 3.2 on, WiredTiger is the default.

MMAPv1 locks whole collections, not individual documents.

When the MMAPv1 storage engine is in use, MongoDB will use memory-mapped files to store data.

db.serverStatus().mem

mem.resident: Roughly equivalent to the amount of RAM in megabytes that the database process uses

If mem.resident exceeds the value of system memory and there’s a large amount of unmapped data on disk, we’ve most likely exceeded system capacity.

If the value of mem.mapped is greater than the amount of system memory, some operations will experience page faults.

The WiredTiger storage engine is a significant improvement over MMAPv1 in performance and concurrency.

By default, MongoDB will reserve 50 percent of the available memory for the WiredTiger data cache.

wiredTiger.cache.bytes currently in the cache – This is the size of the data currently in the cache.

wiredTiger.cache.tracked dirty bytes in the cache – This is the size of the dirty data in the cache.

we can look at the wiredTiger.cache.bytes read into cache value for read-heavy applications. If this value is consistently high, increasing the cache size may improve overall read performance.

check whether the application is read-heavy. If it is, increase the size of the replica set and distribute the read operations to secondary members of the set.

Replication is the propagation of data from one node to another

Replication sets handle this replication.

Sometimes, data isn’t replicated as quickly as we’d like.

use the db.printSlaveReplicationInfo() or the rs.printSlaveReplicationInfo() command to see the status of a replica set from the perspective of the secondary member of the set.

shows how far behind the secondary members are from the primary. This number should be as low as possible.

monitor this metric closely.

watch for any spikes in replication delay.

One replica set is primary. All others are secondary.

use the profiler to gain a deeper understanding of the database’s behavior.

Enabling the profiler can affect system performance, due to the additional activity.

When you run an image and generate a container, you add a new writable layer (the “container layer”) on top of the underlying layers.

Inadvertently including files that are not necessary for building an image results in a larger build context and larger image size.

To exclude files not relevant to the build (without restructuring your source repository) use a .dockerignore file. This file supports exclusion patterns similar to .gitignore files.

minimize image layers by leveraging build cache.

avoid installing extra or unnecessary packages just because they might be “nice to have.”

Decoupling applications into multiple containers makes it easier to scale horizontally and reuse containers

Limiting each container to one process is a good rule of thumb, but it is not a hard and fast rule.

do multi-stage builds and only copy the artifacts you need into the final image. This allows you to include tools and debug information in your intermediate build stages without increasing the size of the final image.

avoid duplication of packages and make the list much easier to update.

When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified.

simply comparing the instruction in the Dockerfile with one of the child images is sufficient.

For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file.

If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

In that case just the command string itself is used to find a match.

Whenever possible, use current official repositories as the basis for your images.

Using RUN apt-get update && apt-get install -y ensures your Dockerfile installs the latest package versions with no further coding or manual intervention.

cache busting

Docker executes these commands using the /bin/sh -c interpreter, which only evaluates the exit code of the last operation in the pipe to determine success.

set -o pipefail && to ensure that an unexpected error prevents the build from inadvertently succeeding.

The CMD instruction should be used to run the software contained by your image, along with any arguments.

CMD should rarely be used in the manner of CMD [“param”, “param”] in conjunction with ENTRYPOINT

The ENV instruction is also useful for providing required environment variables specific to services you wish to containerize,

COPY only supports the basic copying of local files into the container

the best use for ADD is local tar file auto-extraction into the image, as in ADD rootfs.tar.xz /

using ADD to fetch packages from remote URLs is strongly discouraged; you should use curl or wget instead

The best use for ENTRYPOINT is to set the image’s main command, allowing that image to be run as though it was that command (and then use CMD as the default flags).

the image name can double as a reference to the binary as shown in the command

The VOLUME instruction should be used to expose any database storage area, configuration storage, or files/folders created by your docker container.

use VOLUME for any mutable and/or user-serviceable parts of your image

If you absolutely need functionality similar to sudo, such as initializing the daemon as root but running it as non-root), consider using “gosu”.

always use absolute paths for your WORKDIR

An ONBUILD command executes after the current Dockerfile build completes.

Think of the ONBUILD command as an instruction the parent Dockerfile gives to the child Dockerfile

Be careful when putting ADD or COPY in ONBUILD. The “onbuild” image fails catastrophically if the new build’s context is missing the resource being added.

When a Workspace is declared in a job, one or more files or directories can be added. Each addition creates a new layer in the Workspace filesystem. Downstreams jobs can then use this Workspace for its own needs or add more layers on top.

Unlike caching, Workspaces are not shared between runs as they no longer exists once a Workflow is complete.

A prime example is package dependency managers such as Yarn, Bundler, or Pip.

Caches are global within a project, a cache saved on one branch will be used by others so they should only be used for data that is OK to share across Branches

Artifacts are used for longer-term storage of the outputs of your build process.

If your project needs to be packaged in some form or fashion, say an Android app where the .apk file is uploaded to Google Play, that’s a great example of an artifact.

Server side 的 Cache，藉由把 Database 的資料撈出來之後存到別的地方達成。

Server 跟瀏覽器之間的 Cache 機制

瀏覽器會檢視「現在的時間」是否有超過這個 Expires

HTTP 1.1 有一個新的 header 出現了，叫做：Cache-Control

max-age設定成31536000秒，也就是 365 天的意思

max-age會蓋過Expires。因此現在的快取儘管兩個都會放，但其實真正會用到的是max-age。

過期了不代表不能用

在 Server 傳送 Response 的時候，可以多加一個Last-Modified的 Header，表示這個檔案上一次更改是什麼時候。

用If-Modified-Since來跟 Server 指定拿取：某個時間點以後有更改的資料。

mark all jobs as interruptible as it doesn’t make sense to wait for builds and tests based on old information.

only running it when specific files have changed

To prevent the ‘vendor’ and ‘node_modules’ folder from being regenerated in every job, we can configure a build job for composer and npm assets.

To share assets between multiple stages, Gitlab has caches and artifacts. For dependencies we should use caches.

The pull-push policy is the default, but specified here for clarity.

All consecutive runs for the build step with the same ‘composer.lock’ file don’t update the cache.

composer prevents this by caching packages in a global package cache,

When a plugin cache directory is enabled, the terraform init command will still access the plugin distribution server to obtain metadata about which plugins are available, but once a suitable version has been selected it will first check to see if the selected plugin is already available in the cache directory.

When possible, Terraform will use hardlinks or symlinks to avoid storing a separate copy of a cached plugin in multiple directories.

Terraform will never itself delete a plugin from the plugin cache once it's been placed there.

Different programming languages are for different tasks.

Go or C which are compiled to run on bare metal.

To run NodeJS on multiple cores, you have to use something like PM2, but since this you have to keep your code stateless.

Python have very rich and sugary syntax that’s great for working with data while keeping your code small and expressive.

SQL is almost always slower than NoSQL

databases are often read-first or write-first

write-first, just like Cassandra.

store all of your data to your databases and leave nothing at backend

Functional code is stateless by default

It’s better to go for stateless right from the very beginning.

deliver exactly the same responses for same requests.

Sessions? Store them at Redis and allow all of your servers to access it.

Only the first user will trigger a data query, and all others will be receiving exactly the same data straight from the RAM

Only the server output should be cached

Varnish is a great cache option that works with HTTP responses, so it may work with any backend.

a rate limiter – if there’s not enough time have passed since last request, the ongoing request will be denied.

Just set up entry relations and allow your database to calculate external keys for you

Backend should have different responsibilities: hashing, building web pages from data and templates, managing sessions and so on.

a distributed database.

your code has to be stateless

Move anything related to the data to the database.

For load-balancing a database, go for cluster.

DB is balancing requests, as well as your backend.

Users from different continents are separated with DNS.

Keep is scalable, keep is stateless.

Rails 4 automatically adds the sass-rails, coffee-rails and uglifier gems to your Gemfile

reduce the number of requests that a browser makes to render a web page

Starting with version 3.1, Rails defaults to concatenating all JavaScript files into one master .js file and all CSS files into one master .css file

In production, Rails inserts an MD5 fingerprint into each filename so that the file is cached by the web browser

The technique sprockets uses for fingerprinting is to insert a hash of the content into the name, usually at the end.

asset minification or compression

The sass-rails gem is automatically used for CSS compression if included in Gemfile and no config.assets.css_compressor option is set.

Supported languages include Sass for CSS, CoffeeScript for JavaScript, and ERB for both by default.

When a filename is unique and based on its content, HTTP headers can be set to encourage caches everywhere (whether at CDNs, at ISPs, in networking equipment, or in web browsers) to keep their own copy of the content

asset pipeline is technically no longer a core feature of Rails 4

Rails uses for fingerprinting is to insert a hash of the content into the name, usually at the end

Fingerprinting is enabled by default for production and disabled for all other environments

The files in app/assets are never served directly in production.

Paths are traversed in the order that they occur in the search path

You should use app/assets for files that must undergo some pre-processing before they are served.

By default .coffee and .scss files will not be precompiled on their own

app/assets is for assets that are owned by the application, such as custom images, JavaScript files or stylesheets.

lib/assets is for your own libraries' code that doesn't really fit into the scope of the application or those libraries which are shared across applications.

vendor/assets is for assets that are owned by outside entities, such as code for JavaScript plugins and CSS frameworks.

Any path under assets/* will be searched

By default these files will be ready to use by your application immediately using the require_tree directive.

Sprockets uses files named index (with the relevant extensions) for a special purpose

Rails.application.config.assets.paths

causes turbolinks to check if an asset has been updated and if so loads it into the page

if you add an erb extension to a CSS asset (for example, application.css.erb), then helpers like asset_path are available in your CSS rules

If you add an erb extension to a JavaScript asset, making it something such as application.js.erb, then you can use the asset_path helper in your JavaScript code

The asset pipeline automatically evaluates ERB

data URI — a method of embedding the image data directly into the CSS file — you can use the asset_data_uri helper.

Sprockets will also look through the paths specified in config.assets.paths, which includes the standard application paths and any paths added by Rails engines.

image_tag

asset_data_uri

sass-rails provides -url and -path helpers (hyphenated in Sass, underscored in Ruby) for the following asset classes: image, font, video, audio, JavaScript and stylesheet.

Rails.application.config.assets.compress

In JavaScript files, the directives begin with //=

The require_tree directive tells Sprockets to recursively include all JavaScript files in the specified directory into the output.

manifest files contain directives — instructions that tell Sprockets which files to require in order to build a single CSS or JavaScript file.

Sprockets uses manifest files to determine which assets to include and serve.

the family of require directives prevents files from being included twice in the output

which files to require in order to build a single CSS or JavaScript file

Directives are processed top to bottom, but the order in which files are included by require_tree is unspecified.

In JavaScript files, Sprockets directives begin with //=

If require_self is called more than once, only the last call is respected.

require directive is used to tell Sprockets the files you wish to require.

You need not supply the extensions explicitly. Sprockets assumes you are requiring a .js file when done from within a .js file

require_directory

The file extensions used on an asset determine what preprocessing is applied.

Additional layers of preprocessing can be requested by adding other extensions, where each extension is processed in a right-to-left manner

require_self

In development mode, assets are served as separate files in the order they are specified in the manifest file.

when these files are requested they are processed by the processors provided by the coffee-script and sass gems and then sent back to the browser as JavaScript and CSS respectively.

css.scss.erb

js.coffee.erb

Keep in mind the order of these preprocessors is important.

By default Rails assumes that assets have been precompiled and will be served as static assets by your web server

with the Asset Pipeline the :cache and :concat options aren't used anymore

Assets are compiled and cached on the first request after the server is started

RAILS_ENV=production bundle exec rake assets:precompile

Debug mode can also be enabled in Rails helper methods

If you set config.assets.initialize_on_precompile to false, be sure to test rake assets:precompile locally before deploying

By default Rails assumes assets have been precompiled and will be served as static assets by your web server.

a rake task to compile the asset manifests and other files in the pipeline

RAILS_ENV=production bin/rake assets:precompile

a recipe to handle this in deployment

config/initializers/assets.rb

The initialize_on_precompile change tells the precompile task to run without invoking Rails

The X-Sendfile header is a directive to the web server to ignore the response from the application, and instead serve a specified file from disk

the jquery-rails gem which comes with Rails as the standard JavaScript library gem.

Possible options for JavaScript compression are :closure, :uglifier and :yui

concatenate assets

Associations are implemented using macro-style calls, so that you can declaratively add features to your models

A belongs_to association sets up a one-to-one connection with another model, such that each instance of the declaring model "belongs to" one instance of the other model.

belongs_to

A has_one association also sets up a one-to-one connection with another model, but with somewhat different semantics (and consequences).

belongs_to

A has_many association indicates a one-to-many connection with another model.

This association indicates that each instance of the model has zero or more instances of another model.

belongs_to

A has_many :through association is often used to set up a many-to-many connection with another model

This association indicates that the declaring model can be matched with zero or more instances of another model by proceeding through a third model.

through:

through:

The collection of join models can be managed via the API

new join models are created for newly associated objects, and if some are gone their rows are deleted.

The has_many :through association is also useful for setting up "shortcuts" through nested has_many associations

A has_one :through association sets up a one-to-one connection with another model. This association indicates that the declaring model can be matched with one instance of another model by proceeding through a third model.

A has_and_belongs_to_many association creates a direct many-to-many connection with another model, with no intervening model.

id: false

The has_one relationship says that one of something is yours

using t.references :supplier instead.

declare a many-to-many relationship is to use has_many :through. This makes the association indirectly, through a join model

set up a has_many :through relationship if you need to work with the relationship model as an independent entity

set up a has_and_belongs_to_many relationship (though you'll need to remember to create the joining table in the database).

use has_many :through if you need validations, callbacks, or extra attributes on the join model

With polymorphic associations, a model can belong to more than one other model, on a single association.

a polymorphic belongs_to declaration as setting up an interface that any other model can use.

In designing a data model, you will sometimes find a model that should have a relation to itself

add a references column to the model itself

All of the association methods are built around caching, which keeps the result of the most recent query available for further operations.

it is a bad idea to give an association a name that is already used for an instance method of ActiveRecord::Base. The association method would override the base method and break things.

belongs_to associations you need to create foreign keys

has_and_belongs_to_many associations you need to create the appropriate join table

So a join between customer and order models will give the default join table name of "customers_orders" because "c" outranks "o" in lexical ordering.

For example, one would expect the tables "paper_boxes" and "papers" to generate a join table name of "papers_paper_boxes" because of the length of the name "paper_boxes", but it in fact generates a join table name of "paper_boxes_papers" (because the underscore '' is lexicographically _less than 's' in common encodings).

id: false

pass id: false to create_table because that table does not represent a model

By default, associations look for objects only within the current module's scope.

will work fine, because both the Supplier and the Account class are defined within the same scope.

To associate a model with a model in a different namespace, you must specify the complete class name in your association declaration:

class_name

class_name

Active Record provides the :inverse_of option

Every association will attempt to automatically find the inverse association and set the :inverse_of option heuristically (based on the association name)

In database terms, this association says that this class contains the foreign key.

In all of these methods, association is replaced with the symbol passed as the first argument to belongs_to.

(force_reload = false)

The association method returns the associated object, if any. If no associated object is found, it returns nil.

the cached version will be returned.

The association= method assigns an associated object to this object.

Behind the scenes, this means extracting the primary key from the associate object and setting this object's foreign key to the same value.

The build_association method returns a new object of the associated type

The create_association method returns a new object of the associated type

raises ActiveRecord::RecordInvalid if the record is invalid.

dependent

counter_cache

:autosave :class_name :counter_cache :dependent :foreign_key :inverse_of :polymorphic :touch :validate

finding the number of belonging objects more efficient.

Although the :counter_cache option is specified on the model that includes the belongs_to declaration, the actual column must be added to the associated model.

:destroy, when the object is destroyed, destroy will be called on its associated objects.

The :inverse_of option specifies the name of the has_many or has_one association that is the inverse of this association

set the :touch option to :true, then the updated_at or updated_on timestamp on the associated object will be set to the current time whenever this object is saved or destroyed

specify a particular timestamp attribute to update

If you set the :validate option to true, then associated objects will be validated whenever you save this object

where includes readonly select

make your code somewhat more efficient

no need to use includes for immediate associations

will be read-only when retrieved via the association