Group items tagged

When a Workspace is declared in a job, one or more files or directories can be added. Each addition creates a new layer in the Workspace filesystem. Downstreams jobs can then use this Workspace for its own needs or add more layers on top.

Unlike caching, Workspaces are not shared between runs as they no longer exists once a Workflow is complete.

A prime example is package dependency managers such as Yarn, Bundler, or Pip.

Caches are global within a project, a cache saved on one branch will be used by others so they should only be used for data that is OK to share across Branches

Artifacts are used for longer-term storage of the outputs of your build process.

If your project needs to be packaged in some form or fashion, say an Android app where the .apk file is uploaded to Google Play, that’s a great example of an artifact.

Rails 4 automatically adds the sass-rails, coffee-rails and uglifier gems to your Gemfile

reduce the number of requests that a browser makes to render a web page

Starting with version 3.1, Rails defaults to concatenating all JavaScript files into one master .js file and all CSS files into one master .css file

In production, Rails inserts an MD5 fingerprint into each filename so that the file is cached by the web browser

The technique sprockets uses for fingerprinting is to insert a hash of the content into the name, usually at the end.

asset minification or compression

The sass-rails gem is automatically used for CSS compression if included in Gemfile and no config.assets.css_compressor option is set.

Supported languages include Sass for CSS, CoffeeScript for JavaScript, and ERB for both by default.

When a filename is unique and based on its content, HTTP headers can be set to encourage caches everywhere (whether at CDNs, at ISPs, in networking equipment, or in web browsers) to keep their own copy of the content

asset pipeline is technically no longer a core feature of Rails 4

Rails uses for fingerprinting is to insert a hash of the content into the name, usually at the end

Fingerprinting is enabled by default for production and disabled for all other environments

The files in app/assets are never served directly in production.

Paths are traversed in the order that they occur in the search path

You should use app/assets for files that must undergo some pre-processing before they are served.

By default .coffee and .scss files will not be precompiled on their own

app/assets is for assets that are owned by the application, such as custom images, JavaScript files or stylesheets.

lib/assets is for your own libraries' code that doesn't really fit into the scope of the application or those libraries which are shared across applications.

vendor/assets is for assets that are owned by outside entities, such as code for JavaScript plugins and CSS frameworks.

Any path under assets/* will be searched

By default these files will be ready to use by your application immediately using the require_tree directive.

Sprockets uses files named index (with the relevant extensions) for a special purpose

Rails.application.config.assets.paths

causes turbolinks to check if an asset has been updated and if so loads it into the page

if you add an erb extension to a CSS asset (for example, application.css.erb), then helpers like asset_path are available in your CSS rules

If you add an erb extension to a JavaScript asset, making it something such as application.js.erb, then you can use the asset_path helper in your JavaScript code

The asset pipeline automatically evaluates ERB

data URI — a method of embedding the image data directly into the CSS file — you can use the asset_data_uri helper.

Sprockets will also look through the paths specified in config.assets.paths, which includes the standard application paths and any paths added by Rails engines.

image_tag

asset_data_uri

sass-rails provides -url and -path helpers (hyphenated in Sass, underscored in Ruby) for the following asset classes: image, font, video, audio, JavaScript and stylesheet.

Rails.application.config.assets.compress

In JavaScript files, the directives begin with //=

The require_tree directive tells Sprockets to recursively include all JavaScript files in the specified directory into the output.

manifest files contain directives — instructions that tell Sprockets which files to require in order to build a single CSS or JavaScript file.

Sprockets uses manifest files to determine which assets to include and serve.

the family of require directives prevents files from being included twice in the output

which files to require in order to build a single CSS or JavaScript file

Directives are processed top to bottom, but the order in which files are included by require_tree is unspecified.

In JavaScript files, Sprockets directives begin with //=

If require_self is called more than once, only the last call is respected.

require directive is used to tell Sprockets the files you wish to require.

You need not supply the extensions explicitly. Sprockets assumes you are requiring a .js file when done from within a .js file

require_directory

The file extensions used on an asset determine what preprocessing is applied.

Additional layers of preprocessing can be requested by adding other extensions, where each extension is processed in a right-to-left manner

require_self

In development mode, assets are served as separate files in the order they are specified in the manifest file.

when these files are requested they are processed by the processors provided by the coffee-script and sass gems and then sent back to the browser as JavaScript and CSS respectively.

css.scss.erb

js.coffee.erb

Keep in mind the order of these preprocessors is important.

By default Rails assumes that assets have been precompiled and will be served as static assets by your web server

with the Asset Pipeline the :cache and :concat options aren't used anymore

Assets are compiled and cached on the first request after the server is started

RAILS_ENV=production bundle exec rake assets:precompile

Debug mode can also be enabled in Rails helper methods

If you set config.assets.initialize_on_precompile to false, be sure to test rake assets:precompile locally before deploying

By default Rails assumes assets have been precompiled and will be served as static assets by your web server.

a rake task to compile the asset manifests and other files in the pipeline

RAILS_ENV=production bin/rake assets:precompile

a recipe to handle this in deployment

config/initializers/assets.rb

The initialize_on_precompile change tells the precompile task to run without invoking Rails

The X-Sendfile header is a directive to the web server to ignore the response from the application, and instead serve a specified file from disk

the jquery-rails gem which comes with Rails as the standard JavaScript library gem.

Possible options for JavaScript compression are :closure, :uglifier and :yui

concatenate assets

Trunk-based development is a more open model since all developers have access to the main code. This enables teams to iterate quickly and implement CI/CD.

Developers can create short-lived branches with a few small commits compared to other long-lived feature branching strategies.

Gitflow is an alternative Git branching model that uses long-lived feature branches and multiple primary branches.

Gitflow also has separate primary branch lines for development, hotfixes, features, and releases.

Trunk-based development is far more simplified since it focuses on the main branch as the source of fixes and releases.

Trunk-based development eases the friction of code integration.

trunk-based development model reduces these conflicts.

Adding an automated test suite and code coverage monitoring for this stream of commits enables continuous integration.

With continuous integration, developers perform trunk-based development in conjunction with automated tests that run after each committee to a trunk.

Instead of creating a feature branch and waiting to build out the complete specification, developers can instead create a trunk commit that introduces the feature flag and pushes new trunk commits that build out the feature specification within the flag.

Short running unit and integration tests are executed during development and upon code merge.

Automated tests provide a layer of preemptive code review.

A repository with a large amount of active branches has some unfortunate side effects

Merge branches to the trunk at least once a day

The “continuous” in CI/CD implies that updates are constantly flowing.

Pod-to-Service communications

External-to-Service communications

sharing machines requires ensuring that two applications do not try to use the same ports.

Dynamic port allocation brings a lot of complications to the system

do not need to explicitly create links between Pods

almost never need to deal with mapping container ports to host ports.

pods on a node can communicate with all pods on all nodes without NAT

agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node

pods in the host network of a node can communicate with all pods on all nodes without NAT

containers within a Pod share their network namespaces - including their IP address

containers within a Pod must coordinate port usage

“IP-per-pod” model.

request ports on the Node itself which forward to your Pod (called host ports), but this is a very niche operation

The Pod itself is blind to the existence or non-existence of host ports.

AOS is an Intent-Based Networking system that creates and manages complex datacenter environments from a simple integrated platform.

Cisco Application Centric Infrastructure offers an integrated overlay and underlay SDN solution that supports containers, virtual machines, and bare metal servers.

AOS Reference Design currently supports Layer-3 connected hosts that eliminate legacy Layer-2 switching problems.

The AWS VPC CNI offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters.

users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters.

Using this CNI plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.

The CNI allocates AWS Elastic Networking Interfaces (ENIs) to each Kubernetes node and using the secondary IP range from each ENI for pods on the node.

Big Cloud Fabric is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments.

Cilium is L7/HTTP aware and can enforce network policies on L3-L7 using an identity based security model that is decoupled from network addressing.

CNI-Genie is a CNI plugin that enables Kubernetes to simultaneously have access to different implementations of the Kubernetes network model in runtime.

CNI-Genie also supports assigning multiple IP addresses to a pod, each from a different CNI plugin.

cni-ipvlan-vpc-k8s contains a set of CNI and IPAM plugins to provide a simple, host-local, low latency, high throughput, and compliant networking stack for Kubernetes within Amazon Virtual Private Cloud (VPC) environments by making use of Amazon Elastic Network Interfaces (ENI) and binding AWS-managed IPs into Pods using the Linux kernel’s IPvlan driver in L2 mode.

to be straightforward to configure and deploy within a VPC

Contiv provides configurable networking

Contrail, based on Tungsten Fabric, is a truly open, multi-cloud network virtualization and policy management platform.

DANM is a networking solution for telco workloads running in a Kubernetes cluster.

Flannel is a very simple overlay network that satisfies the Kubernetes requirements.

Any traffic bound for that subnet will be routed directly to the VM by the GCE network fabric.

sysctl net.ipv4.ip_forward=1

Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.

Knitter is a network solution which supports multiple networking in Kubernetes.

Kube-OVN is an OVN-based kubernetes network fabric for enterprises.

Kube-router provides a Linux LVS/IPVS-based service proxy, a Linux kernel forwarding-based pod-to-pod networking solution with no overlays, and iptables/ipset-based network policy enforcer.

If you have a “dumb” L2 network, such as a simple switch in a “bare-metal” environment, you should be able to do something similar to the above GCE setup.

Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.

NSX-T can provide network virtualization for a multi-cloud and multi-hypervisor environment and is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks.

NSX-T Container Plug-in (NCP) provides integration between NSX-T and container orchestrators such as Kubernetes

Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.

OpenVSwitch is a somewhat more mature but also complicated way to build an overlay network

OVN is an opensource network virtualization solution developed by the Open vSwitch community.

Project Calico is an open source container networking provider and network policy engine.

Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet

Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking.

Romana is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network

Weave Net runs as a CNI plug-in or stand-alone. In either version, it doesn’t require any configuration or extra code to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.

The network model is implemented by the container runtime on each node.

you don’t need to set up the deployment upfront. Auto DevOps still builds and tests your application. You can define the deployment later.

ship your app first, then explore the customizations later.

Consistency

Auto DevOps works with any Kubernetes cluster.

To use Auto DevOps for individual projects, you can enable it in a project-by-project basis.

Only project Maintainers can enable or disable Auto DevOps at the project level.

We strongly advise you to use GitLab Container Registry with Auto DevOps to simplify configuration and prevent any unforeseen issues.

The GitLab integration with Helm does not support installing applications when behind a proxy.

use all rules keywords, like if, changes, and exists, in the same rule. The rule evaluates to true only when all included keywords evaluate to true.

use parentheses with && and || to build more complicated variable expressions.

Use workflow to specify which types of pipelines can run.

every push to an open merge request’s source branch causes duplicated pipelines.

avoid duplicate pipelines by changing the job rules to avoid either push (branch) pipelines or merge request pipelines.

For behavior similar to the only/except keywords, you can check the value of the $CI_PIPELINE_SOURCE variable

commonly used variables for if clauses

rules:changes expressions to determine when to add jobs to a pipeline

Use !reference tags to reuse rules in different jobs.

Use except to define when a job does not run.

only or except used without refs is the same as only:refs / except/refs

If you change multiple files, but only one file ends in .md, the build job is still skipped.

If you use multiple keywords with only or except, the keywords are evaluated as a single conjoined expression.

only includes the job if all of the keys have at least one condition that matches.

except excludes the job if any of the keys have at least one condition that matches.

To specify a job as manual, add when: manual to the job in the .gitlab-ci.yml file.

Use protected environments to define a list of users authorized to run a manual job.

Use when: delayed to execute scripts after a waiting period, or if you want to avoid jobs immediately entering the pending state.

To split a large job into multiple smaller jobs that run in parallel, use the parallel keyword

run a trigger job multiple times in parallel in a single pipeline, but with different variable values for each instance of the job.

The @ symbol denotes the beginning of a ref’s repository path. To match a ref name that contains the @ character in a regular expression, you must use the hex character code match \x40.

Compare a variable to a string

Check if a variable is undefined

Check if a variable is empty

Check if a variable exists

Check if a variable is empty

Matches are found when using =~.

Matches are not found when using !~.

Join variable expressions together with && or ||

Serverless can also mean applications where some amount of server-side logic is still written by the application developer but unlike traditional architectures is run in stateless compute containers that are event-triggered, ephemeral (may only last for one invocation), and fully managed by a 3rd party.

‘Functions as a service

AWS Lambda is one of the most popular implementations of FaaS at present,

A good example is Auth0 - they started initially with BaaS ‘Authentication as a Service’, but with Auth0 Webtask they are entering the FaaS space.

a typical ecommerce app

a backend data-processing service

with zero administration.

FaaS offerings do not require coding to a specific framework or library.

Horizontal scaling is completely automatic, elastic, and managed by the provider

Functions in FaaS are triggered by event types defined by the provider.

from a deployment-unit point of view FaaS functions are stateless.

allowed the client direct access to a subset of our database

deleted the authentication logic in the original application and have replaced it with a third party BaaS service

The client is in fact well on its way to becoming a Single Page Application.

implement a FaaS function that responds to http requests via an API Gateway

port the search code from the Pet Store server to the Pet Store Search function

replaced a long lived consumer application with a FaaS function that runs within the event driven context

server applications - is a key difference when comparing with other modern architectural trends like containers and PaaS

the only code that needs to change when moving to FaaS is the ‘main method / startup’ code, in that it is deleted, and likely the specific code that is the top-level message handler (the ‘message listener interface’ implementation), but this might only be a change in method signature

With FaaS you need to write the function ahead of time to assume parallelism

Most providers also allow functions to be triggered as a response to inbound http requests, typically in some kind of API gateway

you should assume that for any given invocation of a function none of the in-process or host state that you create will be available to any subsequent invocation.

FaaS functions are either naturally stateless

certain classes of long lived task are not suited to FaaS functions without re-architecture

if you were writing a low-latency trading application you probably wouldn’t want to use FaaS systems at this time

An API Gateway is an HTTP server where routes / endpoints are defined in configuration and each route is associated with a FaaS function.

API Gateway will allow mapping from http request parameters to inputs arguments for the FaaS function

API Gateways may also perform authentication, input validation, response code mapping, etc.

the Serverless Framework makes working with API Gateway + Lambda significantly easier than using the first principles provided by AWS.

Apex - a project to ‘Build, deploy, and manage AWS Lambda functions with ease.'

'Serverless' to mean the union of a couple of other ideas - 'Backend as a Service' and 'Functions as a Service'.

each action also maps to particular CRUD operations in a database

One way to avoid deep nesting (as recommended above) is to generate the collection actions scoped under the parent, so as to get a sense of the hierarchy, but to not nest the member actions.

to only build routes with the minimal amount of information to uniquely identify the resource

The shallow method of the DSL creates a scope inside of which every nesting is shallow

These concerns can be used in resources to avoid code duplication and share behavior across routes

add a member route, just add a member block into the resource block

You can leave out the :on option, this will create the same member route except that the resource id value will be available in params[:photo_id] instead of params[:id].

Singular Resources

use a singular resource to map /profile (rather than /profile/:id) to the show action

Passing a String to get will expect a controller#action format

workaround

organize groups of controllers under a namespace

route /articles (without the prefix /admin) to Admin::ArticlesController

route /admin/articles to ArticlesController (without the Admin:: module prefix)

Nested routes allow you to capture this relationship in your routing.

Resources should never be nested more than 1 level deep.

via the :shallow option

a balance between descriptive routes and deep nesting

:shallow_path prefixes member paths with the specified parameter

Routing Concerns allows you to declare common routes that can be reused inside other resources and routes

Rails can also create paths and URLs from an array of parameters.

use url_for with a set of objects

In helpers like link_to, you can specify just the object in place of the full url_for call

insert the action name as the first element of the array

pass :on to a route, eliminating the block:

Collection Routes

simple routing makes it very easy to map legacy URLs to new Rails actions

add an alternate new action using the :on shortcut

When you set up a regular route, you supply a series of symbols that Rails maps to parts of an incoming HTTP request.

:controller maps to the name of a controller in your application

:action maps to the name of an action within that controller

This route will also route the incoming request of /photos to PhotosController#index, since :action and :id are

use a constraint on :controller that matches the namespace you require

dynamic segments don't accept dots

The params will also include any parameters from the query string

:defaults option.

set params[:format] to "jpg"

specify a name for any route using the :as option

create logout_path and logout_url as named helpers in your application.

Inside the show action of UsersController, params[:username] will contain the username for the user.

should use the get, post, put, patch and delete methods to constrain a route to a particular verb.

use the match method with the :via option to match multiple verbs at once

use the :constraints option to enforce a format for a dynamic segment

constraints

don't need to use anchors

the same name as the hash key and then compare the return value with the hash value.

constraint values should match the corresponding Request object method return type

blacklist

redirect helper

reuse dynamic segments from the match in the path to redirect

this redirection is a 301 "Moved Permanently" redirect.

root method

put the root route at the top of the file

root inside namespaces and scopes

For namespaced controllers you can use the directory notation

use the :constraints option to specify a required format on the implicit id

specify a single constraint to apply to a number of routes by using the block

non-resourceful routes

:id parameter doesn't accept dots

:as option lets you override the normal naming for the named route helpers

use the :as option to prefix the named route helpers that Rails generates for a rout

prevent name collisions

prefix routes with a named parameter

:only option

:except option

alter path names

http://localhost:3000/rails/info/routes

rake routes

setting the CONTROLLER environment variable

Routes should be included in your testing strategy

assert_generates assert_recognizes assert_routing

it is nearly the same thing as an image, except that the top layer is read-write

A running container is defined as a read-write "union view" and

The processes within this process-space may change, delete or create files within the "union view" file that will be captured in the read-write layer

there is no longer a running container

each layer contains a pointer to a parent layer using the Id

The 'docker create' command adds a read-write layer to the top stack based on the image id.  It does not run this container.

The command 'docker start' creates a process space around the union view of the container's layers.

the docker run command starts with an image, creates a container, and starts the container

'git pull' (which is a combination of 'git fetch' and 'git merge')

'docker ps' lists out the inventory of running containers on your system

'docker ps -a' where the 'a' is short for 'all' lists out all the containers on your system, whether stopped or running.

Only those images that have containers attached to them or that have been pulled are considered top-level.

'docker stop' issues a SIGTERM to a running container which politely stops all the processes in that process-space.

results is a normal, but non-running, container

'docker kill' issues a non-polite SIGKILL command to all the processes in a running container.

'docker stop' and 'docker kill' which send actual UNIX signals to a running process

'docker pause' uses a special cgroups feature to freeze/pause a running process-space

'docker rm' removes the read-write layer that defines a container from your host system

'docker rmi' removes the read-layer that defines a "union view" of an image.

'docker commit' takes a container's top-level read-write layer and burns it into a read-only layer.

turns a container (whether running or stopped) into an immutable image

uses the FROM directive in the Dockerfile file as the starting image and iteratively 1) runs (create and start) 2) modifies and 3) commits.

'docker exec' command runs on a running container and executes a process in that running container's process space

'docker inspect' fetches the metadata that has been associated with the top-layer of the container or image

'docker save' creates a single tar file that can be used to import on a different host system

only be run on an image

'docker export' command creates a tar file of the contents of the "union view" and flattens it for consumption for non-Docker usages

'docker history' command takes an image-id and recursively prints out the read-only layers

set it explicitly with the :strategy_class option

explicitly tell OmniAuth where to locate your ca_certificates file

make your model (e.g. app/models/user.rb) omniauthable

devise_for :users was already added to your config/routes.rb

user_omniauth_authorize_path(provider) user_omniauth_callback_path(provider)

devise does not create *_url methods

The symbol passed to the user_omniauth_authorize_path method matches the symbol of the provider passed to Devise's config block

After inserting their credentials, they will be redirected back to your application's callback method

tell Devise in which controller we will implement Omniauth callbacks

find_for_facebook_oauth

implement the method below in your model

All information retrieved from Facebook by OmniAuth is available as a hash at request.env["omniauth.auth"]

Devise removes all the data starting with "devise." from the session whenever a user signs in, so we get automatic session clean up

We pass the :event => :authentication to the sign_in_and_redirect method to force all authentication callbacks to be called