Group items tagged

pulling from git is slow.

Support for containers has existed in the Linux kernel since version 2.6.24 when cgroup support was added

All of the logic that used to live in your cookbooks/playbooks/manifests/etc now lives in a Dockerfile that resides directly in the repository for the application it is designed to build

All of the dependencies of the application are bundled with the container which means no need to build on the fly on every server during deployment.

Containers bring standardization which allows for systems like centralized logging, monitoring, and metrics to easily snap into place no matter what is running in the container.

Dockerfiles do not give you the same level of control over configuration as your application transitions between environments, like dev, staging, and production.

configuration management systems now have hooks for docker integration.

Config management will only be used to install Docker, an orchestration system, configure PAM/SSH auth, and tune OS sysctl values.

allowing your schema and changes to be database independent.

each migration as being a new 'version' of the database

each migration modifies it to add or remove tables, columns, or entries

roll this migration back, it will remove the table

timestamps macro adds two columns, created_at and updated_at

On databases that support transactions with statements that change the schema, migrations are wrapped in a transaction

reversible

use up and down instead of change

Migrations are stored as files in the db/migrate directory, one for each migration class.

a UTC timestamp identifying

Rails uses this timestamp to determine which migration should be run and in what order

"AddXXXToYYY" or "RemoveXXXFromYYY"

use a Ruby DSL

column type as references

part_number:string:index

a migration to remove a column

"CreateXXX"

change_column_null

AddUserRefToProducts

:references

produce join tables if JoinTable is part of the name

CreateJoinTable

The model and scaffold generators will create migrations appropriate for adding a new model.

enclosed by curly braces and follow the field type

create_table

By default, create_table will create a primary key called id

add an index on the new column

when using MySQL, the default is ENGINE=InnoDB

create_join_table creates an HABTM (has and belongs to many) join table

create_join_table also accepts a block

change_table, used for changing existing tables

remove

rename

add_column

change_column

remove_column

change_column_default

place an SQL fragment in the :options option.

limit

precision

scale

polymorphic

default

index

add_foreign_key

use the old style of migration using up and down methods instead of the change method.

.connection.execute

remove_column is reversible if you supply the column type as the third argument

Complex migrations may require processing that Active Record doesn't know how to reverse

reversible

Using reversible will ensure that the instructions are executed in the right order too.

add_column add_foreign_key add_index add_reference add_timestamps change_column_default (must supply a :from and :to option) change_column_null create_join_table create_table disable_extension drop_join_table drop_table (must supply a block) enable_extension remove_column (must supply a type) remove_foreign_key (must supply a second table) remove_index remove_reference remove_timestamps rename_column rename_index rename_table

:column_options option

have the option :null set to false by default

By default, the name of the join table comes from the union of the first two arguments provided to create_join_table

If the column names can not be derived from the table names, you can use the :column and :primary_key options.

figure out the column name

foreign key for a specific column

foreign key by name

should use reversible or write the up and down methods instead of using the change method

If your migration is irreversible, you should raise ActiveRecord::IrreversibleMigration from your down method.

DontUseConstraintForZipcodeValidationMigration

rails db:migrate

specify a target version

all migrations up to and including 20080906120000

run the down method on all the migrations down to, but not including, 20080906120000

rails db:rollback

db:migrate:redo task is a shortcut for doing a rollback and then migrating back up again

STEP parameter

db:setup task will create the database, load the schema and initialize it with the seed data

db:reset task will drop the database and set it up again. This is functionally equivalent to rails db:drop db:setup.

run a specific migration up or down, the db:migrate:up and db:migrate:down

the RAILS_ENV environment variable

db:migrate VERBOSE=false will suppress all output.

must rollback the migration (for example with bin/rails db:rollback), edit your migration and then run rails db:migrate to run the corrected version.

editing existing migrations is not a good idea.

revert method can be helpful when writing a new migration to undo previous migrations in whole or in part

require_relative

revert

Schema Files for

Schema files are also useful if you want a quick look at what attributes an Active Record object has

annotate_models gem automatically adds and updates comments at the top of each model summarizing the schema if you desire that functionality.

database-independent

multiple databases

you can execute custom SQL statements, the schema dumper cannot reconstitute those statements from the database

db:structure:dump

set in config/application.rb by the config.active_record.schema_format setting, which may be either :sql or :ruby.

check them into source control.

db/schema.rb contains the current version number of the database

Validations such as validates :foreign_key, uniqueness: true are one way in which models can enforce data integrity

The :dependent option on associations allows models to automatically destroy child objects when the parent is destroyed.

Migrations can also be used to add or modify data

Initial

To add initial data after a database is created, Rails has a built-in 'seeds' feature that makes the process quick and easy.

db/seeds.rb

rails db:seed

adding pre-commit plugins to your project is done with the .pre-commit-config.yaml configuration file.

The pre-commit config file describes what repositories and hooks are installed.

This configuration says to download the pre-commit-hooks project and run its trailing-whitespace hook

Kubernetes is fully controlled through this API

the main job of kubectl is to carry out HTTP requests to the Kubernetes API

Kubernetes maintains an internal state of resources, and all Kubernetes operations are CRUD operations on these resources.

Kubernetes is a fully resource-centred system

Kubernetes API reference is organised as a list of resource types with their associated operations.

This is how kubectl works for all commands that interact with the Kubernetes cluster.

kubectl simply makes HTTP requests to the appropriate Kubernetes API endpoints.

it's totally possible to control Kubernetes with a tool like curl by manually issuing HTTP requests to the Kubernetes API.

Kubernetes consists of a set of independent components that run as separate processes on the nodes of a cluster.

components on the master nodes

Storage backend: stores resource definitions (usually etcd is used)

API server: provides Kubernetes API and manages storage backend

Controller manager: ensures resource statuses match specifications

Scheduler: schedules Pods to worker nodes

component on the worker nodes

Kubelet: manages execution of containers on a worker node

triggers the ReplicaSet controller, which is a sub-process of the controller manager.

the scheduler, who watches for Pod definitions that are not yet scheduled to a worker node.

creating and updating resources in the storage backend on the master node.

However, these components do not access the storage backend directly, but only through the Kubernetes API.

All other Kubernetes components and users read, watch, and manipulate the state (i.e. resources) of Kubernetes through the Kubernetes API

The storage backend stores the state (i.e. resources) of Kubernetes.

command completion is a shell feature that works by the means of a completion script.

A completion script is a shell script that defines the completion behaviour for a specific command. Sourcing a completion script enables completion for the corresponding command.

kubectl completion zsh

/etc/bash_completion.d directory (create it, if it doesn't exist)

source <(kubectl completion bash)

source <(kubectl completion zsh)

autoload -Uz compinit compinit

the API reference, which contains the full specifications of all resources.

kubectl api-resources

displays the resource names in their plural form (e.g. deployments instead of deployment). It also displays the shortname (e.g. deploy) for those resources that have one. Don't worry about these differences. All of these name variants are equivalent for kubectl.

.spec

custom columns output format comes in. It lets you freely define the columns and the data to display in them. You can choose any field of a resource to be displayed as a separate column in the output

kubectl get pods -o custom-columns='NAME:metadata.name,NODE:spec.nodeName'

kubectl explain pod.spec.

kubectl explain pod.metadata.

browse the resource specifications and try it out with any fields you like!

with kubectl explain, only a subset of the JSONPath capabilities is supported

Many fields of Kubernetes resources are lists, and this operator allows you to select items of these lists. It is often used with a wildcard as [*] to select all items of the list.

kubectl get pods -o custom-columns='NAME:metadata.name,IMAGES:spec.containers[*].image'

a Pod may contain more than one container.

The availability zones for each node are obtained through the special failure-domain.beta.kubernetes.io/zone label.

kubectl get nodes -o yaml kubectl get nodes -o json

The default kubeconfig file is ~/.kube/config

with multiple clusters, then you have connection parameters for multiple clusters configured in your kubeconfig file.

overwrite the default kubeconfig file with the --kubeconfig option for every kubectl command.

Namespace: the namespace to use when connecting to the cluster

a one-to-one mapping between clusters and contexts.

When kubectl reads a kubeconfig file, it always uses the information from the current context.

just change the current context in the kubeconfig file

kubectl also provides the --cluster, --user, --namespace, and --context options that allow you to overwrite individual elements and the current context itself, regardless of what is set in the kubeconfig file.

for switching between clusters and namespaces is kubectx.

kubectl config get-contexts

just have to download the shell scripts named kubectl-ctx and kubectl-ns to any directory in your PATH and make them executable (for example, with chmod +x)

kubectl proxy

kubectl get roles

kubectl get pod

Kubectl plugins are distributed as simple executable files with a name of the form kubectl-x. The prefix kubectl- is mandatory,

To install a plugin, you just have to copy the kubectl-x file to any directory in your PATH and make it executable (for example, with chmod +x)

krew itself is a kubectl plugin

check out the kubectl-plugins GitHub topic

The executable can be of any type, a Bash script, a compiled Go program, a Python script, it really doesn't matter. The only requirement is that it can be directly executed by the operating system.

kubectl plugins can be written in any programming or scripting language.

you can write more sophisticated plugins with real programming languages, for example, using a Kubernetes client library. If you use Go, you can also use the cli-runtime library, which exists specifically for writing kubectl plugins.

a kubeconfig file consists of a set of contexts

changing the current context means changing the cluster, if you have only a single context per cluster.

Login to the container

Baseimage-docker only advocates running multiple OS processes inside a single container.

Password and challenge-response authentication are disabled by default. Only key authentication is allowed.

A tool for running a command as another user

The Docker developers advocate the philosophy of running a single logical service per container. A logical service can consist of multiple OS processes.

All syslog messages are forwarded to "docker logs".

Baseimage-docker advocates running multiple OS processes inside a single container, and a single logical service can consist of multiple OS processes.

Baseimage-docker provides tools to encourage running processes as different users

sometimes it makes sense to run multiple services in a single container, and sometimes it doesn't.

Splitting your logical service into multiple OS processes also makes sense from a security standpoint.

using environment variables to pass parameters to containers is very much the "Docker way"

the shell script must run the daemon without letting it daemonize/fork it.

All executable scripts in /etc/my_init.d, if this directory exists. The scripts are run in lexicographic order.

variables will also be passed to all child processes

Environment variables on Unix are inherited on a per-process basis

there is no good central place for defining environment variables for all applications and services

centrally defining environment variables

One of the ideas behind Docker is that containers should be stateless, easily restartable, and behave like a black box.

a one-shot command in a new container

immediately exit after the command exits,

add additional daemons (e.g. your own app) to the image by creating runit entries.

Nginx is one such example: it removes all environment variables unless you explicitly instruct it to retain them through the env configuration option.

Mechanisms for easily running multiple processes, without violating the Docker philosophy

Ubuntu is not designed to be run inside Docker

According to the Unix process model, the init process -- PID 1 -- inherits all orphaned child processes and must reap them

Syslog-ng seems to be much more stable

cron daemon

Rotates and compresses logs

/sbin/setuser

A tool for installing apt packages that automatically cleans up after itself.

A daemon is a program which runs in the background of its system, such as a web server.

The shell script must be called run, must be executable, and is to be placed in the directory /etc/service/<NAME>. runsv will switch to the directory and invoke ./run after your container starts.

If any script exits with a non-zero exit code, the booting will fail.

If your process is started with a shell script, make sure you exec the actual process, otherwise the shell will receive the signal and not your process.

any environment variables set with docker run --env or with the ENV command in the Dockerfile, will be picked up by my_init

they will not see the environment variables that were originally passed by Docker.

my_init imports environment variables from the directory /etc/container_environment

/etc/container_environment.sh - a dump of the environment variables in Bash format.

modify the environment variables in my_init (and therefore the environment variables in all child processes that are spawned after that point in time), by altering the files in /etc/container_environment

my_init only activates changes in /etc/container_environment when running startup scripts

environment variables don't contain sensitive data, then you can also relax the permissions

Syslog messages are forwarded to the console

syslog-ng is started separately before the runit supervisor process, and shutdown after runit exits.

/sbin/my_init --skip-startup-files --quiet --

By default, no keys are installed, so nobody can login

provide a pregenerated, insecure key (PuTTY format)

RUN /usr/sbin/enable_insecure_key

docker run YOUR_IMAGE /sbin/my_init --enable-insecure-key

RUN cat /tmp/your_key.pub >> /root/.ssh/authorized_keys && rm -f /tmp/your_key.pub

The default baseimage-docker installs syslog-ng, cron and sshd services during the build process

A proper Unix system should run all kinds of important system services.

Ubuntu is not designed to be run inside Docker

When a system is started, the first process in the system is called the init process, with PID 1. The system halts when this processs halts.

Runit (written in C) is much lighter weight than supervisord (written in Python).

Docker runs fine with multiple processes in a container.

Baseimage-docker encourages you to run multiple processes through the use of runit.

If your init process is your app, then it'll probably only shut down itself, not all the other processes in the container.

a Docker container, which is a locked down environment with e.g. no direct access to many kernel resources.

Used for service supervision and management.

A custom tool for running a command as another user.

add additional daemons (e.g. your own app) to the image by creating runit entries.

write a small shell script which runs your daemon, and runit will keep it up and running for you, restarting it when it crashes, etc.

the shell script must run the daemon without letting it daemonize/fork it.

Backing services, such as the app’s database, queueing system, or cache, is one area where dev/prod parity is important

The twelve-factor developer resists the urge to use different backing services between development and production, even when adapters theoretically abstract away any differences in backing services.

declarative provisioning tools such as Chef and Puppet combined with light-weight virtual environments such as Docker and Vagrant allow developers to run local environments which closely approximate production environments.

{% ... %} for Statements

{{ ... }} for Expressions to print to the template output

use a dot (.) to access attributes of a variable

the outer double-curly braces are not part of the variable, but the print statement.

if an object has an item and attribute with the same name. Additionally, the attr() filter only looks up attributes.

Variables can be modified by filters. Filters are separated from the variable by a pipe symbol (|) and may have optional arguments in parentheses.

Multiple filters can be chained

Tests can be used to test a variable against a common expression.

add is plus the name of the test after the variable.

to find out if a variable is defined, you can do name is defined, which will then return true or false depending on whether name is defined in the current template context.

strip whitespace in templates by hand. If you add a minus sign (-) to the start or end of a block (e.g. a For tag), a comment, or a variable expression, the whitespaces before or after that block will be removed

not add whitespace between the tag and the minus sign

mark a block raw

Template inheritance allows you to build a base “skeleton” template that contains all the common elements of your site and defines blocks that child templates can override.

The {% extends %} tag is the key here. It tells the template engine that this template “extends” another template.

access templates in subdirectories with a slash

can’t define multiple {% block %} tags with the same name in the same template

put the name of the block after the end tag for better readability

if the block is replaced by a child template, a variable would appear that was not defined in the block or passed to the context.

setting the block to “scoped” by adding the scoped modifier to a block declaration

Jinja2 functions (macros, super, self.BLOCKNAME) always return template data that is marked as safe.

With the default syntax, control structures appear inside {% ... %} blocks.

the dictsort filter

loop.cycle

use loops recursively

whether the value changed at all,

use it to test if a variable is defined, not empty and not false

Macros are comparable with functions in regular programming languages.

If a macro name starts with an underscore, it’s not exported and can’t be imported.

pass a macro to another macro

caller()

a single trailing newline is stripped if present

other whitespace (spaces, tabs, newlines etc.) is returned unchanged

caller(user)

call(user)

This is a simple dialog rendered by using a macro and a call block.

Filter sections allow you to apply regular Jinja2 filters on a block of template data.

Assignments at top level (outside of blocks, macros or loops) are exported from the template like top level macros and can be imported by other templates.

using namespace objects which allow propagating of changes across scopes

use block assignments to capture the contents of a block into a variable name.

The extends tag can be used to extend one template from another.

Blocks are used for inheritance and act as both placeholders and replacements at the same time.

The include statement is useful to include a template and return the rendered contents of that file into the current namespace

Included templates have access to the variables of the active context by default.

putting often used code into macros

imports are cached and imported templates don’t have access to the current template variables, just the globals by default.

Macros and variables starting with one or more underscores are private and cannot be imported.

By default, included templates are passed the current context and imported templates are not.

Integers and floating point numbers are created by just writing the number down

Everything between two brackets is a list.

Tuples are like lists that cannot be modified (“immutable”).

A dict in Python is a structure that combines keys and values.

// Divide two numbers and return the truncated integer result

(expr) group an expression.

The is and in operators support negation using an infix notation

in Perform a sequence / mapping containment test.

| Applies a filter.

~ Converts all operands into strings and concatenates them.

use inline if expressions.

always an attribute is returned and items are not looked up.

default(value, default_value=u'', boolean=False)¶ If the value is undefined it will return the passed default value, otherwise the value of the variable

dictsort(value, case_sensitive=False, by='key', reverse=False)¶ Sort a dict and yield (key, value) pairs.

format(value, *args, **kwargs)¶ Apply python string formatting on an object

indent(s, width=4, first=False, blank=False, indentfirst=None)¶ Return a copy of the string with each line indented by 4 spaces. The first line and blank lines are not indented by default.

join(value, d=u'', attribute=None)¶ Return a string which is the concatenation of the strings in the sequence.

map()¶ Applies a filter on a sequence of objects or looks up an attribute.

pprint(value, verbose=False)¶ Pretty print a variable. Useful for debugging.

reject()¶ Filters a sequence of objects by applying a test to each object, and rejecting the objects with the test succeeding.

replace(s, old, new, count=None)¶ Return a copy of the value with all occurrences of a substring replaced with a new one.

round(value, precision=0, method='common')¶ Round the number to a given precision