Group items tagged

Twelve-factor strongly favors languages which provide a REPL shell out of the box, and which make it easy to run one-off scripts.

Different programming languages are for different tasks.

Go or C which are compiled to run on bare metal.

To run NodeJS on multiple cores, you have to use something like PM2, but since this you have to keep your code stateless.

Python have very rich and sugary syntax that’s great for working with data while keeping your code small and expressive.

SQL is almost always slower than NoSQL

databases are often read-first or write-first

write-first, just like Cassandra.

store all of your data to your databases and leave nothing at backend

Functional code is stateless by default

It’s better to go for stateless right from the very beginning.

deliver exactly the same responses for same requests.

Sessions? Store them at Redis and allow all of your servers to access it.

Only the first user will trigger a data query, and all others will be receiving exactly the same data straight from the RAM

Only the server output should be cached

Varnish is a great cache option that works with HTTP responses, so it may work with any backend.

a rate limiter – if there’s not enough time have passed since last request, the ongoing request will be denied.

Just set up entry relations and allow your database to calculate external keys for you

Backend should have different responsibilities: hashing, building web pages from data and templates, managing sessions and so on.

a distributed database.

your code has to be stateless

Move anything related to the data to the database.

For load-balancing a database, go for cluster.

DB is balancing requests, as well as your backend.

Users from different continents are separated with DNS.

Keep is scalable, keep is stateless.

{% ... %} for Statements

{{ ... }} for Expressions to print to the template output

use a dot (.) to access attributes of a variable

the outer double-curly braces are not part of the variable, but the print statement.

if an object has an item and attribute with the same name. Additionally, the attr() filter only looks up attributes.

Variables can be modified by filters. Filters are separated from the variable by a pipe symbol (|) and may have optional arguments in parentheses.

Multiple filters can be chained

Tests can be used to test a variable against a common expression.

add is plus the name of the test after the variable.

to find out if a variable is defined, you can do name is defined, which will then return true or false depending on whether name is defined in the current template context.

strip whitespace in templates by hand. If you add a minus sign (-) to the start or end of a block (e.g. a For tag), a comment, or a variable expression, the whitespaces before or after that block will be removed

not add whitespace between the tag and the minus sign

mark a block raw

Template inheritance allows you to build a base “skeleton” template that contains all the common elements of your site and defines blocks that child templates can override.

The {% extends %} tag is the key here. It tells the template engine that this template “extends” another template.

access templates in subdirectories with a slash

can’t define multiple {% block %} tags with the same name in the same template

put the name of the block after the end tag for better readability

if the block is replaced by a child template, a variable would appear that was not defined in the block or passed to the context.

setting the block to “scoped” by adding the scoped modifier to a block declaration

Jinja2 functions (macros, super, self.BLOCKNAME) always return template data that is marked as safe.

With the default syntax, control structures appear inside {% ... %} blocks.

the dictsort filter

loop.cycle

use loops recursively

whether the value changed at all,

use it to test if a variable is defined, not empty and not false

Macros are comparable with functions in regular programming languages.

If a macro name starts with an underscore, it’s not exported and can’t be imported.

pass a macro to another macro

caller()

a single trailing newline is stripped if present

other whitespace (spaces, tabs, newlines etc.) is returned unchanged

caller(user)

call(user)

This is a simple dialog rendered by using a macro and a call block.

Filter sections allow you to apply regular Jinja2 filters on a block of template data.

Assignments at top level (outside of blocks, macros or loops) are exported from the template like top level macros and can be imported by other templates.

using namespace objects which allow propagating of changes across scopes

use block assignments to capture the contents of a block into a variable name.

The extends tag can be used to extend one template from another.

Blocks are used for inheritance and act as both placeholders and replacements at the same time.

The include statement is useful to include a template and return the rendered contents of that file into the current namespace

Included templates have access to the variables of the active context by default.

putting often used code into macros

imports are cached and imported templates don’t have access to the current template variables, just the globals by default.

Macros and variables starting with one or more underscores are private and cannot be imported.

By default, included templates are passed the current context and imported templates are not.

Integers and floating point numbers are created by just writing the number down

Everything between two brackets is a list.

Tuples are like lists that cannot be modified (“immutable”).

A dict in Python is a structure that combines keys and values.

// Divide two numbers and return the truncated integer result

(expr) group an expression.

The is and in operators support negation using an infix notation

in Perform a sequence / mapping containment test.

| Applies a filter.

~ Converts all operands into strings and concatenates them.

use inline if expressions.

always an attribute is returned and items are not looked up.

default(value, default_value=u'', boolean=False)¶ If the value is undefined it will return the passed default value, otherwise the value of the variable

dictsort(value, case_sensitive=False, by='key', reverse=False)¶ Sort a dict and yield (key, value) pairs.

format(value, *args, **kwargs)¶ Apply python string formatting on an object

indent(s, width=4, first=False, blank=False, indentfirst=None)¶ Return a copy of the string with each line indented by 4 spaces. The first line and blank lines are not indented by default.

join(value, d=u'', attribute=None)¶ Return a string which is the concatenation of the strings in the sequence.

map()¶ Applies a filter on a sequence of objects or looks up an attribute.

pprint(value, verbose=False)¶ Pretty print a variable. Useful for debugging.

reject()¶ Filters a sequence of objects by applying a test to each object, and rejecting the objects with the test succeeding.

replace(s, old, new, count=None)¶ Return a copy of the value with all occurrences of a substring replaced with a new one.

round(value, precision=0, method='common')¶ Round the number to a given precision

even if rounded to 0 precision, a float is returned.

select()¶ Filters a sequence of objects by applying a test to each object, and only selecting the objects with the test succeeding.

sort(value, reverse=False, case_sensitive=False, attribute=None)¶ Sort an iterable. Per default it sorts ascending, if you pass it true as first argument it will reverse the sorting.

striptags(value)¶ Strip SGML/XML tags and replace adjacent whitespace by one space.

tojson(value, indent=None)¶ Dumps a structure to JSON so that it’s safe to use in <script> tags.

trim(value)¶ Strip leading and trailing whitespace.

unique(value, case_sensitive=False, attribute=None)¶ Returns a list of unique items from the the given iterable

urlize(value, trim_url_limit=None, nofollow=False, target=None, rel=None)¶ Converts URLs in plain text into clickable links.

defined(value)¶ Return true if the variable is defined

in(value, seq)¶ Check if value is in seq.

mapping(value)¶ Return true if the object is a mapping (dict etc.).

number(value)¶ Return true if the variable is a number.

sameas(value, other)¶ Check if an object points to the same memory address than another object

undefined(value)¶ Like defined() but the other way round.

A joiner is passed a string and will return that string every time it’s called, except the first time (in which case it returns an empty string).

namespace(...)¶ Creates a new container that allows attribute assignment using the {% set %} tag

The with statement makes it possible to create a new inner scope. Variables set within this scope are not visible outside of the scope.

activate and deactivate the autoescaping from within the templates

With both trim_blocks and lstrip_blocks enabled, you can put block tags on their own lines, and the entire block line will be removed when rendered, preserving the whitespace of the contents

The full and explicit dependency specification is applied uniformly to both production and development.

Bundler for Ruby offers the Gemfile manifest format for dependency declaration and bundle exec for dependency isolation.

Pip is used for declaration and Virtualenv for isolation.

requiring only the language runtime and dependency manager installed as prerequisites.

set up everything needed to run the app’s code with a deterministic build command.

If the app needs to shell out to a system tool, that tool should be vendored into the app.

do not rely on the implicit existence of any system tools

Zabbix by default uses "pull" model when a server connects to agents on each monitoring machine, agents periodically gather the info and send it to a server.

Prometheus prefers "pull" model when a server gather info from client machines.

expose metrics for Prometheus (similar to "agents" for Zabbix)

Zabbix uses its own tcp-based communication protocol between agents and a server.

Prometheus offers solution for alerting that is separated from its core into Alertmanager application.

In most cases, the eligible DOM elements are identified by HTML5 data attributes.

using JavaScript to progressively enhance the user experience for capable browsers without negatively impacting clients that do not support or do not enable JavaScript.

jquery-ujs attaches a handler to links with the data-method attribute

jquery-ujs attaches a handler to links or forms with the data-confirm attribute that displays a JavaScript confirmation dialog

Users double click links and buttons all the time.

Links and buttons that have a data-disable-with attribute get a click handler that disables the element and updates the text of the button to that which was provided in the data attribute and disables the button.

Thanks to jquery-ujs and Rails’ respond_with, setting remote: true is likely the quickest way to get your Rails application making AJAX requests.

Cross-Site Request Forgery (CSRF) is an attack wherein the attacker tricks the user into submitting a request to an application the user is likely already authenticated to.

a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).

Each Container in the Pod must independently specify where to mount each volume

localnfs

cephfs

awsElasticBlockStore

glusterfs

vsphereVolume

An awsElasticBlockStore volume mounts an Amazon Web Services (AWS) EBS Volume into your Pod.

an EBS volume can be pre-populated with data, and that data can be “handed off” between Pods.

create an EBS volume using aws ec2 create-volume

the nodes on which Pods are running must be AWS EC2 instances

EBS only supports a single EC2 instance mounting a volume

check that the size and EBS volume type are suitable for your use!

A cephfs volume allows an existing CephFS volume to be mounted into your Pod.

the contents of a cephfs volume are preserved and the volume is merely unmounted.

have your own Ceph server running with the share exported

The configMap resource provides a way to inject configuration data into Pods

When referencing a configMap object, you can simply provide its name in the volume to reference it

volumeMounts: - name: config-vol mountPath: /etc/config volumes: - name: config-vol configMap: name: log-config items: - key: log_level path: log_level

create a ConfigMap before you can use it.

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node.

By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment.

you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem)

volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {}

An fc volume allows an existing fibre channel volume to be mounted in a Pod.

configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

Flocker is an open-source clustered Container data volume manager. It provides management and orchestration of data volumes backed by a variety of storage backends.

emptyDir

flocker

A flocker volume allows a Flocker dataset to be mounted into a Pod

have your own Flocker installation running

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your Pod.

Using a PD on a Pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1

A glusterfs volume allows a Glusterfs (an open source networked filesystem) volume to be mounted into your Pod.

have your own GlusterFS installation running

a powerful escape hatch for some applications

access to Docker internals; use a hostPath of /var/lib/docker

allowing a Pod to specify whether a given hostPath should exist prior to the Pod running, whether it should be created, and what it should exist as

specify a type for a hostPath volume

the files or directories created on the underlying hosts are only writable by root.

hostPath: # directory location on host path: /data # this field is optional type: Directory

An iscsi volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.

have your own iSCSI server running

A feature of iSCSI is that it can be mounted as read-only by multiple consumers simultaneously.

A local volume represents a mounted local storage device such as a disk, partition or directory.

Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.

PersistentVolume spec using a local volume and nodeAffinity

PersistentVolume volumeMode can now be set to “Block” (instead of the default value “Filesystem”) to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer

An nfs volume allows an existing NFS (Network File System) share to be mounted into your Pod.

have your own NFS server running with the share exported

A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.

PersistentVolumes are a way for users to “claim” durable storage (such as a GCE PersistentDisk or an iSCSI volume) without knowing the details of the particular cloud environment.

A projected volume maps several existing volume sources into the same directory.

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Each projected volume source is listed in the spec under sources

A Container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.

RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed

A secret volume is used to pass sensitive information, such as passwords, to Pods

create a secret in the Kubernetes API before you can use it

StorageOS runs as a Container within your Kubernetes environment, making local or attached storage accessible from any node within the Kubernetes cluster.

StorageOS provides block storage to Containers, accessible via a file system.

A vsphereVolume is used to mount a vSphere VMDK Volume into your Pod.

supports both VMFS and VSAN datastore.

create VMDK using one of the following methods before using with Pod.

share one volume for multiple uses in a single Pod.

volumeMounts: - name: workdir1 mountPath: /logs subPathExpr: $(POD_NAME)

env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name

Use the subPathExpr field to construct subPath directory names from Downward API environment variables

enable the VolumeSubpathEnvExpansion feature gate

emptyDir and hostPath volumes will be able to request a certain amount of space using a resource specification, and to select the type of media to use, for clusters that have several media types.

the Container Storage Interface (CSI) and Flexvolume. They enable storage vendors to create custom storage plugins without adding them to the Kubernetes repository.

Container Storage Interface (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the csi volume type to attach, mount, etc. the volumes exposed by the CSI driver.

This feature requires CSIInlineVolume feature gate to be enabled:--feature-gates=CSIInlineVolume=true

In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the “Types of Volumes” section above.

Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts.

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the Container will be propagated back to the host and to all Containers of all Pods that use the same volume.

Edit your Docker’s systemd service file. Set MountFlags as follows:MountFlags=shared

IPA – refers to the FreeIPA Master Server.

IdP – as mentioned earlier, IdP stands for Identity Provider.

SP – stands for Service Provider.   This can be a java application, jboss, etc.  It can also be a simple Apache web server

SAML – stands for Security Assertion Markup Language and refers to mod_auth_mellon here.  This provides the SSO functionality.

Openidc – stands for OpenID Connect.

Auto DevOps works with any Kubernetes cluster;

using the Docker or Kubernetes executor, with privileged mode enabled.

Base domain (needed for Auto Review Apps and Auto Deploy)

Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)

Prometheus (needed for Auto Monitoring)

scrape your Kubernetes cluster.

project level as a variable: KUBE_INGRESS_BASE_DOMAIN

A wildcard DNS A record matching the base domain(s) is required

Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).

review/ (every environment starting with review/)

staging

production

need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.

Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.

Continuous deployment to production using timed incremental rollout

Automatic deployment to staging, manual deployment to production

Auto Build creates a build of the application using an existing Dockerfile or Heroku buildpacks.

If a project’s repository contains a Dockerfile, Auto Build will use docker build to create a Docker image.

Each buildpack requires certain files to be in your project’s repository for Auto Build to successfully build your application.

Auto Test automatically runs the appropriate tests for your application using Herokuish and Heroku buildpacks by analyzing your project to detect the language and framework.

Auto Code Quality uses the Code Quality image to run static analysis and other code checks on the current code.

Static Application Security Testing (SAST) uses the SAST Docker image to run static analysis on the current code and checks for potential security issues.

Dependency Scanning uses the Dependency Scanning Docker image to run analysis on the project dependencies and checks for potential security issues.

License Management uses the License Management Docker image to search the project dependencies for their license.

Vulnerability Static Analysis for containers uses Clair to run static analysis on a Docker image and checks for potential security issues.

Review Apps are temporary application environments based on the branch’s code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur.

The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain.

Your apps should not be manipulated outside of Helm (using Kubernetes directly).

Dynamic Application Security Testing (DAST) uses the popular open source tool OWASP ZAProxy to perform an analysis on the current code and checks for potential security issues.

Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.

add the paths to a file named .gitlab-urls.txt in the root directory, one per line.

After a branch or merge request is merged into the project’s default branch (usually master), Auto Deploy deploys the application to a production environment in the Kubernetes cluster, with a namespace based on the project name and unique project ID

Auto Deploy doesn’t include deployments to staging or canary by default, but the Auto DevOps template contains job definitions for these tasks if you want to enable them.

For internal and private projects a GitLab Deploy Token will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.

If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.

If present, DB_INITIALIZE will be run as a shell command within an application pod as a helm post-install hook.

a post-install hook means that if any deploy succeeds, DB_INITIALIZE will not be processed thereafter.

DB_MIGRATE will be run as a shell command within an application pod as a helm pre-upgrade hook.

Once your application is deployed, Auto Monitoring makes it possible to monitor your application’s server and response metrics right out of the box.

annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254"

While Auto DevOps provides great defaults to get you started, you can customize almost everything to fit your needs; from custom buildpacks, to Dockerfiles, Helm charts, or even copying the complete CI/CD configuration into your project to enable staging and canary deployments, and more.

Auto DevOps uses Helm to deploy your application to Kubernetes.

make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.

specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment.

Your additions will be merged with the Auto DevOps template using the behaviour described for include

copy and paste the contents of the Auto DevOps template into your project and edit this as needed.

In order to support applications that require a database, PostgreSQL is provisioned by default.

Set up the replica variables using a project variable and scale your application by just redeploying it!

You should not scale your application using Kubernetes directly.

Some applications need to define secret variables that are accessible by the deployed application.

Auto DevOps pipelines will take your application secret variables to populate a Kubernetes secret.

Environment variables are generally considered immutable in a Kubernetes pod.

If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to 1 as a CI/CD variable), then the application will be automatically deployed to a staging environment, and a production_manual job will be created for you when you’re ready to manually deploy to production.

If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to 1 as a CI/CD variable) then two manual jobs will be created: canary which will deploy the application to the canary environment production_manual which is to be used by you when you’re ready to manually deploy to production.

If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead of the standard production job, 4 different manual jobs will be created: rollout 10% rollout 25% rollout 50% rollout 100%

To start a job, click on the play icon next to the job’s name.

not all buildpacks support Auto Test yet

When a project has been marked as private, GitLab’s Container Registry requires authentication when downloading containers.

Authentication credentials will be valid while the pipeline is running, allowing for a successful initial deployment.

We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.

The config contains configuration information that can be merged into a packaged chart to create a releasable object.

A release is a running instance of a chart, combined with a specific config.

The Helm Client is a command-line client for end users.

Interacting with the Tiller server

The Tiller Server is an in-cluster server that interacts with the Helm client, and interfaces with the Kubernetes API server.

Combining a chart and configuration to build a release

Installing charts into Kubernetes, and then tracking the subsequent release

the client is responsible for managing charts, and the server is responsible for managing releases.

The Helm client is written in the Go programming language, and uses the gRPC protocol suite to interact with the Tiller server.

The Tiller server is also written in Go. It provides a gRPC server to connect with the client, and it uses the Kubernetes client library to communicate with Kubernetes.

Configuration files are, when possible, written in YAML.

all resources created for a particular provider configuration must be destroyed before that provider configuration is removed, unless the related resources are re-configured to use a different provider configuration first.