Group items tagged

An env_file, is a convenient way to pass many environment variables to a single command in one batch.

If you have a file named .env in your project, it’s only used to put values into the docker-compose.yml file which is in the same folder. Those are used with Docker Compose and Docker Stack.

Just type docker-compose config. This way you’ll see how the docker-compose.yml file content looks after the substitution step has been performed without running anything else.

ARG are also known as build-time variables. They are only available from the moment they are ‘announced’ in the Dockerfile with an ARG instruction up to the moment when the image is built.

ENV variables are also available during the build, as soon as you introduce them with an ENV instruction. However, unlike ARG, they are also accessible by containers started from the final image.

ENV values can be overridden when starting a container,

If you don’t provide a value to expected ARG variables which don’t have a default, you’ll get an error message.

args block

You can use ARG to set the default values of ENV vars.

dynamic on-build env values

2. Pass environment variable values from your host

1. Provide values one by one

3. Take values from a file (env_file)

for each RUN statement, a new container is launched from an intermediate image.

An image is saved by the end of the command, but environment variables do not persist that way.

The precedence is, from stronger to less-strong: stuff the containerized application sets, values from single environment entries, values from the env_file(s) and finally Dockerfile defaults.

rerun just the failed job

Builds without workflows require a build job.

Refer the YAML Anchors/Aliases documentation for information about how to alias and reuse syntax to keep your .circleci/config.yml file small.

workflow orchestration with two parallel jobs

jobs run according to configured requirements, each job waiting to start until the required job finishes successfully

requires: key

fans-out to run a set of acceptance test jobs in parallel, and finally fans-in to run a common deploy job.

Workflows can be configured to wait for manual approval of a job before continuing to the next job

approval is a special job type that is only available to jobs under the workflow key

The name of the job to hold is arbitrary - it could be wait or pause, for example, as long as the job has a type: approval key in it.

schedule a workflow to run at a certain time for specific branches.

The triggers key is only added under your workflows key

using cron syntax to represent Coordinated Universal Time (UTC) for specified branches.

the commit workflow has no triggers key and will run on every git push

The nightly workflow has a triggers key and will run on the specified schedule

use a context to share environment variables

use the same shared environment variables when initiated by a user who is part of the organization.

CircleCI does not run workflows for tags unless you explicitly specify tag filters.

CircleCI branch and tag filters support the Java variant of regex pattern matching.

Each workflow has an associated workspace which can be used to transfer files to downstream jobs as the workflow progresses.

Downstream jobs can attach the workspace to their container filesystem.

Attaching the workspace downloads and unpacks each layer based on the ordering of the upstream jobs in the workflow graph.

Workflows that include jobs running on multiple branches may require data to be shared using workspaces

Files and directories named in the paths: property of persist_to_workspace will be uploaded to the workflow’s temporary workspace relative to the directory specified with the root key.

Configure a job to get saved data by configuring the attach_workspace key.

persist_to_workspace

attach_workspace

To rerun only a workflow’s failed jobs, click the Workflows icon in the app and select a workflow to see the status of each job, then click the Rerun button and select Rerun from failed.

if you do not see your workflows triggering, a configuration error is preventing the workflow from starting.

check your Workflows page of the CircleCI app (not the Job page)

When braces are used, the matching ending brace is the first ‘}’ not escaped by a backslash or within a quoted string, and not within an embedded arithmetic expansion, command substitution, or parameter expansion.

${parameter}

braces are required

If the first character of parameter is an exclamation point (!), and parameter is not a nameref, it introduces a level of variable indirection.

${parameter:-word} If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.

${parameter:?word} If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits.

${parameter:offset} ${parameter:offset:length}

Substring expansion applied to an associative array produces undefined results.

${parameter/pattern/string} The pattern is expanded to produce a pattern just as in filename expansion.

If pattern begins with ‘/’, all matches of pattern are replaced with string.

Normally only the first match is replaced

The ‘^’ operator converts lowercase letters matching pattern to uppercase

the ‘,’ operator converts matching uppercase letters to lowercase.

If the variable is empty (as it is by default) that character is the standard tab character.

“else if” non-nested conditionals

.ONESHELL special target

target-specific and pattern-specific

“shortest stem” method of choosing which pattern

make searches for included makefiles (see Including Other Makefiles)

When you run an image and generate a container, you add a new writable layer (the “container layer”) on top of the underlying layers.

Inadvertently including files that are not necessary for building an image results in a larger build context and larger image size.

To exclude files not relevant to the build (without restructuring your source repository) use a .dockerignore file. This file supports exclusion patterns similar to .gitignore files.

minimize image layers by leveraging build cache.

avoid installing extra or unnecessary packages just because they might be “nice to have.”

Decoupling applications into multiple containers makes it easier to scale horizontally and reuse containers

Limiting each container to one process is a good rule of thumb, but it is not a hard and fast rule.

do multi-stage builds and only copy the artifacts you need into the final image. This allows you to include tools and debug information in your intermediate build stages without increasing the size of the final image.

avoid duplication of packages and make the list much easier to update.

When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified.

simply comparing the instruction in the Dockerfile with one of the child images is sufficient.

For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file.

If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

In that case just the command string itself is used to find a match.

Whenever possible, use current official repositories as the basis for your images.

Using RUN apt-get update && apt-get install -y ensures your Dockerfile installs the latest package versions with no further coding or manual intervention.

cache busting

Docker executes these commands using the /bin/sh -c interpreter, which only evaluates the exit code of the last operation in the pipe to determine success.

set -o pipefail && to ensure that an unexpected error prevents the build from inadvertently succeeding.

The CMD instruction should be used to run the software contained by your image, along with any arguments.

CMD should rarely be used in the manner of CMD [“param”, “param”] in conjunction with ENTRYPOINT

The ENV instruction is also useful for providing required environment variables specific to services you wish to containerize,

COPY only supports the basic copying of local files into the container

the best use for ADD is local tar file auto-extraction into the image, as in ADD rootfs.tar.xz /

using ADD to fetch packages from remote URLs is strongly discouraged; you should use curl or wget instead

The best use for ENTRYPOINT is to set the image’s main command, allowing that image to be run as though it was that command (and then use CMD as the default flags).

the image name can double as a reference to the binary as shown in the command

The VOLUME instruction should be used to expose any database storage area, configuration storage, or files/folders created by your docker container.

use VOLUME for any mutable and/or user-serviceable parts of your image

If you absolutely need functionality similar to sudo, such as initializing the daemon as root but running it as non-root), consider using “gosu”.

always use absolute paths for your WORKDIR

An ONBUILD command executes after the current Dockerfile build completes.

Think of the ONBUILD command as an instruction the parent Dockerfile gives to the child Dockerfile

Be careful when putting ADD or COPY in ONBUILD. The “onbuild” image fails catastrophically if the new build’s context is missing the resource being added.

a single, global ClusterRoleBinding.

RoleBindings per namespace enable to restrict granted permissions to the very namespaces only that Traefik is watching over, thereby following the least-privileges principle.

The scalability can be much better when using a Deployment

you will have a Single-Pod-per-Node model when using a DaemonSet,

start with the Daemonset

The Deployment has easier up and down scaling possibilities.

The DaemonSet automatically scales to all nodes that meets a specific selector and guarantees to fill nodes one at a time.

provide the TLS certificate via a Kubernetes secret in the same namespace as the ingress.

create secret generic

Name-based Routing

Path-based Routing

Traefik will merge multiple Ingress definitions for the same host/path pair into one definition.

specify priority for ingress routes

traefik.frontend.priority

When specifying an ExternalName, Traefik will forward requests to the given host accordingly and use HTTPS when the Service port matches 443.

By default Traefik will pass the incoming Host header to the upstream resource.

traefik.frontend.passHostHeader: "false"

type: ExternalName

By default, Traefik processes every Ingress objects it observes.

It is also possible to set the ingressClass option in Traefik to a particular value. Traefik will only process matching Ingress objects.

It is possible to split Ingress traffic in a fine-grained manner between multiple deployments using service weights.

annotations: traefik.ingress.kubernetes.io/service-weights: | my-app: 99% my-app-canary: 1%

Over time, the ratio may slowly shift towards the canary deployment until it is deemed to replace the previous main application, in steps such as 5%/95%, 10%/90%, 50%/50%, and finally 100%/0%.

A chart is organized as a collection of files inside of a directory.

values.yaml # The default configuration values for this chart

charts/ # A directory containing any charts upon which this chart depends.

version: A SemVer 2 version (required)

apiVersion: The chart API version, always "v1" (required)

Every chart must have a version number. A version must follow the SemVer 2 standard.

When generating a package, the helm package command will use the version that it finds in the Chart.yaml as a token in the package name.

the appVersion field is not related to the version field. It is a way of specifying the version of the application.

appVersion: The version of the app that this contains (optional). This needn't be SemVer.

If the latest version of a chart in the repository is marked as deprecated, then the chart as a whole is considered to be deprecated.

deprecated: Whether this chart is deprecated (optional, boolean)

one chart may depend on any number of other charts.

the preferred method of declaring dependencies is by using a requirements.yaml file inside of your chart.

A requirements.yaml file is a simple file for listing your dependencies.

The repository field is the full URL to the chart repository.

helm dependency update and it will use your dependency file to download all the specified charts into your charts/ directory for you.

When helm dependency update retrieves charts, it will store them as chart archives in the charts/ directory.

All charts are loaded by default.

The condition field holds one or more YAML paths (delimited by commas). If this path exists in the top parent’s values and resolves to a boolean value, the chart will be enabled or disabled based on that boolean value.

The tags field is a YAML list of labels to associate with this chart.

all charts with tags can be enabled or disabled by specifying the tag and a boolean value.

The --set parameter can be used as usual to alter tag and condition values.

The first condition path that exists wins and subsequent ones for that chart are ignored.

The keys containing the values to be imported can be specified in the parent chart’s requirements.yaml file using a YAML list. Each item in the list is a key which is imported from the child chart’s exports field.

specifying the key data in our import list, Helm looks in the exports field of the child chart for data key and imports its contents.

To access values that are not contained in the exports key of the child chart’s values, you will need to specify the source key of the values to be imported (child) and the destination path in the parent chart’s values (parent).

To drop a dependency into your charts/ directory, use the helm fetch command

A dependency can be either a chart archive (foo-1.2.3.tgz) or an unpacked chart directory.

a single release is created with all the objects for the chart and its dependencies.

When Helm renders the charts, it will pass every file in that directory through the template engine.

Chart users may supply a YAML file that contains values. This can be provided on the command line with helm install.

Template files follow the standard conventions for writing Go templates

{{default "minio" .Values.storage}}

Values that are supplied via a values.yaml file (or via the --set flag) are accessible from the .Values object in a template.

Release.Name: The name of the release (not the chart)

Release.IsUpgrade: This is set to true if the current operation is an upgrade or rollback.

Chart: The contents of the Chart.yaml

Files: A map-like object containing all non-special files in the chart.

Files can be accessed using {{index .Files "file.name"}} or using the {{.Files.Get name}} or {{.Files.GetString name}} functions.

access the contents of the file as []byte using {{.Files.GetBytes}}

Any unknown Chart.yaml fields will be dropped

Chart.yaml cannot be used to pass arbitrarily structured data into the template.

A values file is formatted in YAML.

A chart may include a default values.yaml file

accessible inside of templates using the .Values object

Values files can declare values for the top-level chart, as well as for any of the charts that are included in that chart’s charts/ directory.

Charts at a higher level have access to all of the variables defined beneath.

lower level charts cannot access things in parent charts

Values are namespaced, but namespaces are pruned.

Helm supports special “global” value.

a way of sharing one top-level variable with all subcharts, which is useful for things like setting metadata properties like labels.

global variables of parent charts take precedence over the global variables from subcharts.

helm lint

A chart repository is an HTTP server that houses one or more packaged charts

Any HTTP server that can serve YAML files and tar files and can answer GET requests can be used as a repository server.

Helm does not provide tools for uploading charts to remote repository servers.

the only way to add a chart to $HELM_HOME/starters is to manually copy it there.

Helm provides a hook mechanism to allow chart developers to intervene at certain points in a release’s life cycle.

Hooks are declared as an annotation in the metadata section of a manifest

pre-install

post-install: Executes after all resources are loaded into Kubernetes

pre-delete

post-delete: Executes on a deletion request after all of the release’s resources have been deleted.

pre-upgrade

post-upgrade

pre-rollback

post-rollback: Executes on a rollback request after all resources have been modified.

crd-install

test-success: Executes when running helm test and expects the pod to return successfully (return code == 0).

test-failure: Executes when running helm test and expects the pod to fail (return code != 0).

Hooks allow you, the chart developer, an opportunity to perform operations at strategic points in a release lifecycle

Tiller then loads the hook with the lowest weight first (negative to positive)

Tiller returns the release name (and other data) to the client

If the resources is a Job kind, Tiller will wait until the job successfully runs to completion.

good practice to add a hook weight, and set it to 0 if weight is not important.

leave the hook resource alone.

To destroy such resources, you need to either write code to perform this operation in a pre-delete or post-delete hook or add "helm.sh/hook-delete-policy" annotation to the hook template file.

Hooks are just Kubernetes manifest files with special annotations in the metadata section

One resource can implement multiple hooks

no limit to the number of different resources that may implement a given hook.

Hook weights can be positive or negative numbers but must be represented as strings.

sort those hooks in ascending order.

Hook deletion policies

"before-hook-creation" specifies Tiller should delete the previous hook before the new hook is launched.

By default Tiller will wait for 60 seconds for a deleted hook to no longer exist in the API server before timing out.

Custom Resource Definitions (CRDs) are a special kind in Kubernetes.

The crd-install hook is executed very early during an installation, before the rest of the manifests are verified.

A common reason why the hook resource might already exist is that it was not deleted following use on a previous install/upgrade.

Helm uses Go templates for templating your resource files.

two special template functions: include and required

include function allows you to bring in another template, and then pass the results to other template functions.

The required function allows you to declare a particular values entry as required for template rendering.

Quote Strings, Don’t Quote Integers

when working with integers do not quote the values

env variables values which are expected to be string

to include a template, and then perform an operation on that template’s output, Helm has a special include function

The above includes a template called toYaml, passes it $value, and then passes the output of that template to the nindent function.

Go provides a way for setting template options to control behavior when a map is indexed with a key that’s not present in the map

The required function gives developers the ability to declare a value entry as required for template rendering.

The tpl function allows developers to evaluate strings as templates inside a template.

Rendering a external configuration file

(.Files.Get "conf/app.conf")

Image pull secrets are essentially a combination of registry, username, and password.

Automatically Roll Deployments When ConfigMaps or Secrets change

configmaps or secrets are injected as configuration files in containers

a restart may be required should those be updated with a subsequent helm upgrade

The sha256sum function can be used to ensure a deployment’s annotation section is updated if another file changes

helm upgrade --recreate-pods

"helm.sh/resource-policy": keep

resources that should not be deleted when Helm runs a helm delete

create some reusable parts in your chart

In the templates/ directory, any file that begins with an underscore(_) is not expected to output a Kubernetes manifest file.

The current best practice for composing a complex application from discrete parts is to create a top-level umbrella chart that exposes the global configurations, and then use the charts/ subdirectory to embed each of the components.

SAP’s Converged charts: These charts install SAP Converged Cloud a full OpenStack IaaS on Kubernetes. All of the charts are collected together in one GitHub repository, except for a few submodules.

Deis’s Workflow: This chart exposes the entire Deis PaaS system with one chart. But it’s different from the SAP chart in that this umbrella chart is built from each component, and each component is tracked in a different Git repository.

YAML is a superset of JSON

any valid JSON structure ought to be valid in YAML.

As a best practice, templates should follow a YAML-like syntax unless the JSON syntax substantially reduces the risk of a formatting issue.

There are functions in Helm that allow you to generate random data, cryptographic keys, and so on.

a chart repository is a location where packaged charts can be stored and shared.

A chart repository is an HTTP server that houses an index.yaml file and optionally some packaged charts.

Because a chart repository can be any HTTP server that can serve YAML and tar files and can answer GET requests, you have a plethora of options when it comes down to hosting your own chart repository.

It is not required that a chart package be located on the same server as the index.yaml file.

A valid chart repository must have an index file. The index file contains information about each chart in the chart repository.

The Helm project provides an open-source Helm repository server called ChartMuseum that you can host yourself.

$ helm repo index fantastic-charts --url https://fantastic-charts.storage.googleapis.com

A repository will not be added if it does not contain a valid index.yaml

add the repository to their helm client via the helm repo add [NAME] [URL] command with any name they would like to use to reference the repository.

Helm has provenance tools which help chart users verify the integrity and origin of a package.

Integrity is established by comparing a chart to a provenance record

The provenance file contains a chart’s YAML file plus several pieces of verification information

Chart repositories serve as a centralized collection of Helm charts.

Chart repositories must make it possible to serve provenance files over HTTP via a specific request, and must make them available at the same URI path as the chart.

We don’t want to be “the certificate authority” for all chart signers. Instead, we strongly favor a decentralized model, which is part of the reason we chose OpenPGP as our foundational technology.

The Keybase platform provides a public centralized repository for trust information.

A chart contains a number of Kubernetes resources and components that work together.

A test in a helm chart lives under the templates/ directory and is a pod definition that specifies a container with a given command to run.

The pod definition must contain one of the helm test hook annotations: helm.sh/hook: test-success or helm.sh/hook: test-failure

helm test

nest your test suite under a tests/ directory like <chart-name>/templates/tests/

When a PVC is created, the vSphere Cloud Provider checks if the user specified datastore satisfies the gold storage policy requirements. If it does, the vSphere Cloud Provider will provision the PersistentVolume on the user specified datastore. If not, it will create an error telling the user that the specified datastore is not compatible with gold storage policy requirements.

The Kubernetes user will have the ability to specify custom vSAN Storage Capabilities during dynamic volume provisioning.

DNS component in FreeIPA is optional and user may choose to manage all DNS records manually in other third party DNS server.

Clients can be configured to automatically run DNS updates (nsupdate) when their IP address changes and thus keeping its DNS record up-to-date. DNS zones can be configured to synchronize client's reverse (PTR) record along with the forward (A, AAAA) DNS record.

You should only use names which are delegated to you by the parent domain.

DNSSEC validation.

General advice about DNS views is do not use them because views make DNS deployment harder to maintain and security benefits are questionable (when compared with ACL).

FreeIPA LDAP directory information tree is by default accessible to any user in the network

As DNS data are often considered as sensitive and as having access to cn=dns tree would be basically equal to being able to run zone transfer to all FreeIPA managed DNS zones, contents of this tree in LDAP are hidden by default.

standard system log (/var/log/messages or system journal)

BIND configuration (/etc/named.conf) can be updated to produce a more detailed log.

The biggest problem is that many long-running branches emerge that all contain part of the changes.

It is a convention to call your default branch master and to mostly branch from and merge to this.

Nowadays, most organizations practice continuous delivery, which means that your default branch can be deployed.

Continuous delivery removes the need for hotfix and release branches, including all the ceremony they introduce.

Merging everything into the master branch and frequently deploying means you minimize the amount of unreleased code, which is in line with lean and continuous delivery best practices.

GitHub flow assumes you can deploy to production every time you merge a feature branch.

You can deploy a new version by merging master into the production branch. If you need to know what code is in production, you can just checkout the production branch to see.

Production branch

Environment branches

have an environment that is automatically updated to the master branch.

deploy the master branch to staging.

To deploy to pre-production, create a merge request from the master branch to the pre-production branch.

Go live by merging the pre-production branch into the production branch.

Release branches

work with release branches if you need to release software to the outside world.

each branch contains a minor version

After announcing a release branch, only add serious bug fixes to the branch.

merge these bug fixes into master, and then cherry-pick them into the release branch.

Merging into master and then cherry-picking into release is called an “upstream first” policy

Tools such as GitHub and Bitbucket choose the name “pull request” since the first manual action is to pull the feature branch.

Tools such as GitLab and others choose the name “merge request” since the final action is to merge the feature branch.

the merge request automatically updates when new commits are pushed to the branch.

If the assigned person does not feel comfortable, they can request more changes or close the merge request without merging.

In GitLab, it is common to protect the long-lived branches, e.g., the master branch, so that most developers can’t modify them.

After you merge a feature branch, you should remove it from the source control software.

Having a reason for every code change helps to inform the rest of the team and to keep the scope of a feature branch small.

For example, the issue title “As an administrator, I want to remove users without receiving an error” is better than “Admin can’t remove users.”

create a branch for the issue from the master branch

If you open the merge request but do not assign it to anyone, it is a “Work In Progress” merge request.

Start the title of the merge request with [WIP] or WIP: to prevent it from being merged before it’s ready.

When they press the merge button, GitLab merges the code and creates a merge commit that makes this event easily visible later on.

Suppose that a branch is merged but a problem occurs and the issue is reopened. In this case, it is no problem to reuse the same branch name since the first branch was deleted when it was merged.