Group items tagged

To use a secret, a pod needs to reference the secret.

--from-file

You can also create a Secret in a file first, in json or yaml format, and then create that object.

The Secret contains two maps: data and stringData.

Kubernetes automatically creates secrets which contain credentials for accessing the API and it automatically modifies your pods to use this type of secret.

kubectl get and kubectl describe avoid showing the contents of a secret by default.

stringData field is provided for convenience, and allows you to provide secret data as unencoded strings.

where you are deploying an application that uses a Secret to store a configuration file, and you want to populate parts of that configuration file during your deployment process.

a field is specified in both data and stringData, the value from stringData is used.

The keys of data and stringData must consist of alphanumeric characters, ‘-’, ‘_’ or ‘.’.

Newlines are not valid within these strings and must be omitted.

When using the base64 utility on Darwin/macOS users should avoid using the -b option to split long lines.

create a Secret from generators and then apply it to create the object on the Apiserver.

The generated Secrets name has a suffix appended by hashing the contents.

Multiple pods can reference the same secret.

each container needs its own volumeMounts block, but only one .spec.volumes is needed per secret

use .spec.volumes[].secret.items field to change target path of each key:

You can also specify the permission mode bits files part of a secret will have. If you don’t specify any, 0644 is used by default.

Kubelet is checking whether the mounted secret is fresh on every periodic sync.

cache propagation delay depends on the chosen cache type

Multiple pods can reference the same secret.

env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base-64 decoded values of the secret data.

An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry password to the Kubelet so it can pull a private image on behalf of your Pod.

Individual secrets are limited to 1MiB in size.

Kubelet only supports use of secrets for Pods it gets from the API server.

Secrets must be created before they are consumed in pods as environment variables unless they are marked as optional.

Once a pod is scheduled, the kubelet will try to fetch the secret value.

Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret.

volumes: - name: secret-volume secret: secretName: ssh-key-secret

You do not need to escape special characters in passwords from files

make that key begin with a dot

Dotfiles in secret volume

.secret-file

a frontend container which handles user interaction and business logic, but which cannot see the private key;

a signer container that can see the private key, and responds to simple signing requests from the frontend

watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided

watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.

additional precautions with secret objects, such as avoiding writing them to disk where possible.

A secret is only sent to a node if a pod on that node requires it

only the secrets that a pod requests are potentially visible within its containers

each container in a pod has to request the secret volume in its volumeMounts for it to be visible within the container.

In the API server secret data is stored in etcdConsistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

limit access to etcd to admin users

A user who can create a pod that uses a secret can also see the value of that secret.

anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet.

The biggest problem is that many long-running branches emerge that all contain part of the changes.

It is a convention to call your default branch master and to mostly branch from and merge to this.

Nowadays, most organizations practice continuous delivery, which means that your default branch can be deployed.

Continuous delivery removes the need for hotfix and release branches, including all the ceremony they introduce.

Merging everything into the master branch and frequently deploying means you minimize the amount of unreleased code, which is in line with lean and continuous delivery best practices.

GitHub flow assumes you can deploy to production every time you merge a feature branch.

You can deploy a new version by merging master into the production branch. If you need to know what code is in production, you can just checkout the production branch to see.

Production branch

Environment branches

have an environment that is automatically updated to the master branch.

deploy the master branch to staging.

To deploy to pre-production, create a merge request from the master branch to the pre-production branch.

Go live by merging the pre-production branch into the production branch.

Release branches

work with release branches if you need to release software to the outside world.

each branch contains a minor version

After announcing a release branch, only add serious bug fixes to the branch.

merge these bug fixes into master, and then cherry-pick them into the release branch.

Merging into master and then cherry-picking into release is called an “upstream first” policy

Tools such as GitHub and Bitbucket choose the name “pull request” since the first manual action is to pull the feature branch.

Tools such as GitLab and others choose the name “merge request” since the final action is to merge the feature branch.

the merge request automatically updates when new commits are pushed to the branch.

If the assigned person does not feel comfortable, they can request more changes or close the merge request without merging.

In GitLab, it is common to protect the long-lived branches, e.g., the master branch, so that most developers can’t modify them.

After you merge a feature branch, you should remove it from the source control software.

Having a reason for every code change helps to inform the rest of the team and to keep the scope of a feature branch small.

For example, the issue title “As an administrator, I want to remove users without receiving an error” is better than “Admin can’t remove users.”

create a branch for the issue from the master branch

If you open the merge request but do not assign it to anyone, it is a “Work In Progress” merge request.

Start the title of the merge request with [WIP] or WIP: to prevent it from being merged before it’s ready.

When they press the merge button, GitLab merges the code and creates a merge commit that makes this event easily visible later on.

Suppose that a branch is merged but a problem occurs and the issue is reopened. In this case, it is no problem to reuse the same branch name since the first branch was deleted when it was merged.

GitLab closes these issues when the code is merged into the default branch.

If you have an issue that spans across multiple repositories, create an issue for each repository and link all issues to a parent issue.

use an interactive rebase (rebase -i) to squash multiple commits into one or reorder them.

Rebasing creates new commits for all your changes, which can cause confusion because the same change would have multiple identifiers.

if someone has already reviewed your code, rebasing makes it hard to tell what changed since the last review.

it is a bad idea to rebase commits that you have already pushed.

If you revert a merge commit and then change your mind, revert the revert commit to redo the merge.

Often, people avoid merge commits by just using rebase to reorder their commits after the commits on the master branch.

Using rebase prevents a merge commit when merging master into your feature branch, and it creates a neat linear history.

every time you rebase, you have to resolve similar conflicts.

A good way to prevent creating many merge commits is to not frequently merge master into the feature branch.

keep your feature branches short-lived.

If your feature branches often take more than a day of work, try to split your features into smaller units of work.

You could also use feature toggles to hide incomplete features so you can still merge back into master every day.

Your codebase should be clean, but your history should represent what actually happened.

If you rebase code, the history is incorrect, and there is no way for tools to remedy this because they can’t deal with changing commit identifiers

Commit often and push frequently

You should push your feature branch frequently, even when it is not yet ready for review.

A commit message should reflect your intention, not just the contents of the commit.

each merge request must be tested before it is accepted.

test the master branch after each change.

If new commits in master cause merge conflicts with the feature branch, merge master back into the branch to make the CI server re-run the tests.

Do not merge from upstream again if your code can work and merge cleanly without doing so.

You can use role-based access control (RBAC) and other security mechanisms to make sure that users and workloads can get access to the resources they need, while keeping workloads, and the cluster itself, secure. You can set limits on the resources that users and workloads can access by managing policies and container resources.

you need to plan how to scale to relieve increased pressure from more requests to the control plane and worker nodes or scale down to reduce unused resources.

Managed control plane: Let the provider manage the scale and availability of the cluster's control plane, as well as handle patches and upgrades.

The simplest Kubernetes cluster has the entire control plane and worker node services running on the same machine.

You can deploy a control plane using tools such as kubeadm, kops, and kubespray.

Secure communications between control plane services are implemented using certificates.

Separate and backup etcd service: The etcd services can either run on the same machines as other control plane services or run on separate machines

Create multiple control plane systems: For high availability, the control plane should not be limited to a single machine

Some deployment tools set up Raft consensus algorithm to do leader election of Kubernetes services. If the primary goes away, another service elects itself and take over.

Groups of zones are referred to as regions.

if you installed with kubeadm, there are instructions to help you with Certificate Management and Upgrading kubeadm clusters.

Production-quality workloads need to be resilient and anything they rely on needs to be resilient (such as CoreDNS).

Add nodes to the cluster: If you are managing your own cluster you can add nodes by setting up your own machines and either adding them manually or having them register themselves to the cluster’s apiserver.

Set up node health checks: For important workloads, you want to make sure that the nodes and pods running on those nodes are healthy.

Authentication: The apiserver can authenticate users using client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth.

Authorization: When you set out to authorize your regular users, you will probably choose between RBAC and ABAC authorization.

Role-based access control (RBAC): Lets you assign access to your cluster by allowing specific sets of permissions to authenticated users. Permissions can be assigned for a specific namespace (Role) or across the entire cluster (ClusterRole).

Attribute-based access control (ABAC): Lets you create policies based on resource attributes in the cluster and will allow or deny access based on those attributes.

Set limits on workload resources

Set namespace limits: Set per-namespace quotas on things like memory and CPU

Prepare for DNS demand: If you expect workloads to massively scale up, your DNS service must be ready to scale up as well.

Container filesystems are visible to other containers in the pod through the /proc/$pid/root link. This makes debugging easier, but it also means that filesystem secrets are protected only by filesystem permissions.

Login to the container

Baseimage-docker only advocates running multiple OS processes inside a single container.

Password and challenge-response authentication are disabled by default. Only key authentication is allowed.

A tool for running a command as another user

The Docker developers advocate the philosophy of running a single logical service per container. A logical service can consist of multiple OS processes.

All syslog messages are forwarded to "docker logs".

Splitting your logical service into multiple OS processes also makes sense from a security standpoint.

Baseimage-docker provides tools to encourage running processes as different users

sometimes it makes sense to run multiple services in a single container, and sometimes it doesn't.

Baseimage-docker advocates running multiple OS processes inside a single container, and a single logical service can consist of multiple OS processes.

using environment variables to pass parameters to containers is very much the "Docker way"

add additional daemons (e.g. your own app) to the image by creating runit entries.

the shell script must run the daemon without letting it daemonize/fork it.

All executable scripts in /etc/my_init.d, if this directory exists. The scripts are run in lexicographic order.

variables will also be passed to all child processes

Environment variables on Unix are inherited on a per-process basis

there is no good central place for defining environment variables for all applications and services

centrally defining environment variables

One of the ideas behind Docker is that containers should be stateless, easily restartable, and behave like a black box.

a one-shot command in a new container

immediately exit after the command exits,

Nginx is one such example: it removes all environment variables unless you explicitly instruct it to retain them through the env configuration option.

Mechanisms for easily running multiple processes, without violating the Docker philosophy

Ubuntu is not designed to be run inside Docker

According to the Unix process model, the init process -- PID 1 -- inherits all orphaned child processes and must reap them

Syslog-ng seems to be much more stable

cron daemon

Rotates and compresses logs

/sbin/setuser

A tool for installing apt packages that automatically cleans up after itself.

A daemon is a program which runs in the background of its system, such as a web server.

The shell script must be called run, must be executable, and is to be placed in the directory /etc/service/<NAME>. runsv will switch to the directory and invoke ./run after your container starts.

If any script exits with a non-zero exit code, the booting will fail.

If your process is started with a shell script, make sure you exec the actual process, otherwise the shell will receive the signal and not your process.

any environment variables set with docker run --env or with the ENV command in the Dockerfile, will be picked up by my_init

they will not see the environment variables that were originally passed by Docker.

my_init imports environment variables from the directory /etc/container_environment

/etc/container_environment.sh - a dump of the environment variables in Bash format.

modify the environment variables in my_init (and therefore the environment variables in all child processes that are spawned after that point in time), by altering the files in /etc/container_environment

my_init only activates changes in /etc/container_environment when running startup scripts

environment variables don't contain sensitive data, then you can also relax the permissions

Syslog messages are forwarded to the console

syslog-ng is started separately before the runit supervisor process, and shutdown after runit exits.

/sbin/my_init --skip-startup-files --quiet --

By default, no keys are installed, so nobody can login

provide a pregenerated, insecure key (PuTTY format)

RUN /usr/sbin/enable_insecure_key

docker run YOUR_IMAGE /sbin/my_init --enable-insecure-key

RUN cat /tmp/your_key.pub >> /root/.ssh/authorized_keys && rm -f /tmp/your_key.pub

The default baseimage-docker installs syslog-ng, cron and sshd services during the build process

in leveraging regular Ruby classes and object oriented design patterns to build a simple, robust and scaleable authorization system

map to the name of a particular controller action

In the generated ApplicationPolicy, the model object is called record.

record

authorize

authorize would have done something like this: raise "not authorized" unless PostPolicy.new(current_user, @post).update?

pass a second argument to authorize if the name of the permission you want to check doesn't match the action name.

you can chain it

authorize returns the object passed to it

the policy method in both the view and controller.

have some kind of view listing records which a particular user has access to

ActiveRecord::Relation

Instances of this class respond to the method resolve, which should return some kind of result which can be iterated over.

scope.where(published: true)

use this class from your controller via the policy_scope method:

PostPolicy::Scope.new(current_user, Post).resolve

policy_scope(@user.posts).each

This method will raise an exception if authorize has not yet been called.

verify_policy_scoped to your controller. This will raise an exception in the vein of verify_authorized. However, it tracks if policy_scope is used instead of authorize

need to conditionally bypass verification, you can use skip_authorization

skip_policy_scope

Having a mechanism that ensures authorization happens allows developers to thoroughly test authorization scenarios as units on the policy objects themselves.

Pundit doesn't do anything you couldn't have easily done yourself. It's a very small library, it just provides a few neat helpers.

all of the policy and scope classes are just plain Ruby classes

rails g pundit:policy post

define a filter that redirects unauthenticated users to the login page

fail more gracefully

raise Pundit::NotAuthorizedError, "must be logged in" unless user

having rails handle them as a 403 error and serving a 403 error page.

config.action_dispatch.rescue_responses["Pundit::NotAuthorizedError"] = :forbidden

with I18n to generate error messages

retrieve a policy for a record outside the controller or view

define a method in your controller called pundit_user

Pundit strongly encourages you to model your application in such a way that the only context you need for authorization is a user object and a domain model that you want to check authorization for.

Pundit does not allow you to pass additional arguments to policies

set up a permitted_attributes method in your policy

policy(@post).permitted_attributes

permitted_attributes(@post)

Pundit provides a convenient helper method

permit different attributes based on the current action,

If you have defined an action-specific method on your policy for the current action, the permitted_attributes helper will call it instead of calling permitted_attributes on your controller

If you don't have an instance for the first argument to authorize, then you can pass the class

It is not some kind of failsafe mechanism or authorization mechanism.

Pundit will work just fine without using verify_authorized and verify_policy_scoped

rolify just adds some class methods in an existing User class

disambiguate the relationship

a single, global ClusterRoleBinding.

RoleBindings per namespace enable to restrict granted permissions to the very namespaces only that Traefik is watching over, thereby following the least-privileges principle.

The scalability can be much better when using a Deployment

you will have a Single-Pod-per-Node model when using a DaemonSet,

start with the Daemonset

The Deployment has easier up and down scaling possibilities.

The DaemonSet automatically scales to all nodes that meets a specific selector and guarantees to fill nodes one at a time.

provide the TLS certificate via a Kubernetes secret in the same namespace as the ingress.

create secret generic

Name-based Routing

Path-based Routing

Traefik will merge multiple Ingress definitions for the same host/path pair into one definition.

specify priority for ingress routes

traefik.frontend.priority

When specifying an ExternalName, Traefik will forward requests to the given host accordingly and use HTTPS when the Service port matches 443.

By default Traefik will pass the incoming Host header to the upstream resource.

traefik.frontend.passHostHeader: "false"

type: ExternalName

By default, Traefik processes every Ingress objects it observes.

It is also possible to set the ingressClass option in Traefik to a particular value. Traefik will only process matching Ingress objects.

It is possible to split Ingress traffic in a fine-grained manner between multiple deployments using service weights.

annotations: traefik.ingress.kubernetes.io/service-weights: | my-app: 99% my-app-canary: 1%

Over time, the ratio may slowly shift towards the canary deployment until it is deemed to replace the previous main application, in steps such as 5%/95%, 10%/90%, 50%/50%, and finally 100%/0%.

initialize the local CLI

install Tiller into your Kubernetes cluster

helm install

helm init --upgrade

By default, when Tiller is installed, it does not have authentication enabled.

helm repo update

Without a max history set the history is kept indefinitely, leaving a large number of records for helm and tiller to maintain.

helm init --upgrade

Whenever you install a chart, a new release is created.

helm list function will show you a list of all deployed releases.

helm delete

helm status

the Helm server (Tiller).

The Helm client (helm)

brew install kubernetes-helm

it can also be run locally, and configured to talk to a remote Kubernetes cluster.

Role-Based Access Control - RBAC for short

run Tiller in an RBAC-enabled Kubernetes cluster.

run kubectl get pods --namespace kube-system and see Tiller running.

helm inspect

Helm will look for Tiller in the kube-system namespace unless --tiller-namespace or TILLER_NAMESPACE is set.

For development, it is sometimes easier to work on Tiller locally, and configure it to connect to a remote Kubernetes cluster.

helm version should show you both the client and server version.

The --node-selectors flag allows us to specify the node labels required for scheduling the Tiller pod.

--override allows you to specify properties of Tiller’s deployment manifest.

helm init --override manipulates the specified properties of the final manifest (there is no “values” file).

The --output flag allows us skip the installation of Tiller’s deployment manifest and simply output the deployment manifest to stdout in either JSON or YAML format.

switch from the default backend to the secrets backend, you’ll have to do the migration for this on your own.

a beta SQL storage backend that stores release information in an SQL database (only postgres has been tested so far).

Helm requires that kubelet have access to a copy of the socat program to proxy connections to the Tiller API.

A Release is an instance of a chart running in a Kubernetes cluster. One chart can often be installed many times into the same cluster.

helm init --client-only

helm init --dry-run --debug

A panic in Tiller is almost always the result of a failure to negotiate with the Kubernetes API server

Tiller and Helm have to negotiate a common version to make sure that they can safely communicate without breaking API assumptions

helm delete --purge

Helm stores some files in $HELM_HOME, which is located by default in ~/.helm

A Chart is a Helm package. It contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster.

A Repository is the place where charts can be collected and shared.

Set the $HELM_HOME environment variable

Helm installs charts into Kubernetes, creating a new release for each installation. And to find new charts, you can search Helm chart repositories.

chart repository is named stable by default

helm search shows you all of the available charts

helm inspect

To install a new package, use the helm install command. At its simplest, it takes only one argument: The name of the chart.

If you want to use your own release name, simply use the --name flag on helm install

additional configuration steps you can or should take.

Helm does not wait until all of the resources are running before it exits. Many charts require Docker images that are over 600M in size, and may take a long time to install into the cluster.

helm status

helm inspect values

helm inspect values stable/mariadb

override any of these settings in a YAML formatted file, and then pass that file during installation.

helm install -f config.yaml stable/mariadb

--values (or -f): Specify a YAML file with overrides.

--set (and its variants --set-string and --set-file): Specify overrides on the command line.

Values that have been --set can be cleared by running helm upgrade with --reset-values specified.

Chart designers are encouraged to consider the --set usage when designing the format of a values.yaml file.

--set-file key=filepath is another variant of --set. It reads the file and use its content as a value.

inject a multi-line text into values without dealing with indentation in YAML.

An unpacked chart directory

When a new version of a chart is released, or when you want to change the configuration of your release, you can use the helm upgrade command.

Kubernetes charts can be large and complex, Helm tries to perform the least invasive upgrade.

$ helm upgrade -f panda.yaml happy-panda stable/mariadb

deployment

If both are used, --set values are merged into --values with higher precedence.

The helm get command is a useful tool for looking at a release in the cluster.

A release version is an incremental revision. Every time an install, upgrade, or rollback happens, the revision number is incremented by 1.

helm history

you can rollback a deleted resource, and have it re-activate.

helm repo list

helm repo add

helm repo update