Group items tagged

When a Workspace is declared in a job, one or more files or directories can be added. Each addition creates a new layer in the Workspace filesystem. Downstreams jobs can then use this Workspace for its own needs or add more layers on top.

Unlike caching, Workspaces are not shared between runs as they no longer exists once a Workflow is complete.

A prime example is package dependency managers such as Yarn, Bundler, or Pip.

Caches are global within a project, a cache saved on one branch will be used by others so they should only be used for data that is OK to share across Branches

Artifacts are used for longer-term storage of the outputs of your build process.

If your project needs to be packaged in some form or fashion, say an Android app where the .apk file is uploaded to Google Play, that’s a great example of an artifact.

Once you have a cluster running and you want to add/reconnect another node to it, you must supply an address of one of the cluster members in the cluster address URL

It will automatically retrieve the cluster map and reconnect to the rest of the nodes

it's better to list all nodes of the cluster so that any node can join a cluster connecting to any other node, even if one or more are down

The wsrep_cluster_address parameter should be added in my.cnf of each node, listing all the nodes of the cluster,

the minimum recommended number of nodes in a cluster is 3

While two of the members will be engaged in state transfer, the remaining member(s) will be able to keep on serving client requests.

Zabbix by default uses "pull" model when a server connects to agents on each monitoring machine, agents periodically gather the info and send it to a server.

Prometheus prefers "pull" model when a server gather info from client machines.

expose metrics for Prometheus (similar to "agents" for Zabbix)

Zabbix uses its own tcp-based communication protocol between agents and a server.

Prometheus offers solution for alerting that is separated from its core into Alertmanager application.

initialize the local CLI

install Tiller into your Kubernetes cluster

helm install

helm init --upgrade

By default, when Tiller is installed, it does not have authentication enabled.

helm repo update

Without a max history set the history is kept indefinitely, leaving a large number of records for helm and tiller to maintain.

helm init --upgrade

Whenever you install a chart, a new release is created.

helm list function will show you a list of all deployed releases.

helm delete

helm status

the Helm server (Tiller).

The Helm client (helm)

brew install kubernetes-helm

it can also be run locally, and configured to talk to a remote Kubernetes cluster.

Role-Based Access Control - RBAC for short

run Tiller in an RBAC-enabled Kubernetes cluster.

run kubectl get pods --namespace kube-system and see Tiller running.

helm inspect

Helm will look for Tiller in the kube-system namespace unless --tiller-namespace or TILLER_NAMESPACE is set.

For development, it is sometimes easier to work on Tiller locally, and configure it to connect to a remote Kubernetes cluster.

helm version should show you both the client and server version.

The --node-selectors flag allows us to specify the node labels required for scheduling the Tiller pod.

--override allows you to specify properties of Tiller’s deployment manifest.

helm init --override manipulates the specified properties of the final manifest (there is no “values” file).

The --output flag allows us skip the installation of Tiller’s deployment manifest and simply output the deployment manifest to stdout in either JSON or YAML format.

switch from the default backend to the secrets backend, you’ll have to do the migration for this on your own.

a beta SQL storage backend that stores release information in an SQL database (only postgres has been tested so far).

Helm requires that kubelet have access to a copy of the socat program to proxy connections to the Tiller API.

A Release is an instance of a chart running in a Kubernetes cluster. One chart can often be installed many times into the same cluster.

helm init --client-only

helm init --dry-run --debug

A panic in Tiller is almost always the result of a failure to negotiate with the Kubernetes API server

Tiller and Helm have to negotiate a common version to make sure that they can safely communicate without breaking API assumptions

helm delete --purge

Helm stores some files in $HELM_HOME, which is located by default in ~/.helm

A Chart is a Helm package. It contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster.

A Repository is the place where charts can be collected and shared.

Set the $HELM_HOME environment variable

Helm installs charts into Kubernetes, creating a new release for each installation. And to find new charts, you can search Helm chart repositories.

chart repository is named stable by default

helm search shows you all of the available charts

helm inspect

To install a new package, use the helm install command. At its simplest, it takes only one argument: The name of the chart.

If you want to use your own release name, simply use the --name flag on helm install

additional configuration steps you can or should take.

Helm does not wait until all of the resources are running before it exits. Many charts require Docker images that are over 600M in size, and may take a long time to install into the cluster.

helm status

helm inspect values

helm inspect values stable/mariadb

override any of these settings in a YAML formatted file, and then pass that file during installation.

helm install -f config.yaml stable/mariadb

--values (or -f): Specify a YAML file with overrides.

--set (and its variants --set-string and --set-file): Specify overrides on the command line.

Values that have been --set can be cleared by running helm upgrade with --reset-values specified.

Chart designers are encouraged to consider the --set usage when designing the format of a values.yaml file.

--set-file key=filepath is another variant of --set. It reads the file and use its content as a value.

inject a multi-line text into values without dealing with indentation in YAML.

An unpacked chart directory

When a new version of a chart is released, or when you want to change the configuration of your release, you can use the helm upgrade command.

Kubernetes charts can be large and complex, Helm tries to perform the least invasive upgrade.

$ helm upgrade -f panda.yaml happy-panda stable/mariadb

deployment

If both are used, --set values are merged into --values with higher precedence.

The helm get command is a useful tool for looking at a release in the cluster.

A release version is an incremental revision. Every time an install, upgrade, or rollback happens, the revision number is incremented by 1.

helm history

you can rollback a deleted resource, and have it re-activate.

helm repo list

helm repo add

helm repo update

helm create

helm lint

helm package

Charts that are archived can be loaded into chart repositories.

chart repository server

Tiller can be installed into any namespace.

Limiting Tiller to only be able to install into specific namespaces and/or resource types is controlled by Kubernetes RBAC roles and rolebindings

Release names are unique PER TILLER INSTANCE

Charts should only contain resources that exist in a single namespace.

not recommended to have multiple Tillers configured to manage resources in the same namespace.

a client-side Helm plugin. A plugin is a tool that can be accessed through the helm CLI, but which is not part of the built-in Helm codebase.

Helm plugins are add-on tools that integrate seamlessly with Helm. They provide a way to extend the core feature set of Helm, but without requiring every new feature to be written in Go and added to the core tool.

Helm plugins live in $(helm home)/plugins

The Helm plugin model is partially modeled on Git’s plugin model

helm referred to as the porcelain layer, with plugins being the plumbing.

helm plugin install https://github.com/technosophos/helm-template

command is the command that this plugin will execute when it is called.

Environment variables are interpolated before the plugin is executed.

The command itself is not executed in a shell. So you can’t oneline a shell script.

Helm is able to fetch Charts using HTTP/S

Variables like KUBECONFIG are set for the plugin if they are set in the outer environment.

In Kubernetes, granting a role to an application-specific service account is a best practice to ensure that your application is operating in the scope that you have specified.

restrict Tiller’s capabilities to install resources to certain namespaces, or to grant a Helm client running access to a Tiller instance.

Service account with cluster-admin role

The cluster-admin role is created by default in a Kubernetes cluster

Deploy Tiller in a namespace, restricted to deploying resources only in that namespace

Deploy Tiller in a namespace, restricted to deploying resources in another namespace

When running a Helm client in a pod, in order for the Helm client to talk to a Tiller instance, it will need certain privileges to be granted.

SSL Between Helm and Tiller

The Tiller authentication model uses client-side SSL certificates.

creating an internal CA, and using both the cryptographic and identity functions of SSL.

Helm is a powerful and flexible package-management and operations tool for Kubernetes.

default installation applies no security configurations

with a cluster that is well-secured in a private network with no data-sharing or no other users or teams.

With great power comes great responsibility.

Choose the Best Practices you should apply to your helm installation

Role-based access control, or RBAC

Tiller’s gRPC endpoint and its usage by Helm

Kubernetes employ a role-based access control (or RBAC) system (as do modern operating systems) to help mitigate the damage that can be done if credentials are misused or bugs exist.

In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied.

Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.

release information

charts

charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.

Helm’s provenance tools to ensure the provenance and integrity of charts

The biggest problem is that many long-running branches emerge that all contain part of the changes.

It is a convention to call your default branch master and to mostly branch from and merge to this.

Nowadays, most organizations practice continuous delivery, which means that your default branch can be deployed.

Continuous delivery removes the need for hotfix and release branches, including all the ceremony they introduce.

Merging everything into the master branch and frequently deploying means you minimize the amount of unreleased code, which is in line with lean and continuous delivery best practices.

GitHub flow assumes you can deploy to production every time you merge a feature branch.

You can deploy a new version by merging master into the production branch. If you need to know what code is in production, you can just checkout the production branch to see.

Production branch

Environment branches

have an environment that is automatically updated to the master branch.

deploy the master branch to staging.

To deploy to pre-production, create a merge request from the master branch to the pre-production branch.

Go live by merging the pre-production branch into the production branch.

Release branches

work with release branches if you need to release software to the outside world.

each branch contains a minor version

After announcing a release branch, only add serious bug fixes to the branch.

merge these bug fixes into master, and then cherry-pick them into the release branch.

Merging into master and then cherry-picking into release is called an “upstream first” policy

Tools such as GitHub and Bitbucket choose the name “pull request” since the first manual action is to pull the feature branch.

Tools such as GitLab and others choose the name “merge request” since the final action is to merge the feature branch.

the merge request automatically updates when new commits are pushed to the branch.

If the assigned person does not feel comfortable, they can request more changes or close the merge request without merging.

In GitLab, it is common to protect the long-lived branches, e.g., the master branch, so that most developers can’t modify them.

After you merge a feature branch, you should remove it from the source control software.

Having a reason for every code change helps to inform the rest of the team and to keep the scope of a feature branch small.

For example, the issue title “As an administrator, I want to remove users without receiving an error” is better than “Admin can’t remove users.”

create a branch for the issue from the master branch

If you open the merge request but do not assign it to anyone, it is a “Work In Progress” merge request.

Start the title of the merge request with [WIP] or WIP: to prevent it from being merged before it’s ready.

When they press the merge button, GitLab merges the code and creates a merge commit that makes this event easily visible later on.

Suppose that a branch is merged but a problem occurs and the issue is reopened. In this case, it is no problem to reuse the same branch name since the first branch was deleted when it was merged.

GitLab closes these issues when the code is merged into the default branch.

If you have an issue that spans across multiple repositories, create an issue for each repository and link all issues to a parent issue.

use an interactive rebase (rebase -i) to squash multiple commits into one or reorder them.

Rebasing creates new commits for all your changes, which can cause confusion because the same change would have multiple identifiers.

if someone has already reviewed your code, rebasing makes it hard to tell what changed since the last review.

it is a bad idea to rebase commits that you have already pushed.

If you revert a merge commit and then change your mind, revert the revert commit to redo the merge.

Often, people avoid merge commits by just using rebase to reorder their commits after the commits on the master branch.

Using rebase prevents a merge commit when merging master into your feature branch, and it creates a neat linear history.

every time you rebase, you have to resolve similar conflicts.

A good way to prevent creating many merge commits is to not frequently merge master into the feature branch.

keep your feature branches short-lived.

If your feature branches often take more than a day of work, try to split your features into smaller units of work.

You could also use feature toggles to hide incomplete features so you can still merge back into master every day.

Your codebase should be clean, but your history should represent what actually happened.

If you rebase code, the history is incorrect, and there is no way for tools to remedy this because they can’t deal with changing commit identifiers

Commit often and push frequently

You should push your feature branch frequently, even when it is not yet ready for review.

A commit message should reflect your intention, not just the contents of the commit.

each merge request must be tested before it is accepted.

test the master branch after each change.

If new commits in master cause merge conflicts with the feature branch, merge master back into the branch to make the CI server re-run the tests.

Do not merge from upstream again if your code can work and merge cleanly without doing so.

a well-cared for log is a beautiful and useful thing

Reviewing others’ commits and pull requests becomes something worth doing, and suddenly can be done independently.

Understanding why something happened months or years ago becomes not only possible but efficient.

how to write an individual commit message.

What should it not contain?

issue tracking IDs

pull request numbers

Use the body to explain what and why vs. how

Use the imperative mood in the subject line

it’s a good idea to begin the commit message with a single short (less than 50 character) line summarizing the change, followed by a blank line and then a more thorough description.

forces the author to think for a moment about the most concise way to explain what’s going on.

shoot for 50 characters, but consider 72 the hard limit

Imperative mood just means “spoken or written as if giving a command or instruction”.

Git itself uses the imperative whenever it creates a commit on your behalf.

when you write your commit messages in the imperative, you’re following Git’s own built-in conventions.

A properly formed Git commit subject line should always be able to complete the following sentence: If applied, this commit will your subject line here

explaining what changed and why

Code is generally self-explanatory in this regard (and if the code is so complex that it needs to be explained in prose, that’s what source comments are for).

there are tab completion scripts that take much of the pain out of remembering the subcommands and switches.

learn about is SSHKit and the various methods it provides

SSHKit was actually developed and released with Capistrano 3, and it’s basically a lower-level tool that provides methods for connecting and interacting with remote servers

on(): specifies the server to run on

within(): specifies the directory path to run in

with(): specifies the environment variables to run with

run on the application server

within the path specified

with certain environment variables set

execute(): the workhorse that runs the commands on your server

upload(): uploads a file from your local computer to your remote server

capture(): executes a command and returns its output as a string

upload() has the bang symbol (!) because that’s how it’s defined in SSHKit, and it’s just a convention letting us know that the method will block until it finishes.

I recommend you take a look at SSHKit’s example page to learn more

make sure we pushed all our local changes to the remote master branch

run this task before Capistrano runs its own deploy task

actually creates three separate tasks

I created a namespace called deploy to contain these tasks since that’s what they’re related to.

we’re using the callbacks inside a namespace to make sure Capistrano knows which tasks the callbacks are referencing.

custom recipe (a Capistrano term meaning a series of tasks)

/shared: holds files and directories that persist throughout deploys

When you run cap production deploy, you’re actually calling a Capistrano task called deploy, which then sequentially invokes other tasks

Deployment is hard and takes a while to sink in.

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

The generate attribute is used to inform Terragrunt to generate the Terraform code for configuring the backend.

The find_in_parent_folders() helper will automatically search up the directory tree to find the root terragrunt.hcl and inherit the remote_state configuration from it.

Unlike the backend configurations, provider configurations support variables,

if you needed to modify the configuration to expose another parameter (e.g session_name), you would have to then go through each of your modules to make this change.

instructs Terragrunt to create the file provider.tf in the working directory (where Terragrunt calls terraform) before it calls any of the Terraform commands

large modules should be considered harmful.

Large modules are slow, insecure, hard to update, hard to code review, hard to test, and brittle (i.e., you have all your eggs in one basket).

Terragrunt allows you to define your Terraform code once and to promote a versioned, immutable “artifact” of that exact same code from environment to environment.

Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure.

a VE there is no preloaded emulation manager software as in a VM.

LXC will boast bare metal performance characteristics because it only packages the needed applications.

a VM, which packages the entire OS and machine setup, including hard drive, virtual processors and network interfaces. The resulting bloated mass usually takes a long time to boot and consumes a lot of CPU and RAM.

don’t offer some other neat features of VM’s such as IaaS setups and live migration.

LXC as supercharged chroot on Linux. It allows you to not only isolate applications, but even the entire OS.

Libvirt, which allows the use of containers through the LXC driver by connecting to 'lxc:///'.

'LXC', is not compatible with libvirt, but is more flexible with more userspace tools.

Portable deployment across machines

Versioning: Docker includes git-like capabilities for tracking successive versions of a container

Component reuse: Docker allows building or stacking of already created packages.

Shared libraries: There is already a public registry (http://index.docker.io/ ) where thousands have already uploaded the useful containers they have created.

Docker taking the devops world by storm since its launch back in 2013.

LXC, while older, has not been as popular with developers as Docker has proven to be

LXC having a focus on sys admins that’s similar to what solutions like the Solaris operating system, with its Solaris Zones, Linux OpenVZ, and FreeBSD, with its BSD Jails virtualization system

it started out being built on top of LXC, Docker later moved beyond LXC containers to its own execution environment called libcontainer.

LXC tooling sticks close to what system administrators running bare metal servers are used to

The LXC command line provides essential commands that cover routine management tasks, including the creation, launch, and deletion of LXC containers.

Docker containers aim to be even lighter weight in order to support the fast, highly scalable, deployment of applications with microservice architecture.

With backing from Canonical, LXC and LXD have an ecosystem tightly bound to the rest of the open source Linux community.

Docker Swarm

Docker Trusted Registry

Docker Compose

Docker Machine

Kubernetes facilitates the deployment of containers in your data center by representing a cluster of servers as a single system.

Swarm is Docker’s clustering, scheduling and orchestration tool for managing a cluster of Docker hosts. 

rkt is a security minded container engine that uses KVM for VM-based isolation and packs other enhanced security features. 

Apache Mesos can run different kinds of distributed jobs, including containers. 

Elastic Container Service is Amazon’s service for running and orchestrating containerized applications on AWS

LXC offers the advantages of a VE on Linux, mainly the ability to isolate your own private workloads from one another. It is a cheaper and faster solution to implement than a VM, but doing so requires a bit of extra learning and expertise.

for a lot of people, the name “Docker” itself is synonymous with the word “container”.

Docker created a very ergonomic (nice-to-use) tool for working with containers – also called docker.

docker is designed to be installed on a workstation or server and comes with a bunch of tools to make it easy to build and run containers as a developer, or DevOps person.

containerd: This is a daemon process that manages and runs containers.

runc: This is the low-level container runtime (the thing that actually creates and runs containers).

Kubernetes includes a component called dockershim, which allows it to support Docker.

Kubernetes will remove support for Docker directly, and prefer to use only container runtimes that implement its Container Runtime Interface.

Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.

Docker images, are actually images packaged in the Open Container Initiative (OCI) format.

CRI is the API that Kubernetes uses to control the different runtimes that create and manage containers.

CRI makes it easier for Kubernetes to use different container runtimes

containerd is a high-level container runtime that came from Docker, and implements the CRI spec

containerd was separated out of the Docker project, to make Docker more modular.

CRI-O is another high-level container runtime which implements the Container Runtime Interface (CRI).

The idea behind the OCI is that you can choose between different runtimes which conform to the spec.

runc is an OCI-compatible container runtime.

A reference implementation is a piece of software that has implemented all the requirements of a specification or standard.

runc provides all of the low-level functionality for containers, interacting with existing low-level Linux features, like namespaces and control groups.

This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.

differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.

Filebeat is just a tiny binary with no dependencies.

For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

For example, the apache module will point Filebeat to default access.log and error.log paths

Filebeat’s scope is very limited,

Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Filebeat can parse JSON

you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.

You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off

For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well

The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.

rsyslog is the fastest shipper