Group items tagged

Kubernetes is fully controlled through this API

the main job of kubectl is to carry out HTTP requests to the Kubernetes API

Kubernetes maintains an internal state of resources, and all Kubernetes operations are CRUD operations on these resources.

Kubernetes is a fully resource-centred system

Kubernetes API reference is organised as a list of resource types with their associated operations.

This is how kubectl works for all commands that interact with the Kubernetes cluster.

kubectl simply makes HTTP requests to the appropriate Kubernetes API endpoints.

it's totally possible to control Kubernetes with a tool like curl by manually issuing HTTP requests to the Kubernetes API.

Kubernetes consists of a set of independent components that run as separate processes on the nodes of a cluster.

components on the master nodes

Storage backend: stores resource definitions (usually etcd is used)

API server: provides Kubernetes API and manages storage backend

Controller manager: ensures resource statuses match specifications

Scheduler: schedules Pods to worker nodes

component on the worker nodes

Kubelet: manages execution of containers on a worker node

triggers the ReplicaSet controller, which is a sub-process of the controller manager.

the scheduler, who watches for Pod definitions that are not yet scheduled to a worker node.

creating and updating resources in the storage backend on the master node.

However, these components do not access the storage backend directly, but only through the Kubernetes API.

All other Kubernetes components and users read, watch, and manipulate the state (i.e. resources) of Kubernetes through the Kubernetes API

The storage backend stores the state (i.e. resources) of Kubernetes.

command completion is a shell feature that works by the means of a completion script.

A completion script is a shell script that defines the completion behaviour for a specific command. Sourcing a completion script enables completion for the corresponding command.

kubectl completion zsh

/etc/bash_completion.d directory (create it, if it doesn't exist)

source <(kubectl completion bash)

source <(kubectl completion zsh)

autoload -Uz compinit compinit

the API reference, which contains the full specifications of all resources.

kubectl api-resources

displays the resource names in their plural form (e.g. deployments instead of deployment). It also displays the shortname (e.g. deploy) for those resources that have one. Don't worry about these differences. All of these name variants are equivalent for kubectl.

.spec

custom columns output format comes in. It lets you freely define the columns and the data to display in them. You can choose any field of a resource to be displayed as a separate column in the output

kubectl get pods -o custom-columns='NAME:metadata.name,NODE:spec.nodeName'

kubectl explain pod.spec.

kubectl explain pod.metadata.

browse the resource specifications and try it out with any fields you like!

with kubectl explain, only a subset of the JSONPath capabilities is supported

Many fields of Kubernetes resources are lists, and this operator allows you to select items of these lists. It is often used with a wildcard as [*] to select all items of the list.

kubectl get pods -o custom-columns='NAME:metadata.name,IMAGES:spec.containers[*].image'

a Pod may contain more than one container.

The availability zones for each node are obtained through the special failure-domain.beta.kubernetes.io/zone label.

kubectl get nodes -o yaml kubectl get nodes -o json

The default kubeconfig file is ~/.kube/config

with multiple clusters, then you have connection parameters for multiple clusters configured in your kubeconfig file.

overwrite the default kubeconfig file with the --kubeconfig option for every kubectl command.

Namespace: the namespace to use when connecting to the cluster

a one-to-one mapping between clusters and contexts.

When kubectl reads a kubeconfig file, it always uses the information from the current context.

just change the current context in the kubeconfig file

kubectl also provides the --cluster, --user, --namespace, and --context options that allow you to overwrite individual elements and the current context itself, regardless of what is set in the kubeconfig file.

for switching between clusters and namespaces is kubectx.

kubectl config get-contexts

just have to download the shell scripts named kubectl-ctx and kubectl-ns to any directory in your PATH and make them executable (for example, with chmod +x)

kubectl proxy

kubectl get roles

kubectl get pod

Kubectl plugins are distributed as simple executable files with a name of the form kubectl-x. The prefix kubectl- is mandatory,

To install a plugin, you just have to copy the kubectl-x file to any directory in your PATH and make it executable (for example, with chmod +x)

krew itself is a kubectl plugin

check out the kubectl-plugins GitHub topic

The executable can be of any type, a Bash script, a compiled Go program, a Python script, it really doesn't matter. The only requirement is that it can be directly executed by the operating system.

kubectl plugins can be written in any programming or scripting language.

you can write more sophisticated plugins with real programming languages, for example, using a Kubernetes client library. If you use Go, you can also use the cli-runtime library, which exists specifically for writing kubectl plugins.

a kubeconfig file consists of a set of contexts

changing the current context means changing the cluster, if you have only a single context per cluster.

Login to the container

Baseimage-docker only advocates running multiple OS processes inside a single container.

Password and challenge-response authentication are disabled by default. Only key authentication is allowed.

A tool for running a command as another user

The Docker developers advocate the philosophy of running a single logical service per container. A logical service can consist of multiple OS processes.

All syslog messages are forwarded to "docker logs".

Baseimage-docker advocates running multiple OS processes inside a single container, and a single logical service can consist of multiple OS processes.

Baseimage-docker provides tools to encourage running processes as different users

sometimes it makes sense to run multiple services in a single container, and sometimes it doesn't.

Splitting your logical service into multiple OS processes also makes sense from a security standpoint.

using environment variables to pass parameters to containers is very much the "Docker way"

the shell script must run the daemon without letting it daemonize/fork it.

All executable scripts in /etc/my_init.d, if this directory exists. The scripts are run in lexicographic order.

variables will also be passed to all child processes

Environment variables on Unix are inherited on a per-process basis

there is no good central place for defining environment variables for all applications and services

centrally defining environment variables

One of the ideas behind Docker is that containers should be stateless, easily restartable, and behave like a black box.

a one-shot command in a new container

immediately exit after the command exits,

add additional daemons (e.g. your own app) to the image by creating runit entries.

Nginx is one such example: it removes all environment variables unless you explicitly instruct it to retain them through the env configuration option.

Mechanisms for easily running multiple processes, without violating the Docker philosophy

Ubuntu is not designed to be run inside Docker

According to the Unix process model, the init process -- PID 1 -- inherits all orphaned child processes and must reap them

Syslog-ng seems to be much more stable

cron daemon

Rotates and compresses logs

/sbin/setuser

A tool for installing apt packages that automatically cleans up after itself.

A daemon is a program which runs in the background of its system, such as a web server.

The shell script must be called run, must be executable, and is to be placed in the directory /etc/service/<NAME>. runsv will switch to the directory and invoke ./run after your container starts.

If any script exits with a non-zero exit code, the booting will fail.

If your process is started with a shell script, make sure you exec the actual process, otherwise the shell will receive the signal and not your process.

any environment variables set with docker run --env or with the ENV command in the Dockerfile, will be picked up by my_init

they will not see the environment variables that were originally passed by Docker.

my_init imports environment variables from the directory /etc/container_environment

/etc/container_environment.sh - a dump of the environment variables in Bash format.

modify the environment variables in my_init (and therefore the environment variables in all child processes that are spawned after that point in time), by altering the files in /etc/container_environment

my_init only activates changes in /etc/container_environment when running startup scripts

environment variables don't contain sensitive data, then you can also relax the permissions

Syslog messages are forwarded to the console

syslog-ng is started separately before the runit supervisor process, and shutdown after runit exits.

/sbin/my_init --skip-startup-files --quiet --

By default, no keys are installed, so nobody can login

provide a pregenerated, insecure key (PuTTY format)

RUN /usr/sbin/enable_insecure_key

docker run YOUR_IMAGE /sbin/my_init --enable-insecure-key

RUN cat /tmp/your_key.pub >> /root/.ssh/authorized_keys && rm -f /tmp/your_key.pub

The default baseimage-docker installs syslog-ng, cron and sshd services during the build process

$?: Most Recent Exit Code

When you quote $*, it will output all of the arguments received as one single string, separated by a space1 regardless of how they were quoted going in, but it will quote that string so that it doesn't get split up later.

Use COPY and RUN commands in proper order

Merge multiple RUN commands into one

alpine versions should be enough

problems with zombie processes

prepare separate Docker image for each component, and use Docker Compose to easily start multiple containers at the same time

Layers are cached and reused

Layers are immutable

rely on our base image updates

make a cleanup

alpine is a very tiny linux distribution, just about 4 MB in size.

CMD is a default command run after creating container without other command specified.

put your command inside array

entrypoint adds complexity

Entrypoint is a script, that will be run instead of command, and receive command as arguments

Without it, we would not be able to stop our application grecefully (SIGTERM is swallowed by bash script).

ADD has some logic for downloading remote files and extracting archives.

stick with COPY.

ADD

checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

instead of deploying your application manually, you set it to be deployed automatically.

.gitlab-ci.yml, located in the root path of your repository

all the scripts you add to the configuration file are the same as the commands you run on a terminal in your computer.

GitLab will detect it and run your scripts with the tool called GitLab Runner, which works similarly to your terminal.

SysBench is not a tool which you can use to tune configurations of your MySQL servers (unless you prepared LUA scripts with custom workload or your workload happen to be very similar to the benchmark workloads that SysBench comes with)

it is great for is to compare performance of different hardware.

Every new server acquired should go through a warm-up period during which you will stress it to pinpoint potential hardware defects

by executing OLTP workload which overloads the server, or you can also use dedicated benchmarks for CPU, disk and memory.

bulk_insert.lua. This test can be used to benchmark the ability of MySQL to perform multi-row inserts.

All oltp_* scripts share a common table structure. First two of them (oltp_delete.lua and oltp_insert.lua) execute single DELETE and INSERT statements.

oltp_point_select, oltp_update_index and oltp_update_non_index. These will execute a subset of queries - primary key-based selects, index-based updates and non-index-based updates.

you can run different workload patterns using the same benchmark.

Warmup helps to identify “regular” throughput by executing benchmark for a predefined time, allowing to warm up the cache, buffer pools etc.

By default SysBench will attempt to execute queries as fast as possible. To simulate slower traffic this option may be used. You can define here how many transactions should be executed per second.

SysBench gives you ability to generate different types of data distribution.

decide if SysBench should use prepared statements (as long as they are available in the given datastore - for MySQL it means PS will be enabled by default) or not.

sysbench ./sysbench/src/lua/oltp_read_write.lua  help

By default, SysBench will attempt to execute queries in explicit transaction. This way the dataset will stay consistent and not affected: SysBench will, for example, execute INSERT and DELETE on the same row, making sure the data set will not grow (impacting your ability to reproduce results).

specify error codes from MySQL which SysBench should ignore (and not kill the connection).

the two most popular benchmarks - OLTP read only and OLTP read/write.

1 million rows will result in ~240 MB of data. Ten tables, 1000 000 rows each equals to 2.4GB

by default, SysBench looks for ‘sbtest’ schema which has to exist before you prepare the data set. You may have to create it manually.

pass ‘--histogram’ argument to SysBench

~48GB of data (20 tables, 10 000 000 rows each).

if you don’t understand why the performance was like it was, you may draw incorrect conclusions out of the benchmarks.

needs to run before any target

If there’s a target for makefile, and its prerequisites are new, the target will run before anything, because the makefile might change.

containers: directories containing everything-your-application

images: snapshots of containers or base OS (e.g. Ubuntu) images

Dockerfiles: scripts automating the building process of images

Docker containers are basically directories which can be packed (e.g. tar-archived) like any other, then shared and run across various different machines and platforms (hosts).

Linux Containers can be defined as a combination various kernel-level features (i.e. things that Linux-kernel can do) which allow management of applications (and resources they use) contained within their own environment

Each container is layered like an onion and each action taken within a container consists of putting another block (which actually translates to a simple change within the file system) on top of the previous one.

Each docker container starts from a docker image which forms the base for other applications and layers to come.

Docker images constitute the base of docker containers from which everything starts to form

a solid, consistent and dependable base with everything that is needed to run the applications

As more layers (tools, applications etc.) are added on top of the base, new images can be formed by committing these changes.

a Dockerfile for automated image building

Dockerfiles are scripts containing a successive series of instructions, directions, and commands which are to be executed to form a new docker image.

As you work with a container and continue to perform actions on it (e.g. download and install software, configure files etc.), to have it keep its state, you need to “commit”.

Please remember to “commit” all your changes.

When you "run" any process using an image, in return, you will have a container.

When the process is not actively running, this container will be a non-running container. Nonetheless, all of them will reside on your system until you remove them via rm command.

To create a new container, you need to use a base image and specify a command to run.

you can not change the command you run after having created a container (hence specifying one during "creation")

If you would like to save the progress and changes you made with a container, you can use “commit”

turns your container to an image

A proper Unix system should run all kinds of important system services.

Ubuntu is not designed to be run inside Docker

When a system is started, the first process in the system is called the init process, with PID 1. The system halts when this processs halts.

Runit (written in C) is much lighter weight than supervisord (written in Python).

Docker runs fine with multiple processes in a container.

Baseimage-docker encourages you to run multiple processes through the use of runit.

If your init process is your app, then it'll probably only shut down itself, not all the other processes in the container.

a Docker container, which is a locked down environment with e.g. no direct access to many kernel resources.

Used for service supervision and management.

A custom tool for running a command as another user.

add additional daemons (e.g. your own app) to the image by creating runit entries.

write a small shell script which runs your daemon, and runit will keep it up and running for you, restarting it when it crashes, etc.

the shell script must run the daemon without letting it daemonize/fork it.

A client application is connected to the primary via MySQL Router

MySQL Shell also requires Python 2.7 and above to run cluster provisioning scripts

AdminAPI, which enables you to create and administer an InnoDB cluster, using either JavaScript or Python scripting

Caches the metadata of the InnoDB cluster and performs high availability routing to the MySQL Server instances which make up the cluster

Group Replication mechanism to allow data to be replicated from the primary to the secondaries in the cluster

AdminAPI is available as of MySQL Shell 1.0.8.

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

Infrastructure as Code takes advantage of the software development process, making use of quality assurance and test automation techniques.

Consistency/Standardization

With standard configuration files and software-based configuration, there is greater consistency between all equipment of the same type. A key IaC concept is idempotence.

Idempotence makes it easy to troubleshoot, test, stabilize, and upgrade all the equipment.

Infrastructure as Code is central to the culture of DevOps, which is a mix of development and operations

A declarative approach describes the final state of a device, but does not mandate how it should get there. The specific IaC tool makes all the procedural decisions. The end state is typically defined through a configuration file, a JSON specification, or a similar encoding.

An imperative approach defines specific functions or procedures that must be used to configure the device. It focuses on what must happen, but does not necessarily describe the final state. Imperative techniques typically use scripts for the implementation.

With a push configuration, the central server pushes the configuration to the destination device.

If a device is mutable, its configuration can be changed while it is active

an immutable approach ensures consistency and avoids drift. However, it usually takes more time to remove or rebuild a configuration than it does to change it.

System administrators should consider security issues as part of the development process.

Ansible is a very popular open source IaC application from Red Hat

Ansible is often used in conjunction with Kubernetes and Docker.

Linode offers a collection of several Ansible guides for a more comprehensive overview.

Pulumi permits the use of a variety of programming languages to deploy and manage infrastructure within a cloud environment.

Terraform allows users to provision data center infrastructure using either JSON or Terraform’s own declarative language.

Terraform manages resources through the use of providers, which are similar to APIs.

Maps must contain at least the name option, which is the same image name as used for the string setting.

The runner starts a Docker container using the defined entrypoint. The default from Dockerfile that may be overridden in the .gitlab-ci.yml file.

sends the script to the container’s shell stdin and receives the output.

entrypoint: [""]

entrypoint: ["/bin/sh", "-c"]

A DOCKER_AUTH_CONFIG CI/CD variable

A build system, git, and development headers for many popular libraries, so that the most popular Ruby, Python and Node.js native extensions can be compiled without problems.

Nginx 1.18. Disabled by default

production-grade features, such as process monitoring, administration and status inspection.

Redis 5.0. Not installed by default.

The image has an app user with UID 9999 and home directory /home/app. Your application is supposed to run as this user.

Passenger works like a mod_ruby, mod_nodejs, etc. It changes Nginx into an application server and runs your app from Nginx.

placing a .conf file in the directory /etc/nginx/sites-enabled

The best way to configure Nginx is by adding .conf files to /etc/nginx/main.d and /etc/nginx/conf.d

To preserve these variables, place an Nginx config file ending with *.conf in the directory /etc/nginx/main.d, in which you tell Nginx to preserve these variables.

By default, Phusion Passenger sets all of the following environment variables to the value production

PASSENGER_APP_ENV environment variable

passenger-docker autogenerates an Nginx configuration file (/etc/nginx/conf.d/00_app_env.conf) during container boot.

The configuration file is in /etc/redis/redis.conf. Modify it as you see fit, but make sure daemonize no is set.

You can add additional daemons to the image by creating runit entries.

the shell script must run the daemon without letting it daemonize/fork it.

We use RVM to install and to manage Ruby interpreters.