Group items tagged

While you may get Bob and Lola, and someone else may get Stan and Amy, in fact they are both sending requests to the same elastic pool of resources.

ipa dnszone-mod example.com. --update-policy="grant keyname name example.com A;"

ipa dnszone-mod example.com. --dynamic-update=1

initialize the local CLI

install Tiller into your Kubernetes cluster

helm install

helm init --upgrade

By default, when Tiller is installed, it does not have authentication enabled.

helm repo update

Without a max history set the history is kept indefinitely, leaving a large number of records for helm and tiller to maintain.

helm init --upgrade

Whenever you install a chart, a new release is created.

helm list function will show you a list of all deployed releases.

helm delete

helm status

the Helm server (Tiller).

The Helm client (helm)

brew install kubernetes-helm

it can also be run locally, and configured to talk to a remote Kubernetes cluster.

Role-Based Access Control - RBAC for short

run Tiller in an RBAC-enabled Kubernetes cluster.

run kubectl get pods --namespace kube-system and see Tiller running.

helm inspect

Helm will look for Tiller in the kube-system namespace unless --tiller-namespace or TILLER_NAMESPACE is set.

For development, it is sometimes easier to work on Tiller locally, and configure it to connect to a remote Kubernetes cluster.

helm version should show you both the client and server version.

The --node-selectors flag allows us to specify the node labels required for scheduling the Tiller pod.

--override allows you to specify properties of Tiller’s deployment manifest.

helm init --override manipulates the specified properties of the final manifest (there is no “values” file).

The --output flag allows us skip the installation of Tiller’s deployment manifest and simply output the deployment manifest to stdout in either JSON or YAML format.

switch from the default backend to the secrets backend, you’ll have to do the migration for this on your own.

a beta SQL storage backend that stores release information in an SQL database (only postgres has been tested so far).

Helm requires that kubelet have access to a copy of the socat program to proxy connections to the Tiller API.

A Release is an instance of a chart running in a Kubernetes cluster. One chart can often be installed many times into the same cluster.

helm init --client-only

helm init --dry-run --debug

A panic in Tiller is almost always the result of a failure to negotiate with the Kubernetes API server

Tiller and Helm have to negotiate a common version to make sure that they can safely communicate without breaking API assumptions

helm delete --purge

Helm stores some files in $HELM_HOME, which is located by default in ~/.helm

A Chart is a Helm package. It contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster.

A Repository is the place where charts can be collected and shared.

Set the $HELM_HOME environment variable

Helm installs charts into Kubernetes, creating a new release for each installation. And to find new charts, you can search Helm chart repositories.

chart repository is named stable by default

helm search shows you all of the available charts

helm inspect

To install a new package, use the helm install command. At its simplest, it takes only one argument: The name of the chart.

If you want to use your own release name, simply use the --name flag on helm install

additional configuration steps you can or should take.

Helm does not wait until all of the resources are running before it exits. Many charts require Docker images that are over 600M in size, and may take a long time to install into the cluster.

helm status

helm inspect values

helm inspect values stable/mariadb

override any of these settings in a YAML formatted file, and then pass that file during installation.

helm install -f config.yaml stable/mariadb

--values (or -f): Specify a YAML file with overrides.

--set (and its variants --set-string and --set-file): Specify overrides on the command line.

Values that have been --set can be cleared by running helm upgrade with --reset-values specified.

Chart designers are encouraged to consider the --set usage when designing the format of a values.yaml file.

--set-file key=filepath is another variant of --set. It reads the file and use its content as a value.

inject a multi-line text into values without dealing with indentation in YAML.

An unpacked chart directory

When a new version of a chart is released, or when you want to change the configuration of your release, you can use the helm upgrade command.

Kubernetes charts can be large and complex, Helm tries to perform the least invasive upgrade.

$ helm upgrade -f panda.yaml happy-panda stable/mariadb

deployment

If both are used, --set values are merged into --values with higher precedence.

The helm get command is a useful tool for looking at a release in the cluster.

A release version is an incremental revision. Every time an install, upgrade, or rollback happens, the revision number is incremented by 1.

helm history

you can rollback a deleted resource, and have it re-activate.

helm repo list

helm repo add

helm repo update

helm create

helm lint

helm package

Charts that are archived can be loaded into chart repositories.

chart repository server

Tiller can be installed into any namespace.

Limiting Tiller to only be able to install into specific namespaces and/or resource types is controlled by Kubernetes RBAC roles and rolebindings

Release names are unique PER TILLER INSTANCE

Charts should only contain resources that exist in a single namespace.

not recommended to have multiple Tillers configured to manage resources in the same namespace.

a client-side Helm plugin. A plugin is a tool that can be accessed through the helm CLI, but which is not part of the built-in Helm codebase.

Helm plugins are add-on tools that integrate seamlessly with Helm. They provide a way to extend the core feature set of Helm, but without requiring every new feature to be written in Go and added to the core tool.

Helm plugins live in $(helm home)/plugins

The Helm plugin model is partially modeled on Git’s plugin model

helm referred to as the porcelain layer, with plugins being the plumbing.

helm plugin install https://github.com/technosophos/helm-template

command is the command that this plugin will execute when it is called.

Environment variables are interpolated before the plugin is executed.

The command itself is not executed in a shell. So you can’t oneline a shell script.

Helm is able to fetch Charts using HTTP/S

Variables like KUBECONFIG are set for the plugin if they are set in the outer environment.

In Kubernetes, granting a role to an application-specific service account is a best practice to ensure that your application is operating in the scope that you have specified.

restrict Tiller’s capabilities to install resources to certain namespaces, or to grant a Helm client running access to a Tiller instance.

Service account with cluster-admin role

The cluster-admin role is created by default in a Kubernetes cluster

Deploy Tiller in a namespace, restricted to deploying resources only in that namespace

Deploy Tiller in a namespace, restricted to deploying resources in another namespace

When running a Helm client in a pod, in order for the Helm client to talk to a Tiller instance, it will need certain privileges to be granted.

SSL Between Helm and Tiller

The Tiller authentication model uses client-side SSL certificates.

creating an internal CA, and using both the cryptographic and identity functions of SSL.

Helm is a powerful and flexible package-management and operations tool for Kubernetes.

default installation applies no security configurations

with a cluster that is well-secured in a private network with no data-sharing or no other users or teams.

With great power comes great responsibility.

Choose the Best Practices you should apply to your helm installation

Role-based access control, or RBAC

Tiller’s gRPC endpoint and its usage by Helm

Kubernetes employ a role-based access control (or RBAC) system (as do modern operating systems) to help mitigate the damage that can be done if credentials are misused or bugs exist.

In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied.

Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.

release information

charts

charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.

Helm’s provenance tools to ensure the provenance and integrity of charts

DNS zones can be on the same servers

Master zones, contain a read/write copy of the zone data.

If you want to have redundancy, you must have the zone data accessible on multiple servers.

The Slave zone is a read-only copy of the zone data.

Most of the times Slave DNS zones are copies of Master zones.

the primary purposes of a Slave zone is to serve as a backup

With respect to permissions, all users on the system except those with a user role of No Access have read access to objects in partition Common, and by default, partition Common is their current partition.

The current partition is the specific partition to which the system is currently set for a logged-in user.

A partition access assignment gives a user some level of access to the specified partition.

assigning partition access to a user does not necessarily give the user full access to all objects in the partition

user account objects also reside in partitions

when you first install the BIG-IP system, every existing user account (root and admin) resides in partition Common

the partition in which a user account object resides does not affect the partition or partitions to which that user is granted access to manage other BIG-IP objects

the object it references resides in partition Common

a referenced object must reside either in the same partition as the object that is referencing it

Nginx is built to handle many concurrent connections at the same time.

provides you with flexibility in easily adding backend servers or taking them down as needed for maintenance

Proxying in Nginx is accomplished by manipulating a request aimed at the Nginx server and passing it to other servers for the actual processing

When a request matches a location with a proxy_pass directive inside, the request is forwarded to the URL given by the directive

The request coming from Nginx on behalf of a client will look different than a request coming directly from a client

Nginx gets rid of any empty headers

Nginx, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request

The "Host" header is re-written to the value defined by the $proxy_host variable.

The upstream should not expect this connection to be persistent

Headers with empty values are completely removed from the passed request.

by default, this will be set to the value of $proxy_host, a variable that will contain the domain name or IP address and port taken directly from the proxy_pass definition

This is selected by default as it is the only address Nginx can be sure the upstream server responds to

$http_host: Sets the "Host" header to the "Host" header from the client request.

preference to: the host name from the request line itself

set the "Host" header to the $host variable. It is the most flexible and will usually provide the proxied servers with a "Host" header filled in as accurately as possible

This variable takes the value of the original X-Forwarded-For header retrieved from the client and adds the Nginx server's IP address to the end.

Once defined, this name will be available for use within proxy passes as if it were a regular domain name

By default, this is just a simple round-robin selection process (each request will be routed to a different host in turn)

Specifies that new connections should always be given to the backend that has the least number of active connections.

distributes requests to different servers based on the client's IP address.

mainly used with memcached proxying

As for the hash method, you must provide the key to hash against

Server Weight

Nginx's buffering and caching capabilities

Without buffers, data is sent from the proxied server and immediately begins to be transmitted to the client.

With buffers, the Nginx proxy will temporarily store the backend's response and then feed this data to the client

Nginx defaults to a buffering design

can be set in the http, server, or location contexts.

the sizing directives are configured per request, so increasing them beyond your need can affect your performance

When buffering is "off" only the buffer defined by the proxy_buffer_size directive will be used

A high availability (HA) setup is an infrastructure without a single point of failure, and your load balancers are a part of this configuration.

multiple load balancers (one active and one or more passive) behind a static IP address that can be remapped from one server to another.

Nginx also provides a way to cache content from backend servers

proxy_cache backcache;

The proxy_cache_bypass directive is set to the $http_cache_control variable. This will contain an indicator as to whether the client is explicitly requesting a fresh, non-cached version of the resource

For private content, you should set the Cache-Control header to "no-cache", "no-store", or "private" depending on the nature of the data

the ultimate challenge is to fuse the logic of the database and data replication with the logic of having several servers coordinated in a consistent and simple way

MySQL Group Replication provides distributed state machine replication with strong coordination between servers.

For a transaction to commit, the majority of the group have to agree on the order of a given transaction in the global sequence of transactions

group communication protocols

the Paxos algorithm. It acts as the group communication systems engine.

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

Infrastructure as Code takes advantage of the software development process, making use of quality assurance and test automation techniques.

Consistency/Standardization

With standard configuration files and software-based configuration, there is greater consistency between all equipment of the same type. A key IaC concept is idempotence.

Idempotence makes it easy to troubleshoot, test, stabilize, and upgrade all the equipment.

Infrastructure as Code is central to the culture of DevOps, which is a mix of development and operations

A declarative approach describes the final state of a device, but does not mandate how it should get there. The specific IaC tool makes all the procedural decisions. The end state is typically defined through a configuration file, a JSON specification, or a similar encoding.

An imperative approach defines specific functions or procedures that must be used to configure the device. It focuses on what must happen, but does not necessarily describe the final state. Imperative techniques typically use scripts for the implementation.

With a push configuration, the central server pushes the configuration to the destination device.

If a device is mutable, its configuration can be changed while it is active

an immutable approach ensures consistency and avoids drift. However, it usually takes more time to remove or rebuild a configuration than it does to change it.

System administrators should consider security issues as part of the development process.

Ansible is a very popular open source IaC application from Red Hat

Ansible is often used in conjunction with Kubernetes and Docker.

Linode offers a collection of several Ansible guides for a more comprehensive overview.

Pulumi permits the use of a variety of programming languages to deploy and manage infrastructure within a cloud environment.

Terraform allows users to provision data center infrastructure using either JSON or Terraform’s own declarative language.

Terraform manages resources through the use of providers, which are similar to APIs.

从集群中获取每个 pod ip 地址，然后也能在集群内部直接通过 podIP:Port 来获取对应的服务。

pod 是经常变化的，每次更新 ip 地址都可能会发生变化，如果直接访问容器 ip 的话，会有很大的问题。

“服务”（service），每个服务都一个固定的虚拟 ip（这个 ip 也被称为 cluster IP），自动并且动态地绑定后面的 pod，所有的网络请求直接访问服务 ip，服务会自动向后端做转发。

实现 service 这一功能的关键，就是 kube-proxy。

kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1

iptables 完全实现 iptables 来实现 service，是目前默认的方式，也是推荐的方式，效率很高（只有内核中 netfilter 一些损耗）。

可以在终端上启动 kube-proxy，也可以使用诸如 systemd 这样的工具来管理它

Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure.

a VE there is no preloaded emulation manager software as in a VM.

LXC will boast bare metal performance characteristics because it only packages the needed applications.

a VM, which packages the entire OS and machine setup, including hard drive, virtual processors and network interfaces. The resulting bloated mass usually takes a long time to boot and consumes a lot of CPU and RAM.

don’t offer some other neat features of VM’s such as IaaS setups and live migration.

LXC as supercharged chroot on Linux. It allows you to not only isolate applications, but even the entire OS.

Libvirt, which allows the use of containers through the LXC driver by connecting to 'lxc:///'.

'LXC', is not compatible with libvirt, but is more flexible with more userspace tools.

Portable deployment across machines

Versioning: Docker includes git-like capabilities for tracking successive versions of a container

Component reuse: Docker allows building or stacking of already created packages.

Shared libraries: There is already a public registry (http://index.docker.io/ ) where thousands have already uploaded the useful containers they have created.

Docker taking the devops world by storm since its launch back in 2013.

LXC, while older, has not been as popular with developers as Docker has proven to be

LXC having a focus on sys admins that’s similar to what solutions like the Solaris operating system, with its Solaris Zones, Linux OpenVZ, and FreeBSD, with its BSD Jails virtualization system

it started out being built on top of LXC, Docker later moved beyond LXC containers to its own execution environment called libcontainer.

LXC tooling sticks close to what system administrators running bare metal servers are used to

The LXC command line provides essential commands that cover routine management tasks, including the creation, launch, and deletion of LXC containers.

Docker containers aim to be even lighter weight in order to support the fast, highly scalable, deployment of applications with microservice architecture.

With backing from Canonical, LXC and LXD have an ecosystem tightly bound to the rest of the open source Linux community.

Docker Swarm

Docker Trusted Registry

Docker Compose

Docker Machine

Kubernetes facilitates the deployment of containers in your data center by representing a cluster of servers as a single system.

Swarm is Docker’s clustering, scheduling and orchestration tool for managing a cluster of Docker hosts. 

rkt is a security minded container engine that uses KVM for VM-based isolation and packs other enhanced security features. 

Apache Mesos can run different kinds of distributed jobs, including containers. 

Elastic Container Service is Amazon’s service for running and orchestrating containerized applications on AWS

LXC offers the advantages of a VE on Linux, mainly the ability to isolate your own private workloads from one another. It is a cheaper and faster solution to implement than a VM, but doing so requires a bit of extra learning and expertise.

the containers within a pod can communicate with each other over localhost

we can never create a standalone container: the closest we can do is create a pod, with a single container in it.

Kubernetes is a declarative system (by opposition to imperative systems).

All we can do, is describe what we want to have, and wait for Kubernetes to take action to reconcile what we have, with what we want to have.

The specification of a replica set looks very much like the specification of a pod, except that it carries a number, indicating how many replicas

What happens if we change that definition? Suddenly, there are zero pods matching the new specification.

the creation of new pods could happen in a more gradual manner.

the specification for a deployment looks very much like the one for a replica set: it features a pod specification, and a number of replicas.

Deployments, however, don’t create or delete pods directly.

When we update a deployment and adjust the number of replicas, it passes that update down to the replica set.

When we update the pod specification, the deployment creates a new replica set with the updated pod specification. That replica set has an initial size of zero. Then, the size of that replica set is progressively increased, while decreasing the size of the other replica set.

we are going to fade in (turn up the volume) on the new replica set, while we fade out (turn down the volume) on the old one.

A readiness probe is a test that we add to a container specification.

Kubernetes supports three ways of implementing readiness probes:Running a command inside a container;Making an HTTP(S) request against a container; orOpening a TCP socket against a container.

When we roll out a new version, Kubernetes will wait for the new pod to mark itself as “ready” before moving on to the next one.

MaxSurge indicates how many extra pods we are willing to run during a rolling update, while MaxUnavailable indicates how many pods we can lose during the rolling update.

Setting MaxUnavailable to 0 means, “do not shutdown any old pod before a new one is up and ready to serve traffic“.

Setting MaxSurge to 100% means, “immediately start all the new pods“, implying that we have enough spare capacity on our cluster, and that we want to go as fast as possible.

the replica set doesn’t look at the pods’ specifications, but only at their labels.

A replica set contains a selector, which is a logical expression that “selects” (just like a SELECT query in SQL) a number of pods.

Selectors are also used by services, which act as the load balancers for Kubernetes traffic, internal and external.

during a rollout, the deployment doesn’t reconfigure or inform the load balancer that pods are started and stopped. It happens automatically through the selector of the service associated to the load balancer.

In blue/green deployment, we want to instantly switch over all the traffic from the old version to the new, instead of doing it progressively

We can achieve blue/green deployment by creating multiple deployments (in the Kubernetes sense), and then switching from one to another by changing the selector of our service

kubectl label pods -l app=blue,version=v1.5 status=enabled

kubectl label pods -l app=blue,version=v1.4 status-

deployment.yaml: A basic manifest for creating a Kubernetes deployment

using the suffix .yaml for YAML files and .tpl for helpers.

helm get manifest

The helm get manifest command takes a release name (full-coral) and prints out all of the Kubernetes resources that were uploaded to the server. Each file begins with --- to indicate the start of a YAML document

Names should be unique to a release

The name: field is limited to 63 characters because of limitations to the DNS system.

release names are limited to 53 characters

{{ .Release.Name }}

The values that are passed into a template can be thought of as namespaced objects, where a dot (.) separates each namespaced element.

The Release object is one of the built-in objects for Helm

Using --dry-run will make it easier to test your code, but it won’t ensure that Kubernetes itself will accept the templates you generate.

Objects are passed into a template from the template engine.

create new objects within your templates

Objects can be simple, and have just one value. Or they can contain other objects or functions.

Release is one of the top-level objects that you can access in your templates.

Release.Namespace: The namespace to be released into (if the manifest doesn’t override)

Chart: The contents of the Chart.yaml file.

Files: This provides access to all non-special files in a chart.

Files.Get is a function for getting a file by name

Files.GetBytes is a function for getting the contents of a file as an array of bytes instead of as a string. This is useful for things like images.

Template: Contains information about the current template that is being executed

BasePath: The namespaced path to the templates directory of the current chart

Go’s naming convention

use only initial lower case letters in order to distinguish local names from those built-in.

If this is a subchart, the values.yaml file of a parent chart

Individual parameters passed with --set

While structuring data this way is possible, the recommendation is that you keep your values trees shallow, favoring flatness.

If you need to delete a key from the default values, you may override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.

Kubernetes would then fail because you can not declare more than one livenessProbe handler.