Group items tagged

Once you have a cluster running and you want to add/reconnect another node to it, you must supply an address of one of the cluster members in the cluster address URL

It will automatically retrieve the cluster map and reconnect to the rest of the nodes

it's better to list all nodes of the cluster so that any node can join a cluster connecting to any other node, even if one or more are down

The wsrep_cluster_address parameter should be added in my.cnf of each node, listing all the nodes of the cluster,

the minimum recommended number of nodes in a cluster is 3

While two of the members will be engaged in state transfer, the remaining member(s) will be able to keep on serving client requests.

initialize the local CLI

install Tiller into your Kubernetes cluster

helm install

helm init --upgrade

By default, when Tiller is installed, it does not have authentication enabled.

helm repo update

Without a max history set the history is kept indefinitely, leaving a large number of records for helm and tiller to maintain.

helm init --upgrade

Whenever you install a chart, a new release is created.

helm list function will show you a list of all deployed releases.

helm delete

helm status

the Helm server (Tiller).

The Helm client (helm)

brew install kubernetes-helm

it can also be run locally, and configured to talk to a remote Kubernetes cluster.

Role-Based Access Control - RBAC for short

run Tiller in an RBAC-enabled Kubernetes cluster.

run kubectl get pods --namespace kube-system and see Tiller running.

helm inspect

Helm will look for Tiller in the kube-system namespace unless --tiller-namespace or TILLER_NAMESPACE is set.

For development, it is sometimes easier to work on Tiller locally, and configure it to connect to a remote Kubernetes cluster.

helm version should show you both the client and server version.

The --node-selectors flag allows us to specify the node labels required for scheduling the Tiller pod.

--override allows you to specify properties of Tiller’s deployment manifest.

helm init --override manipulates the specified properties of the final manifest (there is no “values” file).

The --output flag allows us skip the installation of Tiller’s deployment manifest and simply output the deployment manifest to stdout in either JSON or YAML format.

switch from the default backend to the secrets backend, you’ll have to do the migration for this on your own.

a beta SQL storage backend that stores release information in an SQL database (only postgres has been tested so far).

Helm requires that kubelet have access to a copy of the socat program to proxy connections to the Tiller API.

A Release is an instance of a chart running in a Kubernetes cluster. One chart can often be installed many times into the same cluster.

helm init --client-only

helm init --dry-run --debug

A panic in Tiller is almost always the result of a failure to negotiate with the Kubernetes API server

Tiller and Helm have to negotiate a common version to make sure that they can safely communicate without breaking API assumptions

helm delete --purge

Helm stores some files in $HELM_HOME, which is located by default in ~/.helm

A Chart is a Helm package. It contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster.

A Repository is the place where charts can be collected and shared.

Set the $HELM_HOME environment variable

Helm installs charts into Kubernetes, creating a new release for each installation. And to find new charts, you can search Helm chart repositories.

chart repository is named stable by default

helm search shows you all of the available charts

helm inspect

To install a new package, use the helm install command. At its simplest, it takes only one argument: The name of the chart.

If you want to use your own release name, simply use the --name flag on helm install

additional configuration steps you can or should take.

Helm does not wait until all of the resources are running before it exits. Many charts require Docker images that are over 600M in size, and may take a long time to install into the cluster.

helm status

helm inspect values

helm inspect values stable/mariadb

override any of these settings in a YAML formatted file, and then pass that file during installation.

helm install -f config.yaml stable/mariadb

--values (or -f): Specify a YAML file with overrides.

--set (and its variants --set-string and --set-file): Specify overrides on the command line.

Values that have been --set can be cleared by running helm upgrade with --reset-values specified.

Chart designers are encouraged to consider the --set usage when designing the format of a values.yaml file.

--set-file key=filepath is another variant of --set. It reads the file and use its content as a value.

inject a multi-line text into values without dealing with indentation in YAML.

An unpacked chart directory

When a new version of a chart is released, or when you want to change the configuration of your release, you can use the helm upgrade command.

Kubernetes charts can be large and complex, Helm tries to perform the least invasive upgrade.

$ helm upgrade -f panda.yaml happy-panda stable/mariadb

deployment

If both are used, --set values are merged into --values with higher precedence.

The helm get command is a useful tool for looking at a release in the cluster.

A release version is an incremental revision. Every time an install, upgrade, or rollback happens, the revision number is incremented by 1.

helm history

you can rollback a deleted resource, and have it re-activate.

helm repo list

helm repo add

helm repo update

helm create

helm lint

helm package

Charts that are archived can be loaded into chart repositories.

chart repository server

Tiller can be installed into any namespace.

Limiting Tiller to only be able to install into specific namespaces and/or resource types is controlled by Kubernetes RBAC roles and rolebindings

Release names are unique PER TILLER INSTANCE

Charts should only contain resources that exist in a single namespace.

not recommended to have multiple Tillers configured to manage resources in the same namespace.

a client-side Helm plugin. A plugin is a tool that can be accessed through the helm CLI, but which is not part of the built-in Helm codebase.

Helm plugins are add-on tools that integrate seamlessly with Helm. They provide a way to extend the core feature set of Helm, but without requiring every new feature to be written in Go and added to the core tool.

Helm plugins live in $(helm home)/plugins

The Helm plugin model is partially modeled on Git’s plugin model

helm referred to as the porcelain layer, with plugins being the plumbing.

helm plugin install https://github.com/technosophos/helm-template

command is the command that this plugin will execute when it is called.

Environment variables are interpolated before the plugin is executed.

The command itself is not executed in a shell. So you can’t oneline a shell script.

Helm is able to fetch Charts using HTTP/S

Variables like KUBECONFIG are set for the plugin if they are set in the outer environment.

In Kubernetes, granting a role to an application-specific service account is a best practice to ensure that your application is operating in the scope that you have specified.

restrict Tiller’s capabilities to install resources to certain namespaces, or to grant a Helm client running access to a Tiller instance.

Service account with cluster-admin role

The cluster-admin role is created by default in a Kubernetes cluster

Deploy Tiller in a namespace, restricted to deploying resources only in that namespace

Deploy Tiller in a namespace, restricted to deploying resources in another namespace

When running a Helm client in a pod, in order for the Helm client to talk to a Tiller instance, it will need certain privileges to be granted.

SSL Between Helm and Tiller

The Tiller authentication model uses client-side SSL certificates.

creating an internal CA, and using both the cryptographic and identity functions of SSL.

Helm is a powerful and flexible package-management and operations tool for Kubernetes.

default installation applies no security configurations

with a cluster that is well-secured in a private network with no data-sharing or no other users or teams.

With great power comes great responsibility.

Choose the Best Practices you should apply to your helm installation

Role-based access control, or RBAC

Tiller’s gRPC endpoint and its usage by Helm

Kubernetes employ a role-based access control (or RBAC) system (as do modern operating systems) to help mitigate the damage that can be done if credentials are misused or bugs exist.

In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied.

Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.

release information

charts

charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.

Helm’s provenance tools to ensure the provenance and integrity of charts

SysBench is not a tool which you can use to tune configurations of your MySQL servers (unless you prepared LUA scripts with custom workload or your workload happen to be very similar to the benchmark workloads that SysBench comes with)

it is great for is to compare performance of different hardware.

Every new server acquired should go through a warm-up period during which you will stress it to pinpoint potential hardware defects

by executing OLTP workload which overloads the server, or you can also use dedicated benchmarks for CPU, disk and memory.

bulk_insert.lua. This test can be used to benchmark the ability of MySQL to perform multi-row inserts.

All oltp_* scripts share a common table structure. First two of them (oltp_delete.lua and oltp_insert.lua) execute single DELETE and INSERT statements.

oltp_point_select, oltp_update_index and oltp_update_non_index. These will execute a subset of queries - primary key-based selects, index-based updates and non-index-based updates.

you can run different workload patterns using the same benchmark.

Warmup helps to identify “regular” throughput by executing benchmark for a predefined time, allowing to warm up the cache, buffer pools etc.

By default SysBench will attempt to execute queries as fast as possible. To simulate slower traffic this option may be used. You can define here how many transactions should be executed per second.

SysBench gives you ability to generate different types of data distribution.

decide if SysBench should use prepared statements (as long as they are available in the given datastore - for MySQL it means PS will be enabled by default) or not.

sysbench ./sysbench/src/lua/oltp_read_write.lua  help

By default, SysBench will attempt to execute queries in explicit transaction. This way the dataset will stay consistent and not affected: SysBench will, for example, execute INSERT and DELETE on the same row, making sure the data set will not grow (impacting your ability to reproduce results).

specify error codes from MySQL which SysBench should ignore (and not kill the connection).

the two most popular benchmarks - OLTP read only and OLTP read/write.

1 million rows will result in ~240 MB of data. Ten tables, 1000 000 rows each equals to 2.4GB

by default, SysBench looks for ‘sbtest’ schema which has to exist before you prepare the data set. You may have to create it manually.

pass ‘--histogram’ argument to SysBench

~48GB of data (20 tables, 10 000 000 rows each).

if you don’t understand why the performance was like it was, you may draw incorrect conclusions out of the benchmarks.

combining existing solutions and existing blocks: KeepAlived + ProxySQl + MySQL.

Keepalived implements a set of checkers to dynamically and adaptively maintain and manage load-balanced server pool according to their health.

Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions.

mysql_variables: contains global variables that control the functionality for handling the incoming MySQL traffic.

mysql_users: contains rows for the mysql_users table from the admin interface. Basically, these define the users which can connect to the proxy, and the users with which the proxy can connect to the backend servers.

mysql_servers: contains rows for the mysql_servers table from the admin interface. Basically, these define the backend servers towards which the incoming MySQL traffic is routed.

mysql_query_rules: contains rows for the mysql_query_rules table from the admin interface. Basically, these define the rules used to classify and route the incoming MySQL traffic, according to various criteria (patterns matched, user used to run the query, etc.).

Docker 的资源限制用的就是 cgroups

Percona：我们的备份、慢日志分析、过载保护等功能都是基于 pt-tools 工具包来实现的。

Consul：分布式的服务发现和配置共享软件

容器调度的开源产品主要有 Kubernetes 和 mesos

适合自己现状的需求才是最好的

有机会做到计算调度和存储调度分离的情况下我们可能会转向 Kubernetes 的方案

根据这个需求按照我们的资源筛选规则 (比如主从不能在同一台机器、内存配置不允许超卖等等)，从现有的资源池中匹配出可用资源，然后依次创建主从关系、创建高可用管理、检查集群复制状态、推送集群信息到中间件 (选用了中间件的情况下) 控制中心、最后将以上相关信息都同步到 CMDB。

每一个工作都是通过服务端发送消息到 agent，然后由 agent 执行对应的脚本，脚本会返回指定格式的执行结果

备份工具我们是用 percona-xtrabackup

zabbix 来实现监控告警

grafana 是监控画图界的扛把子，功能齐全的度量仪表盘和图形编辑器，经过简单配置就能完成各种监控图形的展示。

(MariaDB 不支持写 table，只能写 file)，极大减少了从库复制带来的 IOPS。