Group items tagged

Percona XtraBackup works by remembering the log sequence number (LSN) when it starts, and then copying away the data files.

Percona XtraBackup runs a background process that watches the transaction log files, and copies changes from it.

Percona XtraBackup uses Backup locks where available as a lightweight alternative to FLUSH TABLES WITH READ LOCK.

Locking is only done for MyISAM and other non-InnoDB tables after Percona XtraBackup finishes backing up all InnoDB/XtraDB data and logs.

When backup locks are supported by the server, xtrabackup first copies InnoDB data, runs the LOCK TABLES FOR BACKUP and then copies the MyISAM tables.

the STDERR of xtrabackup is not written in any file. You will have to redirect it to a file, e.g., xtrabackup OPTIONS 2> backupout.log

During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files, using the copied transaction log file. After this is done, the database is ready to restore and use.

the tools enable you to do operations such as streaming and incremental backups with various combinations of copying the data files, copying the log files, and applying the logs to the data.

To restore a backup with xtrabackup you can use the --copy-back or --move-back options.

In full backups, two types of operations are performed to make the database consistent: committed transactions are replayed from the log file against the data files, and uncommitted transactions are rolled back.

You should use the --apply-log-only option to prevent the rollback phase.

An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s LSN.

Incremental backups do not actually compare the data files to the previous backup’s data files.

Incremental backups simply read the pages and compare their LSN to the last backup’s LSN.

The xtrabackup binary writes a file called xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn, which is the database’s LSN at the end of the backup.

from_lsn is the starting LSN of the backup and for incremental it has to be the same as to_lsn (if it is the last checkpoint) of the previous/base backup.

If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups will be useless.

If you restore it and start MySQL, InnoDB will detect that the rollback phase was not performed, and it will do that in the background, as it usually does for a crash recovery upon start.

xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc1

The final data is in /data/backups/base, not in the incremental directory.

xtrabackup --prepare --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc2

Even if the --apply-log-only was used on the last step, backup would still be consistent but in that case server would perform the rollback phase.

SysBench is not a tool which you can use to tune configurations of your MySQL servers (unless you prepared LUA scripts with custom workload or your workload happen to be very similar to the benchmark workloads that SysBench comes with)

it is great for is to compare performance of different hardware.

Every new server acquired should go through a warm-up period during which you will stress it to pinpoint potential hardware defects

by executing OLTP workload which overloads the server, or you can also use dedicated benchmarks for CPU, disk and memory.

bulk_insert.lua. This test can be used to benchmark the ability of MySQL to perform multi-row inserts.

All oltp_* scripts share a common table structure. First two of them (oltp_delete.lua and oltp_insert.lua) execute single DELETE and INSERT statements.

oltp_point_select, oltp_update_index and oltp_update_non_index. These will execute a subset of queries - primary key-based selects, index-based updates and non-index-based updates.

you can run different workload patterns using the same benchmark.

Warmup helps to identify “regular” throughput by executing benchmark for a predefined time, allowing to warm up the cache, buffer pools etc.

By default SysBench will attempt to execute queries as fast as possible. To simulate slower traffic this option may be used. You can define here how many transactions should be executed per second.

SysBench gives you ability to generate different types of data distribution.

decide if SysBench should use prepared statements (as long as they are available in the given datastore - for MySQL it means PS will be enabled by default) or not.

sysbench ./sysbench/src/lua/oltp_read_write.lua  help

By default, SysBench will attempt to execute queries in explicit transaction. This way the dataset will stay consistent and not affected: SysBench will, for example, execute INSERT and DELETE on the same row, making sure the data set will not grow (impacting your ability to reproduce results).

specify error codes from MySQL which SysBench should ignore (and not kill the connection).

the two most popular benchmarks - OLTP read only and OLTP read/write.

1 million rows will result in ~240 MB of data. Ten tables, 1000 000 rows each equals to 2.4GB

by default, SysBench looks for ‘sbtest’ schema which has to exist before you prepare the data set. You may have to create it manually.

pass ‘--histogram’ argument to SysBench

~48GB of data (20 tables, 10 000 000 rows each).

if you don’t understand why the performance was like it was, you may draw incorrect conclusions out of the benchmarks.

These specialized roles create efficiencies within each segment while potentially creating inefficiencies across the entire life cycle.

Grouping differing specialists together into one team can reduce silos, but having different people do each role adds communication overhead, introduces bottlenecks, and inhibits the effectiveness of feedback loops.

devops principles

develops a system also be responsible for operating and supporting that system

Each development team owns deployment issues, performance bugs, capacity planning, alerting gaps, partner support, and so on.

Those centralized teams act as force multipliers by turning their specialized knowledge into reusable building blocks.

Full cycle developers are expected to be knowledgeable and effective in all areas of the software life cycle.

A full cycle developer thinks and acts like an SWE, SDET, and SRE. At times they create software that solves business problems, at other times they write test cases for that, and still other times they automate operational aspects of that system.

the need for continuous delivery pipelines, monitoring/observability, and so on.

Tooling and automation help to scale expertise, but no tool will solve every problem in the developer productivity and operations space

replication can provide increased read capacity as clients can send read operations to different servers.

A replica set contains several data bearing nodes and optionally one arbiter node.

A replica set can have only one primary capable of confirming writes with { w: "majority" } write concern; although in some circumstances, another mongod instance may transiently believe itself to also be primary.

The secondaries replicate the primary’s oplog and apply the operations to their data sets such that the secondaries’ data sets reflect the primary’s data set

add a mongod instance to a replica set as an arbiter. An arbiter participates in elections but does not hold data

An arbiter will always be an arbiter whereas a primary may step down and become a secondary and a secondary may become the primary during an election.

Secondaries replicate the primary’s oplog and apply the operations to their data sets asynchronously.

These slow oplog messages are logged for the secondaries in the diagnostic log under the REPL component with the text applied op: <oplog entry> took <num>ms.

Replication lag refers to the amount of time that it takes to copy (i.e. replicate) a write operation on the primary to a secondary.

When a primary does not communicate with the other members of the set for more than the configured electionTimeoutMillis period (10 seconds by default), an eligible secondary calls for an election to nominate itself as the new primary.

The replica set cannot process write operations until the election completes successfully.

The median time before a cluster elects a new primary should not typically exceed 12 seconds, assuming default replica configuration settings.

Factors such as network latency may extend the time required for replica set elections to complete, which in turn affects the amount of time your cluster may operate without a primary.

MongoDB drivers can detect the loss of the primary and automatically retry certain write operations a single time, providing additional built-in handling of automatic failovers and elections

By default, clients read from the primary [1]; however, clients can specify a read preference to send read operations to secondaries.

While you cannot use it to access templates, you can use it to access other files in the chart.

Release: This object describes the release itself.

Values: Values passed into the template from the values.yaml file

--set has a higher precedence than the default values.yaml file

Values files can contain more structured content

If you need to delete a key from the default values, you may override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.

Kubernetes would then fail because you can not declare more than one livenessProbe handler.

Drawing on a concept from UNIX, pipelines are a tool for chaining together a series of template commands to compactly express a series of transformations.

the default function: default DEFAULT_VALUE GIVEN_VALUE

all static default values should live in the values.yaml, and should not be repeated using the default command (otherwise they would be redundant).

the default command is perfect for computed values, which can not be declared inside values.yaml.

When lookup returns an object, it will return a dictionary.

The synopsis of the lookup function is lookup apiVersion, kind, namespace, name -> resource or resource list

When no object is found, an empty value is returned. This can be used to check for the existence of an object.

The lookup function uses Helm's existing Kubernetes connection configuration to query Kubernetes.

Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run, so the lookup function will return an empty list (i.e. dict) in such a case.

the operators (eq, ne, lt, gt, and, or and so on) are all implemented as functions. In pipelines, operations can be grouped with parentheses ((, and )).

After the block type keyword and any labels, the block body is delimited by the { and } characters

Identifiers can contain letters, digits, underscores (_), and hyphens (-). The first character of an identifier must not be a digit, to avoid ambiguity with literal numbers.

The # single-line comment style is the default comment style and should be used in most cases.

he idiomatic style is to use the Unix convention

Indent two spaces for each nesting level.

align their equals signs

Use empty lines to separate logical groups of arguments within a block.

Use one blank line to separate the arguments from the blocks.

"meta-arguments" (as defined by the Terraform language semantics)

Avoid separating multiple blocks of the same type with other blocks of a different type, unless the block types are defined by semantics to form a family.

Resource names must start with a letter or underscore, and may contain only letters, digits, underscores, and dashes.

By convention, resource type names start with their provider's preferred local name.

Most publicly available providers are distributed on the Terraform Registry, which also hosts their documentation.

The Terraform language defines several meta-arguments, which can be used with any resource type to change the behavior of resources.

use precondition and postcondition blocks to specify assumptions and guarantees about how the resource operates.

Some resource types provide a special timeouts nested block argument that allows you to customize how long certain operations are allowed to take before being considered to have failed.

Timeouts are handled entirely by the resource type implementation in the provider

Most resource types do not support the timeouts block at all.

A resource block declares that you want a particular infrastructure object to exist with the given settings.

Destroy resources that exist in the state but no longer exist in the configuration.

Destroy and re-create resources whose arguments have changed but which cannot be updated in-place due to remote API limitations.

Expressions within a Terraform module can access information about resources in the same module, and you can use that information to help configure other resources. Use the <RESOURCE TYPE>.<NAME>.<ATTRIBUTE> syntax to reference a resource attribute in an expression.

resources often provide read-only attributes with information obtained from the remote API; this often includes things that can't be known until the resource is created, like the resource's unique random ID.

data sources, which are a special type of resource used only for looking up information.

some dependencies cannot be recognized implicitly in configuration.

local-only resource types exist for generating private keys, issuing self-signed TLS certificates, and even generating random ids.

The count meta-argument accepts a whole number, and creates that many instances of the resource or module.

the count value must be known before Terraform performs any remote resource actions. This means count can't refer to any resource attributes that aren't known until after a configuration is applied

Within nested provisioner or connection blocks, the special self object refers to the current resource instance, not the resource block as a whole.

Terraform reads data resources during the planning phase when possible, but announces in the plan when it must defer reading resources until the apply phase to preserve the order of operations.

local-only data sources exist for rendering templates, reading local files, and rendering AWS IAM policies.

As with managed resources, when count or for_each is present it is important to distinguish the resource itself from the multiple resource instances it creates. Each instance will separately read from its data source with its own variant of the constraint arguments, producing an indexed result.

Data instance arguments may refer to computed values, in which case the attributes of the instance itself cannot be resolved until all of its arguments are defined. I

Pod-to-Service communications

External-to-Service communications

sharing machines requires ensuring that two applications do not try to use the same ports.

Dynamic port allocation brings a lot of complications to the system

do not need to explicitly create links between Pods

almost never need to deal with mapping container ports to host ports.

pods on a node can communicate with all pods on all nodes without NAT

agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node

pods in the host network of a node can communicate with all pods on all nodes without NAT

containers within a Pod share their network namespaces - including their IP address

containers within a Pod must coordinate port usage

“IP-per-pod” model.

request ports on the Node itself which forward to your Pod (called host ports), but this is a very niche operation

The Pod itself is blind to the existence or non-existence of host ports.

AOS is an Intent-Based Networking system that creates and manages complex datacenter environments from a simple integrated platform.

Cisco Application Centric Infrastructure offers an integrated overlay and underlay SDN solution that supports containers, virtual machines, and bare metal servers.

AOS Reference Design currently supports Layer-3 connected hosts that eliminate legacy Layer-2 switching problems.

The AWS VPC CNI offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters.

users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters.

Using this CNI plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.

The CNI allocates AWS Elastic Networking Interfaces (ENIs) to each Kubernetes node and using the secondary IP range from each ENI for pods on the node.

Big Cloud Fabric is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments.

Cilium is L7/HTTP aware and can enforce network policies on L3-L7 using an identity based security model that is decoupled from network addressing.

CNI-Genie is a CNI plugin that enables Kubernetes to simultaneously have access to different implementations of the Kubernetes network model in runtime.

CNI-Genie also supports assigning multiple IP addresses to a pod, each from a different CNI plugin.

cni-ipvlan-vpc-k8s contains a set of CNI and IPAM plugins to provide a simple, host-local, low latency, high throughput, and compliant networking stack for Kubernetes within Amazon Virtual Private Cloud (VPC) environments by making use of Amazon Elastic Network Interfaces (ENI) and binding AWS-managed IPs into Pods using the Linux kernel’s IPvlan driver in L2 mode.

to be straightforward to configure and deploy within a VPC

Contiv provides configurable networking

Contrail, based on Tungsten Fabric, is a truly open, multi-cloud network virtualization and policy management platform.

DANM is a networking solution for telco workloads running in a Kubernetes cluster.

Flannel is a very simple overlay network that satisfies the Kubernetes requirements.

Any traffic bound for that subnet will be routed directly to the VM by the GCE network fabric.

sysctl net.ipv4.ip_forward=1

Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.

Knitter is a network solution which supports multiple networking in Kubernetes.

Kube-OVN is an OVN-based kubernetes network fabric for enterprises.

Kube-router provides a Linux LVS/IPVS-based service proxy, a Linux kernel forwarding-based pod-to-pod networking solution with no overlays, and iptables/ipset-based network policy enforcer.

If you have a “dumb” L2 network, such as a simple switch in a “bare-metal” environment, you should be able to do something similar to the above GCE setup.

Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.

NSX-T can provide network virtualization for a multi-cloud and multi-hypervisor environment and is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks.

NSX-T Container Plug-in (NCP) provides integration between NSX-T and container orchestrators such as Kubernetes

Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.

OpenVSwitch is a somewhat more mature but also complicated way to build an overlay network

OVN is an opensource network virtualization solution developed by the Open vSwitch community.

Project Calico is an open source container networking provider and network policy engine.

Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet

Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking.

Romana is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network

Weave Net runs as a CNI plug-in or stand-alone. In either version, it doesn’t require any configuration or extra code to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.

The network model is implemented by the container runtime on each node.

Microservices also bring a set of additional benefits, such as easier scaling, the possibility to use multiple programming languages and technologies, and others.

Java is a frequent choice for building a microservices architecture as it is a mature language tested over decades and has a multitude of microservices-favorable frameworks, such as legendary Spring, Jersey, Play, and others.

A monolithic architecture keeps it all simple. An app has just one server and one database.

All the connections between units are inside-code calls.

split our application into microservices and got a set of units completely independent for deployment and maintenance.

Each of microservices responsible for a certain business function communicates either via sync HTTP/REST or async AMQP protocols.

ensure seamless communication between newly created distributed components.

The gateway became an entry point for all clients’ requests.

We also set the Zuul 2 framework for our gateway service so that the application could leverage the benefits of non-blocking HTTP calls.

we've implemented the Eureka server as our server discovery that keeps a list of utilized user profile and order servers to help them discover each other.

We also have a message broker (RabbitMQ) as an intermediary between the notification server and the rest of the servers to allow async messaging in-between.

microservices can definitely help when it comes to creating complex applications that deal with huge loads and need continuous improvement and scaling.