Group items tagged

allowing your schema and changes to be database independent.

each migration as being a new 'version' of the database

each migration modifies it to add or remove tables, columns, or entries

roll this migration back, it will remove the table

timestamps macro adds two columns, created_at and updated_at

On databases that support transactions with statements that change the schema, migrations are wrapped in a transaction

reversible

use up and down instead of change

Migrations are stored as files in the db/migrate directory, one for each migration class.

a UTC timestamp identifying

Rails uses this timestamp to determine which migration should be run and in what order

"AddXXXToYYY" or "RemoveXXXFromYYY"

use a Ruby DSL

column type as references

part_number:string:index

a migration to remove a column

"CreateXXX"

change_column_null

AddUserRefToProducts

:references

produce join tables if JoinTable is part of the name

CreateJoinTable

The model and scaffold generators will create migrations appropriate for adding a new model.

enclosed by curly braces and follow the field type

create_table

By default, create_table will create a primary key called id

add an index on the new column

when using MySQL, the default is ENGINE=InnoDB

create_join_table creates an HABTM (has and belongs to many) join table

create_join_table also accepts a block

change_table, used for changing existing tables

remove

rename

add_column

change_column

remove_column

change_column_default

place an SQL fragment in the :options option.

limit

precision

scale

polymorphic

default

index

add_foreign_key

use the old style of migration using up and down methods instead of the change method.

.connection.execute

remove_column is reversible if you supply the column type as the third argument

Complex migrations may require processing that Active Record doesn't know how to reverse

reversible

Using reversible will ensure that the instructions are executed in the right order too.

add_column add_foreign_key add_index add_reference add_timestamps change_column_default (must supply a :from and :to option) change_column_null create_join_table create_table disable_extension drop_join_table drop_table (must supply a block) enable_extension remove_column (must supply a type) remove_foreign_key (must supply a second table) remove_index remove_reference remove_timestamps rename_column rename_index rename_table

:column_options option

have the option :null set to false by default

By default, the name of the join table comes from the union of the first two arguments provided to create_join_table

If the column names can not be derived from the table names, you can use the :column and :primary_key options.

figure out the column name

foreign key for a specific column

foreign key by name

should use reversible or write the up and down methods instead of using the change method

If your migration is irreversible, you should raise ActiveRecord::IrreversibleMigration from your down method.

DontUseConstraintForZipcodeValidationMigration

rails db:migrate

specify a target version

all migrations up to and including 20080906120000

run the down method on all the migrations down to, but not including, 20080906120000

rails db:rollback

db:migrate:redo task is a shortcut for doing a rollback and then migrating back up again

STEP parameter

db:setup task will create the database, load the schema and initialize it with the seed data

db:reset task will drop the database and set it up again. This is functionally equivalent to rails db:drop db:setup.

run a specific migration up or down, the db:migrate:up and db:migrate:down

the RAILS_ENV environment variable

db:migrate VERBOSE=false will suppress all output.

must rollback the migration (for example with bin/rails db:rollback), edit your migration and then run rails db:migrate to run the corrected version.

editing existing migrations is not a good idea.

revert method can be helpful when writing a new migration to undo previous migrations in whole or in part

require_relative

revert

Schema Files for

Schema files are also useful if you want a quick look at what attributes an Active Record object has

annotate_models gem automatically adds and updates comments at the top of each model summarizing the schema if you desire that functionality.

database-independent

multiple databases

you can execute custom SQL statements, the schema dumper cannot reconstitute those statements from the database

db:structure:dump

set in config/application.rb by the config.active_record.schema_format setting, which may be either :sql or :ruby.

check them into source control.

db/schema.rb contains the current version number of the database

Validations such as validates :foreign_key, uniqueness: true are one way in which models can enforce data integrity

The :dependent option on associations allows models to automatically destroy child objects when the parent is destroyed.

Migrations can also be used to add or modify data

Initial

To add initial data after a database is created, Rails has a built-in 'seeds' feature that makes the process quick and easy.

db/seeds.rb

rails db:seed

must specify the public subnet in which the NAT gateway should reside

update the route table associated with one or more of your private subnets to point Internet-bound traffic to the NAT gateway.

NAT gateway is created in a specific Availability Zone and implemented with redundancy in that zone.

ensure that resources use the NAT gateway in the same Availability Zone

A NAT gateway supports 5 Gbps of bandwidth and automatically scales up to 45 Gbps

You can associate exactly one Elastic IP address with a NAT gateway

A NAT gateway supports the following protocols: TCP, UDP, and ICMP

cannot associate a security group with a NAT gateway.

create a NAT gateway in the same subnet as your NAT instance, and then replace the existing route in your route table that points to the NAT instance with a route that points to the NAT gateway

A NAT gateway cannot send traffic over VPC endpoints, VPN connections, AWS Direct Connect, or VPC peering connections.

When you run an image and generate a container, you add a new writable layer (the “container layer”) on top of the underlying layers.

Inadvertently including files that are not necessary for building an image results in a larger build context and larger image size.

To exclude files not relevant to the build (without restructuring your source repository) use a .dockerignore file. This file supports exclusion patterns similar to .gitignore files.

minimize image layers by leveraging build cache.

avoid installing extra or unnecessary packages just because they might be “nice to have.”

Decoupling applications into multiple containers makes it easier to scale horizontally and reuse containers

Limiting each container to one process is a good rule of thumb, but it is not a hard and fast rule.

do multi-stage builds and only copy the artifacts you need into the final image. This allows you to include tools and debug information in your intermediate build stages without increasing the size of the final image.

avoid duplication of packages and make the list much easier to update.

When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified.

simply comparing the instruction in the Dockerfile with one of the child images is sufficient.

For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file.

If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

In that case just the command string itself is used to find a match.

Whenever possible, use current official repositories as the basis for your images.

Using RUN apt-get update && apt-get install -y ensures your Dockerfile installs the latest package versions with no further coding or manual intervention.

cache busting

Docker executes these commands using the /bin/sh -c interpreter, which only evaluates the exit code of the last operation in the pipe to determine success.

set -o pipefail && to ensure that an unexpected error prevents the build from inadvertently succeeding.

The CMD instruction should be used to run the software contained by your image, along with any arguments.

CMD should rarely be used in the manner of CMD [“param”, “param”] in conjunction with ENTRYPOINT

The ENV instruction is also useful for providing required environment variables specific to services you wish to containerize,

COPY only supports the basic copying of local files into the container

the best use for ADD is local tar file auto-extraction into the image, as in ADD rootfs.tar.xz /

using ADD to fetch packages from remote URLs is strongly discouraged; you should use curl or wget instead

The best use for ENTRYPOINT is to set the image’s main command, allowing that image to be run as though it was that command (and then use CMD as the default flags).

the image name can double as a reference to the binary as shown in the command

The VOLUME instruction should be used to expose any database storage area, configuration storage, or files/folders created by your docker container.

use VOLUME for any mutable and/or user-serviceable parts of your image

If you absolutely need functionality similar to sudo, such as initializing the daemon as root but running it as non-root), consider using “gosu”.

always use absolute paths for your WORKDIR

An ONBUILD command executes after the current Dockerfile build completes.

Think of the ONBUILD command as an instruction the parent Dockerfile gives to the child Dockerfile

Be careful when putting ADD or COPY in ONBUILD. The “onbuild” image fails catastrophically if the new build’s context is missing the resource being added.

Communication between pods is more complicated, however, and requires a separate networking component that can transparently route traffic from a pod on one node to a pod on another.

pod network plugins. For this cluster, you will use Flannel, a stable and performant option.

Passing the argument --pod-network-cidr=10.244.0.0/16 specifies the private subnet that the pod IPs will be assigned from.

kubectl apply -f descriptor.[yml|json] is the syntax for telling kubectl to create the objects described in the descriptor.[yml|json] file.

deploy Nginx using Deployments and Services

NodePort, a scheme that will make the pod accessible through an arbitrary port opened on each node of the cluster

load balancing requests to multiple pods

Pods are ubiquitous in Kubernetes, so understanding them will facilitate your work

how controllers such as deployments work since they are used frequently in stateless applications for scaling and the automated healing of unhealthy applications.

Understanding the types of services and the options they have is essential for running both stateless and stateful applications.

A Pod models an application-specific “logical host”

being executed on the same physical or virtual machine would mean being executed on the same logical host.

The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation

Containers in different Pods have distinct IP addresses and can not communicate by IPC without special configuration. These containers usually communicate with each other via Pod IP addresses.

a Pod is modelled as a group of Docker containers with shared namespaces and shared filesystem volumes

Pods are considered to be relatively ephemeral (rather than durable) entities.

Pods are created, assigned a unique ID (UID), and scheduled to nodes where they remain until termination (according to restart policy) or deletion.

it can be replaced by an identical Pod

When something is said to have the same lifetime as a Pod, such as a volume, that means that it exists as long as that Pod (with that UID) exists.

uses a persistent volume for shared storage between the containers

Pods serve as unit of deployment, horizontal scaling, and replication

flat shared networking space

Containers within the Pod see the system hostname as being the same as the configured name for the Pod.

Volumes enable data to survive container restarts and to be shared among the applications within the Pod.

Individual Pods are not intended to run multiple instances of the same application

The individual containers may be versioned, rebuilt and redeployed independently.

Controllers like StatefulSet can also provide support to stateful Pods.

When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container.

Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server.

grace period

Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers.

When the grace period expires, any processes still running in the Pod are killed with SIGKILL.

By default, all deletes are graceful within 30 seconds.

You must specify an additional flag --force along with --grace-period=0 in order to perform force deletions.

Force deletion of a Pod is defined as deletion of a Pod from the cluster state and etcd immediately.

Processes within the container get almost the same privileges that are available to processes outside a container.

load balancing, connection failover and decoupling of the application tier from the underlying database topologies.

ProxySQL as a Kubernetes service (centralized deployment)

running as a service makes ProxySQL pods live independently from the applications and can be easily scaled and clustered together with the help of Kubernetes ConfigMap.

ProxySQL's multi-layer configuration system makes pod clustering possible with ConfigMap.

create ProxySQL pods and attach a Kubernetes service to be accessed by the other pods within the Kubernetes network or externally.

Default to 6033 for MySQL load-balanced connections and 6032 for ProxySQL administration console.

separated by "---" delimiter

deploy two ProxySQL pods as a ReplicaSet that matches containers labelled with "app=proxysql,tier=frontend".

A Kubernetes service is an abstraction layer which defines the logical set of pods and a policy by which to access them

The range of valid ports for NodePort resource is 30000-32767.

ConfigMap - To store ProxySQL configuration file as a volume so it can be mounted to multiple pods and can be remounted again if the pod is being rescheduled to the other Kubernetes node.

VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

A vSphere datastore is an abstraction which hides storage details (such as LUNs) and provides a uniform interface for storing persistent data.

the datastores can be of the type vSAN, VMFS, NFS & VVol.

VMFS (Virtual Machine File System) is a cluster file system that allows virtualization to scale beyond a single node for multiple VMware ESX servers.

NFS (Network File System) is a distributed file protocol to access storage over network like local storage.

vSphere Cloud Provider supports every storage primitive exposed by Kubernetes

Kubernetes PVs are defined in Pod specifications.

Nginx is built to handle many concurrent connections at the same time.

provides you with flexibility in easily adding backend servers or taking them down as needed for maintenance

Proxying in Nginx is accomplished by manipulating a request aimed at the Nginx server and passing it to other servers for the actual processing

When a request matches a location with a proxy_pass directive inside, the request is forwarded to the URL given by the directive

The request coming from Nginx on behalf of a client will look different than a request coming directly from a client

Nginx gets rid of any empty headers

Nginx, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request

The "Host" header is re-written to the value defined by the $proxy_host variable.

The upstream should not expect this connection to be persistent

Headers with empty values are completely removed from the passed request.

by default, this will be set to the value of $proxy_host, a variable that will contain the domain name or IP address and port taken directly from the proxy_pass definition

This is selected by default as it is the only address Nginx can be sure the upstream server responds to

$http_host: Sets the "Host" header to the "Host" header from the client request.

preference to: the host name from the request line itself

set the "Host" header to the $host variable. It is the most flexible and will usually provide the proxied servers with a "Host" header filled in as accurately as possible

This variable takes the value of the original X-Forwarded-For header retrieved from the client and adds the Nginx server's IP address to the end.

Once defined, this name will be available for use within proxy passes as if it were a regular domain name

By default, this is just a simple round-robin selection process (each request will be routed to a different host in turn)

Specifies that new connections should always be given to the backend that has the least number of active connections.

distributes requests to different servers based on the client's IP address.

mainly used with memcached proxying

As for the hash method, you must provide the key to hash against

Server Weight

Nginx's buffering and caching capabilities

Without buffers, data is sent from the proxied server and immediately begins to be transmitted to the client.

With buffers, the Nginx proxy will temporarily store the backend's response and then feed this data to the client

Nginx defaults to a buffering design

can be set in the http, server, or location contexts.

the sizing directives are configured per request, so increasing them beyond your need can affect your performance

When buffering is "off" only the buffer defined by the proxy_buffer_size directive will be used

A high availability (HA) setup is an infrastructure without a single point of failure, and your load balancers are a part of this configuration.

multiple load balancers (one active and one or more passive) behind a static IP address that can be remapped from one server to another.

Nginx also provides a way to cache content from backend servers

proxy_cache backcache;

The proxy_cache_bypass directive is set to the $http_cache_control variable. This will contain an indicator as to whether the client is explicitly requesting a fresh, non-cached version of the resource

For private content, you should set the Cache-Control header to "no-cache", "no-store", or "private" depending on the nature of the data

The values of fields may include other documents, arrays, and arrays of documents.

reduce need for expensive joins

MongoDB stores documents in collections.

Read-only Views

Indexes support faster queries and can include keys from embedded documents and arrays.

MongoDB's replication facility, called replica set

A replica set is a group of MongoDB servers that maintain the same data set, providing redundancy and increasing data availability.

Sharding distributes data across a cluster of machines.

MongoDB supports creating zones of data based on the shard key.

MongoDB provides pluggable storage engine API

Kubebuilder is a comprehensive development kit for building and publishing Kubernetes APIs and Controllers using CRDs

Design declarative APIs for operators, not imperative APIs. This aligns well with Kubernetes APIs that are declarative in nature.

scaling, backup, restore, and monitoring. An operator should be made up of multiple controllers that specifically handle each of the those features.

the operator can have a main controller to spawn and manage application instances, a backup controller to handle backup operations, and a restore controller to handle restore operations.

each controller should correspond to a specific CRD so that the domain of each controller's responsibility is clear.

If you keep a log for every container, you will likely end up with unmanageable amount of logs.

integrate application-specific details to the log messages such as adding a prefix for the application name.

you may have to use external logging tools such as Google Stackdriver, Elasticsearch, Fluentd, or Kibana to perform the aggregations.

adding labels to metrics to facilitate aggregation and analysis by monitoring systems.

a more viable option is for application pods to expose a metrics HTTP endpoint for monitoring tools to scrape.

A good way to achieve this is to use open-source application-specific exporters for exposing Prometheus-style metrics.

the containers within a pod can communicate with each other over localhost

we can never create a standalone container: the closest we can do is create a pod, with a single container in it.

Kubernetes is a declarative system (by opposition to imperative systems).

All we can do, is describe what we want to have, and wait for Kubernetes to take action to reconcile what we have, with what we want to have.

The specification of a replica set looks very much like the specification of a pod, except that it carries a number, indicating how many replicas

What happens if we change that definition? Suddenly, there are zero pods matching the new specification.

the creation of new pods could happen in a more gradual manner.

the specification for a deployment looks very much like the one for a replica set: it features a pod specification, and a number of replicas.

Deployments, however, don’t create or delete pods directly.

When we update a deployment and adjust the number of replicas, it passes that update down to the replica set.

When we update the pod specification, the deployment creates a new replica set with the updated pod specification. That replica set has an initial size of zero. Then, the size of that replica set is progressively increased, while decreasing the size of the other replica set.

we are going to fade in (turn up the volume) on the new replica set, while we fade out (turn down the volume) on the old one.

A readiness probe is a test that we add to a container specification.

Kubernetes supports three ways of implementing readiness probes:Running a command inside a container;Making an HTTP(S) request against a container; orOpening a TCP socket against a container.

When we roll out a new version, Kubernetes will wait for the new pod to mark itself as “ready” before moving on to the next one.

MaxSurge indicates how many extra pods we are willing to run during a rolling update, while MaxUnavailable indicates how many pods we can lose during the rolling update.

Setting MaxUnavailable to 0 means, “do not shutdown any old pod before a new one is up and ready to serve traffic“.

Setting MaxSurge to 100% means, “immediately start all the new pods“, implying that we have enough spare capacity on our cluster, and that we want to go as fast as possible.

the replica set doesn’t look at the pods’ specifications, but only at their labels.

A replica set contains a selector, which is a logical expression that “selects” (just like a SELECT query in SQL) a number of pods.

Selectors are also used by services, which act as the load balancers for Kubernetes traffic, internal and external.

during a rollout, the deployment doesn’t reconfigure or inform the load balancer that pods are started and stopped. It happens automatically through the selector of the service associated to the load balancer.

In blue/green deployment, we want to instantly switch over all the traffic from the old version to the new, instead of doing it progressively

We can achieve blue/green deployment by creating multiple deployments (in the Kubernetes sense), and then switching from one to another by changing the selector of our service

kubectl label pods -l app=blue,version=v1.5 status=enabled

kubectl label pods -l app=blue,version=v1.4 status-

This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.

differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.

Filebeat is just a tiny binary with no dependencies.

For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

For example, the apache module will point Filebeat to default access.log and error.log paths

Filebeat’s scope is very limited,

Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Filebeat can parse JSON

you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.

You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off

For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well

The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.