Group items tagged

This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.

differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.

Filebeat is just a tiny binary with no dependencies.

For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

For example, the apache module will point Filebeat to default access.log and error.log paths

Filebeat’s scope is very limited,

Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Filebeat can parse JSON

you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.

You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off

For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well

The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.

rsyslog is the fastest shipper

Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).

It’s also one of the lightest parsers you can find, depending on the configured memory buffers.

rsyslog requires more work to get the configuration right

the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.

rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).

rsyslog also works well when you need that ultimate performance.

syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).

a modular syslog daemon, that can do much more than just syslog

Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.

Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.

syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance

Fluentd plugins are in Ruby and very easy to write.

structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).

Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.

Splunk isn’t a log shipper, it’s a commercial logging solution

everything goes through graylog-server, from authentication to queries.

Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.

it depends

Pods are immutable, which means that when they die, they are not resurrected. The Kubernetes cluster creates new pods in the same node or in a new node once a pod dies. 

For internal application access within a Kubernetes cluster, ClusterIP is the preferred method

Kubernetes Ingress is an API object that provides routing rules to manage external users' access to the services in a Kubernetes cluster, typically via HTTPS/HTTP.

content-based routing, support for multiple protocols, and authentication.

Ingress is made up of an Ingress API object and the Ingress Controller.

Kubernetes Ingress is an API object that describes the desired state for exposing services to the outside of the Kubernetes cluster.

An Ingress Controller reads and processes the Ingress Resource information and usually runs as pods within the Kubernetes cluster.  

If Kubernetes Ingress is the API object that provides routing rules to manage external access to services, Ingress Controller is the actual implementation of the Ingress API.

Layer 7 (L7) refers to the application level of the OSI stack—external connections load-balanced across pods, based on requests.

if Kubernetes Ingress is a computer, then Ingress Controller is a programmer using the computer and taking action.

the Ingress Controller is an application that runs in a Kubernetes cluster and configures an HTTP load balancer according to Ingress Resources.

ClusterIP is the preferred option for internal service access and uses an internal IP address to access the service

A NodePort is a virtual machine (VM) used to expose a service on a Static Port number.

a NodePort would be used to expose a single service (with no load-balancing requirements for multiple services).

Ingress enables you to consolidate the traffic-routing rules into a single resource and runs as part of a Kubernetes cluster.

An application is accessed from the Internet via Port 80 (HTTP) or Port 443 (HTTPS), and Ingress is an object that allows access to your Kubernetes services from outside the Kubernetes cluster. 

To implement Ingress, you need to configure an Ingress Controller in your cluster—it is responsible for processing Ingress Resource information and allowing traffic based on the Ingress Rules.

Each control plane node creates a local etcd member and this etcd member communicates only with the kube-apiserver of this node.

This topology couples the control planes and etcd members on the same nodes.

An HA cluster with external etcd is a topology where the distributed data storage cluster provided by etcd is external to the cluster formed by the nodes that run control plane components.

etcd members run on separate hosts, and each etcd host communicates with the kube-apiserver of each control plane node.

This topology decouples the control plane and etcd member. It therefore provides an HA setup where losing a control plane instance or an etcd member has less impact and does not affect the cluster redundancy as much as the stacked HA topology.

name-based virtual hosting

Edge routerA router that enforces the firewall policy for your cluster.

A Kubernetes ServiceA way to expose an application running on a set of Pods as a network service. that identifies a set of Pods using labelTags objects with identifying attributes that are meaningful and relevant to users. selectors.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.

An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.

Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

Ingress frequently uses annotations to configure some options depending on the Ingress controller,

An optional host.

A list of paths

A backend is a combination of Service and port names

has an associated backend

Both the host and path must match the content of an incoming request before the load balancer directs traffic to the referenced Service.

HTTP (and HTTPS) requests to the Ingress that matches the host and path of the rule are sent to the listed backend.

An Ingress with no rules sends all traffic to a single default backend.

Ingress controllers and load balancers may take a minute or two to allocate an IP address.

A fanout configuration routes traffic from a single IP address to more than one Service, based on the HTTP URI being requested.

nginx.ingress.kubernetes.io/rewrite-target: /

describe ingress

get ingress

Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.

an Ingress resource without any hosts defined in the rules, then any web traffic to the IP address of your Ingress controller can be matched without a name based virtual host being required.

secure an Ingress by specifying a SecretStores sensitive information, such as passwords, OAuth tokens, and ssh keys. that contains a TLS private key and certificate.

An Ingress controller is bootstrapped with some load balancing policy settings that it applies to all Ingress, such as the load balancing algorithm, backend weight scheme, and others.

review the controller specific documentation to see how they handle health checks

edit ingress

After you save your changes, kubectl updates the resource in the API server, which tells the Ingress controller to reconfigure the load balancer.

kubectl replace -f on a modified Ingress YAML file.

Node: A worker machine in Kubernetes, part of a cluster.

in most common Kubernetes deployments, nodes in the cluster are not part of the public internet.

Edge router: A router that enforces the firewall policy for your cluster.

Service: A Kubernetes Service that identifies a set of Pods using label selectors.

An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.

The Ingress spec has all the information needed to configure a load balancer or proxy server.

An Ingress with no rules sends all traffic to a single default backend and .spec.defaultBackend is the backend that should handle requests in that case.

If defaultBackend is not set, the handling of requests that do not match any of the rules will be up to the ingress controller

A common usage for a Resource backend is to ingress data to an object storage backend with static assets.

Exact: Matches the URL path exactly and with case sensitivity.

Prefix: Matches based on a URL path prefix split by /. Matching is case sensitive and done on a path element by element basis.

multiple paths within an Ingress will match a request. In those cases precedence will be given first to the longest matching path.

Hosts can be precise matches (for example “foo.bar.com”) or a wildcard (for example “*.foo.com”).

Each Ingress should specify a class, a reference to an IngressClass resource that contains additional configuration including the name of the controller that should implement the class.

secure an Ingress by specifying a Secret that contains a TLS private key and certificate.

The Ingress resource only supports a single TLS port, 443, and assumes TLS termination at the ingress point (traffic to the Service and its Pods is in plaintext).

hosts in the tls section need to explicitly match the host in the rules section.

Terraform reads data resources during the planning phase when possible, but announces in the plan when it must defer reading resources until the apply phase to preserve the order of operations.

local-only data sources exist for rendering templates, reading local files, and rendering AWS IAM policies.

As with managed resources, when count or for_each is present it is important to distinguish the resource itself from the multiple resource instances it creates. Each instance will separately read from its data source with its own variant of the constraint arguments, producing an indexed result.

Data instance arguments may refer to computed values, in which case the attributes of the instance itself cannot be resolved until all of its arguments are defined. I

The Terraform Registry is the main directory of publicly available Terraform providers, and hosts providers for most major infrastructure platforms.

Dependency Lock File documents an additional HCL file that can be included with a configuration, which tells Terraform to always use a specific set of provider versions.

Terraform CLI finds and installs providers when initializing a working directory. It can automatically download providers from a Terraform registry, or load them from a local mirror or cache.

To save time and bandwidth, Terraform CLI supports an optional plugin cache. You can enable the cache using the plugin_cache_dir setting in the CLI configuration file.

you can use Terraform CLI to create a dependency lock file and commit it to version control along with your configuration.

Make sure that the br_netfilter module is loaded.

you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config,

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you.

one minor version skew between the kubelet and the control plane is supported, but the kubelet version may never exceed the API server version.

Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.

Associations are implemented using macro-style calls, so that you can declaratively add features to your models

A belongs_to association sets up a one-to-one connection with another model, such that each instance of the declaring model "belongs to" one instance of the other model.

belongs_to

A has_one association also sets up a one-to-one connection with another model, but with somewhat different semantics (and consequences).

belongs_to

A has_many association indicates a one-to-many connection with another model.

This association indicates that each instance of the model has zero or more instances of another model.

belongs_to

A has_many :through association is often used to set up a many-to-many connection with another model

This association indicates that the declaring model can be matched with zero or more instances of another model by proceeding through a third model.

through:

through:

The collection of join models can be managed via the API

new join models are created for newly associated objects, and if some are gone their rows are deleted.

The has_many :through association is also useful for setting up "shortcuts" through nested has_many associations

A has_one :through association sets up a one-to-one connection with another model. This association indicates that the declaring model can be matched with one instance of another model by proceeding through a third model.

A has_and_belongs_to_many association creates a direct many-to-many connection with another model, with no intervening model.

id: false

The has_one relationship says that one of something is yours

using t.references :supplier instead.

declare a many-to-many relationship is to use has_many :through. This makes the association indirectly, through a join model

set up a has_many :through relationship if you need to work with the relationship model as an independent entity

set up a has_and_belongs_to_many relationship (though you'll need to remember to create the joining table in the database).

use has_many :through if you need validations, callbacks, or extra attributes on the join model

With polymorphic associations, a model can belong to more than one other model, on a single association.

a polymorphic belongs_to declaration as setting up an interface that any other model can use.

In designing a data model, you will sometimes find a model that should have a relation to itself

add a references column to the model itself

All of the association methods are built around caching, which keeps the result of the most recent query available for further operations.

it is a bad idea to give an association a name that is already used for an instance method of ActiveRecord::Base. The association method would override the base method and break things.

belongs_to associations you need to create foreign keys

has_and_belongs_to_many associations you need to create the appropriate join table

So a join between customer and order models will give the default join table name of "customers_orders" because "c" outranks "o" in lexical ordering.

For example, one would expect the tables "paper_boxes" and "papers" to generate a join table name of "papers_paper_boxes" because of the length of the name "paper_boxes", but it in fact generates a join table name of "paper_boxes_papers" (because the underscore '' is lexicographically _less than 's' in common encodings).

id: false

pass id: false to create_table because that table does not represent a model

By default, associations look for objects only within the current module's scope.

will work fine, because both the Supplier and the Account class are defined within the same scope.

To associate a model with a model in a different namespace, you must specify the complete class name in your association declaration:

class_name

class_name

Active Record provides the :inverse_of option

Every association will attempt to automatically find the inverse association and set the :inverse_of option heuristically (based on the association name)

In database terms, this association says that this class contains the foreign key.

In all of these methods, association is replaced with the symbol passed as the first argument to belongs_to.

(force_reload = false)

The association method returns the associated object, if any. If no associated object is found, it returns nil.

the cached version will be returned.

The association= method assigns an associated object to this object.

Behind the scenes, this means extracting the primary key from the associate object and setting this object's foreign key to the same value.

The build_association method returns a new object of the associated type

The create_association method returns a new object of the associated type

raises ActiveRecord::RecordInvalid if the record is invalid.

dependent

counter_cache

:autosave :class_name :counter_cache :dependent :foreign_key :inverse_of :polymorphic :touch :validate

finding the number of belonging objects more efficient.

Although the :counter_cache option is specified on the model that includes the belongs_to declaration, the actual column must be added to the associated model.

:destroy, when the object is destroyed, destroy will be called on its associated objects.

The :inverse_of option specifies the name of the has_many or has_one association that is the inverse of this association

set the :touch option to :true, then the updated_at or updated_on timestamp on the associated object will be set to the current time whenever this object is saved or destroyed

specify a particular timestamp attribute to update

If you set the :validate option to true, then associated objects will be validated whenever you save this object

where includes readonly select

make your code somewhat more efficient

no need to use includes for immediate associations

will be read-only when retrieved via the association

The select method lets you override the SQL SELECT clause that is used to retrieve data about the associated object

using the association.nil?

In database terms, this association says that the other class contains the foreign key.

the cached version will be returned.

:as :autosave :class_name :dependent :foreign_key :inverse_of :primary_key :source :source_type :through :validate

Setting the :as option indicates that this is a polymorphic association

:nullify causes the foreign key to be set to NULL. Callbacks are not executed.

It's necessary not to set or leave :nullify option for those associations that have NOT NULL database constraints.

The :source_type option specifies the source association type for a has_one :through association that proceeds through a polymorphic association.

The :source option specifies the source association name for a has_one :through association.

The :through option specifies a join model through which to perform the query

more efficient by including representatives in the association from suppliers to accounts

When you assign an object to a has_one association, that object is automatically saved (in order to update its foreign key).

If either of these saves fails due to validation errors, then the assignment statement returns false and the assignment itself is cancelled.

If the parent object (the one declaring the has_one association) is unsaved (that is, new_record? returns true) then the child objects are not saved.

If you want to assign an object to a has_one association without saving the object, use the association.build method

collection(force_reload = false) collection<<(object, ...) collection.delete(object, ...) collection.destroy(object, ...) collection=(objects) collection_singular_ids collection_singular_ids=(ids) collection.clear collection.empty? collection.size collection.find(...) collection.where(...) collection.exists?(...) collection.build(attributes = {}, ...) collection.create(attributes = {}) collection.create!(attributes = {})

In all of these methods, collection is replaced with the symbol passed as the first argument to has_many, and collection_singular is replaced with the singularized version of that symbol.

The collection<< method adds one or more objects to the collection by setting their foreign keys to the primary key of the calling model

The collection.delete method removes one or more objects from the collection by setting their foreign keys to NULL.

objects will be destroyed if they're associated with dependent: :destroy, and deleted if they're associated with dependent: :delete_all

The collection.destroy method removes one or more objects from the collection by running destroy on each object.

The collection_singular_ids method returns an array of the ids of the objects in the collection.

The collection_singular_ids= method makes the collection contain only the objects identified by the supplied primary key values, by adding and deleting as appropriate

The default strategy for has_many :through associations is delete_all, and for has_many associations is to set the foreign keys to NULL.

The collection.clear method removes all objects from the collection according to the strategy specified by the dependent option

uses the same syntax and options as ActiveRecord::Base.find

The collection.where method finds objects within the collection based on the conditions supplied but the objects are loaded lazily meaning that the database is queried only when the object(s) are accessed.

The collection.build method returns one or more new objects of the associated type. These objects will be instantiated from the passed attributes, and the link through their foreign key will be created, but the associated objects will not yet be saved.

The collection.create method returns a new object of the associated type. This object will be instantiated from the passed attributes, the link through its foreign key will be created, and, once it passes all of the validations specified on the associated model, the associated object will be saved.

:as :autosave :class_name :dependent :foreign_key :inverse_of :primary_key :source :source_type :through :validate

:nullify causes the foreign keys to be set to NULL. Callbacks are not executed.

where includes readonly select

:conditions :through :polymorphic :foreign_key

By convention, Rails assumes that the column used to hold the primary key of the association is id. You can override this and explicitly specify the primary key with the :primary_key option.

The :source option specifies the source association name for a has_many :through association.

You only need to use this option if the name of the source association cannot be automatically inferred from the association name.

The :source_type option specifies the source association type for a has_many :through association that proceeds through a polymorphic association.

The :through option specifies a join model through which to perform the query.

has_many :through associations provide a way to implement many-to-many relationships,

By default, this is true: associated objects will be validated when this object is saved.

where extending group includes limit offset order readonly select uniq

If you use a hash-style where option, then record creation via this association will be automatically scoped using the hash

The extending method specifies a named module to extend the association proxy.

Association extensions

The group method supplies an attribute name to group the result set by, using a GROUP BY clause in the finder SQL.

more efficient by including line items in the association from customers to orders

The limit method lets you restrict the total number of objects that will be fetched through an association.

The offset method lets you specify the starting offset for fetching objects via an association

The order method dictates the order in which associated objects will be received (in the syntax used by an SQL ORDER BY clause).

Use the distinct method to keep the collection free of duplicates.

mostly useful together with the :through option

-> { distinct }

.all.inspect

If you want to make sure that, upon insertion, all of the records in the persisted association are distinct (so that you can be sure that when you inspect the association that you will never find duplicate records), you should add a unique index on the table itself

Do not attempt to use include? to enforce distinctness in an association.

When you assign an object to a has_many association, that object is automatically saved (in order to update its foreign key).

If any of these saves fails due to validation errors, then the assignment statement returns false and the assignment itself is cancelled.

If the parent object (the one declaring the has_many association) is unsaved (that is, new_record? returns true) then the child objects are not saved when they are added

assign an object to a has_many association without saving the object, use the collection.build method

collection(force_reload = false) collection<<(object, ...) collection.delete(object, ...) collection.destroy(object, ...) collection=(objects) collection_singular_ids collection_singular_ids=(ids) collection.clear collection.empty? collection.size collection.find(...) collection.where(...) collection.exists?(...) collection.build(attributes = {}) collection.create(attributes = {}) collection.create!(attributes = {})

If the join table for a has_and_belongs_to_many association has additional columns beyond the two foreign keys, these columns will be added as attributes to records retrieved via that association.

If you require this sort of complex behavior on the table that joins two models in a many-to-many relationship, you should use a has_many :through association instead of has_and_belongs_to_many.

The collection.delete method removes one or more objects from the collection by deleting records in the join table

The collection.destroy method removes one or more objects from the collection by running destroy on each record in the join table, including running callbacks.

The collection.clear method removes every object from the collection by deleting the rows from the joining table.

The collection.find method finds objects within the collection. It uses the same syntax and options as ActiveRecord::Base.find.

The collection.where method finds objects within the collection based on the conditions supplied but the objects are loaded lazily meaning that the database is queried only when the object(s) are accessed.

The collection.exists? method checks whether an object meeting the supplied conditions exists in the collection.

The collection.build method returns a new object of the associated type.

The collection.create method returns a new object of the associated type.

it passes all of the validations specified on the associated model

:association_foreign_key :autosave :class_name :foreign_key :join_table :validate

The :foreign_key and :association_foreign_key options are useful when setting up a many-to-many self-join.

Rails assumes that the column in the join table used to hold the foreign key pointing to the other model is the name of that model with the suffix _id added.

If you set the :autosave option to true, Rails will save any loaded members and destroy members that are marked for destruction whenever you save the parent object.

By convention, Rails assumes that the column in the join table used to hold the foreign key pointing to this model is the name of this model with the suffix _id added.

By default, this is true: associated objects will be validated when this object is saved.

where extending group includes limit offset order readonly select uniq

set conditions via a hash

In this case, using @parts.assemblies.create or @parts.assemblies.build will create orders where the factory column has the value "Seattle"

using a GROUP BY clause in the finder SQL.

Use the uniq method to remove duplicates from the collection.

assign an object to a has_and_belongs_to_many association, that object is automatically saved (in order to update the join table).

If any of these saves fails due to validation errors, then the assignment statement returns false and the assignment itself is cancelled.

If the parent object (the one declaring the has_and_belongs_to_many association) is unsaved (that is, new_record? returns true) then the child objects are not saved when they are added.

If you want to assign an object to a has_and_belongs_to_many association without saving the object, use the collection.build method.

Normal callbacks hook into the life cycle of Active Record objects, allowing you to work with those objects at various points

define association callbacks by adding options to the association declaration

stack callbacks on a single event by passing them as an array

If a before_add callback throws an exception, the object does not get added to the collection.

if a before_remove callback throws an exception, the object does not get removed from the collection

extend these objects through anonymous modules, adding new finders, creators, or other methods.

order_number

use a named extension module

proxy_association.owner returns the object that the association is a part of.

The resource endpoints should return representations of the resource as data, usually XML or JSON.

an existing API should never be modified, except for critical bugfixes

using unique database ids in the route chain allows users to access short routes, and simplifies resource lookup

require reauthentication on a per-request level

Defensive programming is a software design principle that dictates that a piece of software should be designed to continue functioning in unforeseen circumstances.

DI means that you can declare components very freely and then from any other component, just ask for an instance of it and it will be granted

do test-driven development iteratively in AngularJS!

only do DOM manipulation in a directive

with ngClass we can dynamically update the class;

ngBind allows two-way data binding;

ngShow and ngHide programmatically show or hide an element;

The less DOM manipulation, the easier directives are to test, the easier they are to style, the easier they are to change in the future, and the more re-usable and distributable they are.

still wrong.

Before doing DOM manipulation anywhere in your application, ask yourself if you really need to.

a few things wrong with this

jQuery was never necessary

use angular.element and our component will still work when dropped into a project that doesn't have jQuery.

just use angular.element

the element that is passed to the link function would already be a jQuery element!

directives aren't just collections of jQuery-like functions

Directives are actually extensions of HTML

If HTML doesn't do something you need it to do, you write a directive to do it for you, and then use it just as if it was part of HTML.

think how the team would accomplish it to fit right in with ngClick, ngClass, et al.

Don't even use jQuery. Don't even include it.

ry to think about how to do it within the confines the AngularJS.

In jQuery, selectors are used to find DOM elements and then bind/register event handlers to them.

Views are (declarative) HTML that contain AngularJS directives