Group items tagged

MongoDB stores documents in collections.

MongoDB provides the db.createCollection() method to explicitly create a collection with various options, such as setting the maximum size or the documentation validation rules.

To change the structure of the documents in a collection, such as add new fields, remove existing fields, or change the field values to a new type, update the documents to the new structure.

Collections are assigned an immutable UUID.

To retrieve the UUID for a collection, run either the listCollections command or the db.getCollectionInfos() method.

This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.

differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.

Filebeat is just a tiny binary with no dependencies.

For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

For example, the apache module will point Filebeat to default access.log and error.log paths

Filebeat’s scope is very limited,

Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Filebeat can parse JSON

you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.

You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off

For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well

The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.

rsyslog is the fastest shipper

Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).

It’s also one of the lightest parsers you can find, depending on the configured memory buffers.

rsyslog requires more work to get the configuration right

the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.

rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).

rsyslog also works well when you need that ultimate performance.

syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).

a modular syslog daemon, that can do much more than just syslog

Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.

Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.

syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance

Fluentd plugins are in Ruby and very easy to write.

structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).

Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.

Splunk isn’t a log shipper, it’s a commercial logging solution

everything goes through graylog-server, from authentication to queries.

Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.

it depends

deployment.yaml: A basic manifest for creating a Kubernetes deployment

using the suffix .yaml for YAML files and .tpl for helpers.

helm get manifest

The helm get manifest command takes a release name (full-coral) and prints out all of the Kubernetes resources that were uploaded to the server. Each file begins with --- to indicate the start of a YAML document

Names should be unique to a release

The name: field is limited to 63 characters because of limitations to the DNS system.

release names are limited to 53 characters

{{ .Release.Name }}

The values that are passed into a template can be thought of as namespaced objects, where a dot (.) separates each namespaced element.

The Release object is one of the built-in objects for Helm

Using --dry-run will make it easier to test your code, but it won’t ensure that Kubernetes itself will accept the templates you generate.

Objects are passed into a template from the template engine.

create new objects within your templates

Objects can be simple, and have just one value. Or they can contain other objects or functions.

Release is one of the top-level objects that you can access in your templates.

Release.Namespace: The namespace to be released into (if the manifest doesn’t override)

Chart: The contents of the Chart.yaml file.

Files: This provides access to all non-special files in a chart.

Files.Get is a function for getting a file by name

Files.GetBytes is a function for getting the contents of a file as an array of bytes instead of as a string. This is useful for things like images.

Template: Contains information about the current template that is being executed

BasePath: The namespaced path to the templates directory of the current chart

Go’s naming convention

use only initial lower case letters in order to distinguish local names from those built-in.

If this is a subchart, the values.yaml file of a parent chart

Individual parameters passed with --set

While structuring data this way is possible, the recommendation is that you keep your values trees shallow, favoring flatness.

If you need to delete a key from the default values, you may override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.

Kubernetes would then fail because you can not declare more than one livenessProbe handler.

When injecting strings from the .Values object into the template, we ought to quote these strings.

quote

Template functions follow the syntax functionName arg1 arg2...

While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates.

Drawing on a concept from UNIX, pipelines are a tool for chaining together a series of template commands to compactly express a series of transformations.

pipelines are an efficient way of getting several things done in sequence

The repeat function will echo the given string the given number of times

default DEFAULT_VALUE GIVEN_VALUE. This function allows you to specify a default value inside of the template, in case the value is omitted.

all static default values should live in the values.yaml, and should not be repeated using the default command

Operators are implemented as functions that return a boolean value.

To use eq, ne, lt, gt, and, or, not etcetera place the operator at the front of the statement followed by its parameters just as you would a function.

if and

if or

with to specify a scope

range, which provides a “for each”-style loop

block declares a special kind of fillable template area

A pipeline is evaluated as false if the value is: a boolean false a numeric zero an empty string a nil (empty or null) an empty collection (map, slice, tuple, dict, array)

incorrect YAML because of the whitespacing

When the template engine runs, it removes the contents inside of {{ and }}, but it leaves the remaining whitespace exactly as is.

{{- (with the dash and space added) indicates that whitespace should be chomped left, while -}} means whitespace to the right should be consumed.

Newlines are whitespace!

an * at the end of the line indicates a newline character that would be removed

the indent function

Scopes can be changed. with can allow you to set the current scope (.) to a particular object.

range

The range function will “range over” (iterate through) the pizzaToppings list.

Just like with sets the scope of ., so does a range operator.

The toppings: |- line is declaring a multi-line string.

not a YAML list. It’s a big string.

the data in ConfigMaps data is composed of key/value pairs, where both the key and the value are simple strings.

The |- marker in YAML takes a multi-line string.

range can be used to iterate over collections that have a key and a value (like a map or dict).

In Helm templates, a variable is a named reference to another object. It follows the form $name

Variables are assigned with a special assignment operator: :=

{{- $relname := .Release.Name -}}

capture both the index and the value

the integer index (starting from zero) to $index and the value to $topping

For data structures that have both a key and a value, we can use range to get both

one variable that is always global - $ - this variable will always point to the root context.

$.

$.

Helm template language is its ability to declare multiple templates and use them together.

A named template (sometimes called a partial or a subtemplate) is simply a template defined inside of a file, and given a name.

If you declare two templates with the same name, whichever one is loaded last will be the one used.

naming convention is to prefix each defined template with the name of the chart: {{ define "mychart.labels" }}

docker build --cache-from "$CI_REGISTRY_IMAGE:latest" -t "$CI_REGISTRY_IMAGE:new-tag"

But if you maintain a CHANGELOG in this format, and/or your Git tags are also your Docker tags, you can get the previous version and use cache the this image version.

script: - export PREVIOUS_VERSION=$(perl -lne 'print "v${1}" if /^##\s\[(\d\.\d\.\d)\]\s-\s\d{4}(?:-\d{2}){2}\s*$/' CHANGELOG.md | sed -n '2 p') - docker build --cache-from "$CI_REGISTRY_IMAGE:$PREVIOUS_VERSION" -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG" -f ./prod.Dockerfile .

« Docker layer caching » is enough to optimize the build time.

We're building a Docker image, dependencies are installed inside a container.We can't cache a dependencies directory if it doesn't exists in the job workspace.

Dependencies will always be installed from a container but will be extracted by the GitLab Runner in the job workspace. Our goal is to send the cached version in the build context.

- docker cp app:/var/www/html/vendor/ ./vendor

after_script

- docker cp app:/var/www/html/node_modules/ ./node_modules

To avoid old dependencies to be mixed with the new ones, at the risk of keeping unused dependencies in cache, which would make cache and images heavier.

If you need to cache directories in testing jobs, it's easier: use volumes !

version your cache keys !

sharing Docker image between jobs

In every job, we automatically get artifacts from previous stages.

docker save $DOCKER_CI_IMAGE | gzip > app.tar.gz

I personally use the « push / pull » technique,