Group items tagged

The lock file is always named .terraform.lock.hcl, and this name is intended to signify that it is a lock file for various items that Terraform caches in the .terraform

Terraform automatically creates or updates the dependency lock file each time you run the terraform init command.

If a particular provider has no existing recorded selection, Terraform will select the newest available version that matches the given version constraint, and then update the lock file to include that selection.

the "trust on first use" model

you can pre-populate checksums for a variety of different platforms in your lock file using the terraform providers lock command, which will then allow future calls to terraform init to verify that the packages available in your chosen mirror match the official packages from the provider's origin registry.

The h1: and zh: prefixes on these values represent different hashing schemes, each of which represents calculating a checksum using a different algorithm.

zh:: a mnemonic for "zip hash"

h1:: a mnemonic for "hash scheme 1", which is the current preferred hashing scheme.

To determine whether there still exists a dependency on a given provider, Terraform uses two sources of truth: the configuration itself, and the state.

Version constraints within the configuration itself determine which versions of dependencies are potentially compatible, but after selecting a specific version of each dependency Terraform remembers the decisions it made in a dependency lock file so that it can (by default) make the same decisions again in future.

The lock file is always named .terraform.lock.hcl

Developers rely heavily on app logs via syslog (mounted /dev/log) and Alpine uses busybox syslog by default.

Ubuntu officially launched minimal ubuntu images for cloud / container use

CMD instruction should be used to run the software contained by your image, along with any arguments

CMD should be given an interactive shell (bash, python, perl, etc)

COPY them individually, rather than all at once

COPY is preferred

using ADD to fetch packages from remote URLs is strongly discouraged

always use COPY

The best use for ENTRYPOINT is to set the image's main command, allowing that image to be run as though it was that command (and then use CMD as the default flags).

the image name can double as a reference to the binary as shown in the command above

ENTRYPOINT instruction can also be used in combination with a helper script

The VOLUME instruction should be used to expose any database storage area, configuration storage, or files/folders created by your docker container.

use USER to change to a non-root user

avoid installing or using sudo

avoid switching USER back and forth frequently.

always use absolute paths for your WORKDIR

ONBUILD is only useful for images that are going to be built FROM a given image

The “onbuild” image will fail catastrophically if the new build's context is missing the resource being added.

DevOps is a culture, a movement, a philosophy.

a firm handshake between development and operations

DevOps isn’t magic, and transformations don’t happen overnight.

Culture is the #1 success factor in DevOps.

Building a culture of shared responsibility, transparency and faster feedback is the foundation of every high performing DevOps team.

 'not our problem' mentality

DevOps is that change in mindset of looking at the development process holistically and breaking down the barrier between Dev and Ops.

Speed is everything.

Open communication helps Dev and Ops teams swarm on issues, fix incidents, and unblock the release pipeline faster.

Unplanned work is a reality that every team faces–a reality that most often impacts team productivity.

“cross-functional collaboration.”

All the tooling and automation in the world are useless if they aren’t accompanied by a genuine desire on the part of development and IT/Ops professionals to work together.

Forming project- or product-oriented teams to replace function-based teams is a step in the right direction.

sharing a common goal and having a plan to reach it together

join sprint planning sessions, daily stand-ups, and sprint demos.

DevOps culture across every department

open channels of communication, and talk regularly

automation eliminates repetitive manual work, yields repeatable processes, and creates reliable systems.

continuous delivery: the practice of running each code change through a gauntlet of automated tests, often facilitated by cloud-based infrastructure, then packaging up successful builds and promoting them up toward production using automated deploys.

automated deploys alert IT/Ops to server “drift” between environments, which reduces or eliminates surprises when it’s time to release.

when DevOps uses automated deploys to send thoroughly tested code to identically provisioned environments, “Works on my machine!” becomes irrelevant.

A DevOps mindset sees opportunities for continuous improvement everywhere.

regular retrospectives

A/B testing

failure is inevitable. So you might as well set up your team to absorb it, recover, and learn from it (some call this “being anti-fragile”).

Postmortems focus on where processes fell down and how to strengthen them – not on which team member f'ed up the code.

Our engineers are responsible for QA, writing, and running their own tests to get the software out to customers.

How long did it take to go from development to deployment? 

How long does it take to recover after a system failure?

service level agreements (SLAs)

DevOps is big on the idea that the same people who build an application should be involved in shipping and running it.

developers and operators pair with each other in each phase of the application’s lifecycle.

Workspaces persist data between jobs in a single Workflow.

Caching persists data between the same job in different Workflow builds.

run using the machine executor which enables reuse of recently used machine executor runs,

docker executor which can compose Docker containers to run your tests and any services they require

macos executor

Steps are a collection of executable commands which are run during a job

checkout: key is required to checkout your code

run: enables addition of arbitrary, multi-line shell command scripting

orchestrating job runs with parallel, sequential, and manual approval workflows.

Login to the container

Baseimage-docker only advocates running multiple OS processes inside a single container.

Password and challenge-response authentication are disabled by default. Only key authentication is allowed.

A tool for running a command as another user

The Docker developers advocate the philosophy of running a single logical service per container. A logical service can consist of multiple OS processes.

All syslog messages are forwarded to "docker logs".

Baseimage-docker advocates running multiple OS processes inside a single container, and a single logical service can consist of multiple OS processes.

Baseimage-docker provides tools to encourage running processes as different users

sometimes it makes sense to run multiple services in a single container, and sometimes it doesn't.

Splitting your logical service into multiple OS processes also makes sense from a security standpoint.

using environment variables to pass parameters to containers is very much the "Docker way"

the shell script must run the daemon without letting it daemonize/fork it.

All executable scripts in /etc/my_init.d, if this directory exists. The scripts are run in lexicographic order.

variables will also be passed to all child processes

Environment variables on Unix are inherited on a per-process basis

there is no good central place for defining environment variables for all applications and services

centrally defining environment variables

One of the ideas behind Docker is that containers should be stateless, easily restartable, and behave like a black box.

a one-shot command in a new container

immediately exit after the command exits,

add additional daemons (e.g. your own app) to the image by creating runit entries.

Nginx is one such example: it removes all environment variables unless you explicitly instruct it to retain them through the env configuration option.

Mechanisms for easily running multiple processes, without violating the Docker philosophy

Ubuntu is not designed to be run inside Docker

According to the Unix process model, the init process -- PID 1 -- inherits all orphaned child processes and must reap them

Syslog-ng seems to be much more stable

cron daemon

Rotates and compresses logs

/sbin/setuser

A tool for installing apt packages that automatically cleans up after itself.

A daemon is a program which runs in the background of its system, such as a web server.

The shell script must be called run, must be executable, and is to be placed in the directory /etc/service/<NAME>. runsv will switch to the directory and invoke ./run after your container starts.

If any script exits with a non-zero exit code, the booting will fail.

If your process is started with a shell script, make sure you exec the actual process, otherwise the shell will receive the signal and not your process.

any environment variables set with docker run --env or with the ENV command in the Dockerfile, will be picked up by my_init

they will not see the environment variables that were originally passed by Docker.

my_init imports environment variables from the directory /etc/container_environment

/etc/container_environment.sh - a dump of the environment variables in Bash format.

modify the environment variables in my_init (and therefore the environment variables in all child processes that are spawned after that point in time), by altering the files in /etc/container_environment

my_init only activates changes in /etc/container_environment when running startup scripts

environment variables don't contain sensitive data, then you can also relax the permissions

Syslog messages are forwarded to the console

syslog-ng is started separately before the runit supervisor process, and shutdown after runit exits.

/sbin/my_init --skip-startup-files --quiet --

By default, no keys are installed, so nobody can login

provide a pregenerated, insecure key (PuTTY format)

RUN /usr/sbin/enable_insecure_key

docker run YOUR_IMAGE /sbin/my_init --enable-insecure-key

RUN cat /tmp/your_key.pub >> /root/.ssh/authorized_keys && rm -f /tmp/your_key.pub

The default baseimage-docker installs syslog-ng, cron and sshd services during the build process

Commands are reusable sets of steps that you can invoke with specific parameters within an existing job.

you can pass my-executor as the value of a name key under executor. This method is primarily employed when passing parameters to executor invocations.

Development orbs are mutable and expire after 90 days.

Production Orbs are immutable and durable.

CircleCI allows development orbs that have versions that start with dev:

Production orbs are immutable

Each installation of CircleCI, including circleci.com, has only one registry where orbs can be kept.

Organization Admins publish production orbs.

Organization members publish development orbs

You must invoke jobs in the workflow stanza of config.yml file, making sure to pass any necessary parameters as subkeys to the job.

the abstraction of the infrastructure layer, which is now considered code. Deployment of a new application may require the deployment of new infrastructure code as well.

"big bang" deployments update whole or large parts of an application in one fell swoop.

Big bang deployments required the business to conduct extensive development and testing before release, often associated with the "waterfall model" of large sequential releases.

Rollbacks are often costly, time-consuming, or even impossible.

In a rolling deployment, an application’s new version gradually replaces the old one.

new and old versions will coexist without affecting functionality or user experience.

Each container is modified to download the latest image from the app vendor’s site.

two identical production environments work in parallel.

Once the testing results are successful, application traffic is routed from blue to green.

In a blue-green deployment, both systems use the same persistence layer or database back end.

You can use the primary database by blue for write operations and use the secondary by green for read operations.

Blue-green deployments rely on traffic routing.

long TTL values can delay these changes.

The main challenge of canary deployment is to devise a way to route some users to the new application.

Using an application logic to unlock new features to specific users and groups.

With CD, the CI-built code artifact is packaged and always ready to be deployed in one or more environments.

Use Build Automation tools to automate environment builds

Use configuration management tools

Enable automated rollbacks for deployments

An application performance monitoring (APM) tool can help your team monitor critical performance metrics including server response times after deployments.

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

for a lot of people, the name “Docker” itself is synonymous with the word “container”.

Docker created a very ergonomic (nice-to-use) tool for working with containers – also called docker.

docker is designed to be installed on a workstation or server and comes with a bunch of tools to make it easy to build and run containers as a developer, or DevOps person.

containerd: This is a daemon process that manages and runs containers.

runc: This is the low-level container runtime (the thing that actually creates and runs containers).

Kubernetes includes a component called dockershim, which allows it to support Docker.

Kubernetes will remove support for Docker directly, and prefer to use only container runtimes that implement its Container Runtime Interface.

Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.

Docker images, are actually images packaged in the Open Container Initiative (OCI) format.

CRI is the API that Kubernetes uses to control the different runtimes that create and manage containers.

CRI makes it easier for Kubernetes to use different container runtimes

containerd is a high-level container runtime that came from Docker, and implements the CRI spec

containerd was separated out of the Docker project, to make Docker more modular.

CRI-O is another high-level container runtime which implements the Container Runtime Interface (CRI).

The idea behind the OCI is that you can choose between different runtimes which conform to the spec.

runc is an OCI-compatible container runtime.

A reference implementation is a piece of software that has implemented all the requirements of a specification or standard.

runc provides all of the low-level functionality for containers, interacting with existing low-level Linux features, like namespaces and control groups.

If you return None, Errbot will not respond with any kind of message when executing the command.

a file that ends with the extension .plug and it is used by Errbot to identify and load plugins.

The key Module should point to a module that Python can find and import.

The key Name should be identical to the name you gave to the class in your plugin file

The presence of __init__.py indicates lib is a Python regular package.

the !status command,