Group items tagged

DevOps is a culture, a movement, a philosophy.

a firm handshake between development and operations

DevOps isn’t magic, and transformations don’t happen overnight.

Culture is the #1 success factor in DevOps.

Building a culture of shared responsibility, transparency and faster feedback is the foundation of every high performing DevOps team.

 'not our problem' mentality

DevOps is that change in mindset of looking at the development process holistically and breaking down the barrier between Dev and Ops.

Speed is everything.

Open communication helps Dev and Ops teams swarm on issues, fix incidents, and unblock the release pipeline faster.

Unplanned work is a reality that every team faces–a reality that most often impacts team productivity.

“cross-functional collaboration.”

All the tooling and automation in the world are useless if they aren’t accompanied by a genuine desire on the part of development and IT/Ops professionals to work together.

Forming project- or product-oriented teams to replace function-based teams is a step in the right direction.

sharing a common goal and having a plan to reach it together

join sprint planning sessions, daily stand-ups, and sprint demos.

DevOps culture across every department

open channels of communication, and talk regularly

automation eliminates repetitive manual work, yields repeatable processes, and creates reliable systems.

continuous delivery: the practice of running each code change through a gauntlet of automated tests, often facilitated by cloud-based infrastructure, then packaging up successful builds and promoting them up toward production using automated deploys.

automated deploys alert IT/Ops to server “drift” between environments, which reduces or eliminates surprises when it’s time to release.

when DevOps uses automated deploys to send thoroughly tested code to identically provisioned environments, “Works on my machine!” becomes irrelevant.

A DevOps mindset sees opportunities for continuous improvement everywhere.

regular retrospectives

A/B testing

failure is inevitable. So you might as well set up your team to absorb it, recover, and learn from it (some call this “being anti-fragile”).

Postmortems focus on where processes fell down and how to strengthen them – not on which team member f'ed up the code.

Our engineers are responsible for QA, writing, and running their own tests to get the software out to customers.

How long did it take to go from development to deployment? 

How long does it take to recover after a system failure?

service level agreements (SLAs)

DevOps is big on the idea that the same people who build an application should be involved in shipping and running it.

developers and operators pair with each other in each phase of the application’s lifecycle.

If the directive block is left empty, you should omit the curly braces entirely.

Arguments that contain whitespace must be enclosed in quotes ".

To configure multiple sites with a single Caddyfile, you must use curly braces around each one to separate their configurations:

he opening curly brace must be at the end of the same line.

The closing curly brace must be on its own line.

All directives must appear inside a site's definition.

Site addresses can also be defined under a specific path or have wildcards in place of individual domain labels from the left side

using a path in your site address will route requests by longest matching prefix.

Use of environment variables is allowed in addresses and arguments. They must be enclosed in curly braces,

you may not specify the same site address more than once

a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

To use a volume, a Pod specifies what volumes to provide for the Pod (the .spec.volumes field) and where to mount those into Containers (the .spec.containers.volumeMounts field).

Each Container in the Pod must independently specify where to mount each volume

localnfs

cephfs

awsElasticBlockStore

glusterfs

vsphereVolume

An awsElasticBlockStore volume mounts an Amazon Web Services (AWS) EBS Volume into your Pod.

an EBS volume can be pre-populated with data, and that data can be “handed off” between Pods.

create an EBS volume using aws ec2 create-volume

the nodes on which Pods are running must be AWS EC2 instances

EBS only supports a single EC2 instance mounting a volume

check that the size and EBS volume type are suitable for your use!

A cephfs volume allows an existing CephFS volume to be mounted into your Pod.

the contents of a cephfs volume are preserved and the volume is merely unmounted.

have your own Ceph server running with the share exported

The configMap resource provides a way to inject configuration data into Pods

When referencing a configMap object, you can simply provide its name in the volume to reference it

volumeMounts: - name: config-vol mountPath: /etc/config volumes: - name: config-vol configMap: name: log-config items: - key: log_level path: log_level

create a ConfigMap before you can use it.

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node.

By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment.

you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem)

volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: {}

An fc volume allows an existing fibre channel volume to be mounted in a Pod.

configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

Flocker is an open-source clustered Container data volume manager. It provides management and orchestration of data volumes backed by a variety of storage backends.

emptyDir

flocker

A flocker volume allows a Flocker dataset to be mounted into a Pod

have your own Flocker installation running

A gcePersistentDisk volume mounts a Google Compute Engine (GCE) Persistent Disk into your Pod.

Using a PD on a Pod controlled by a ReplicationController will fail unless the PD is read-only or the replica count is 0 or 1

A glusterfs volume allows a Glusterfs (an open source networked filesystem) volume to be mounted into your Pod.

have your own GlusterFS installation running

a powerful escape hatch for some applications

access to Docker internals; use a hostPath of /var/lib/docker

allowing a Pod to specify whether a given hostPath should exist prior to the Pod running, whether it should be created, and what it should exist as

specify a type for a hostPath volume

the files or directories created on the underlying hosts are only writable by root.

hostPath: # directory location on host path: /data # this field is optional type: Directory

An iscsi volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.

have your own iSCSI server running

A feature of iSCSI is that it can be mounted as read-only by multiple consumers simultaneously.

A local volume represents a mounted local storage device such as a disk, partition or directory.

Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume’s node constraints by looking at the node affinity on the PersistentVolume.

PersistentVolume spec using a local volume and nodeAffinity

PersistentVolume volumeMode can now be set to “Block” (instead of the default value “Filesystem”) to expose the local volume as a raw block device.

When using local volumes, it is recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer

An nfs volume allows an existing NFS (Network File System) share to be mounted into your Pod.

have your own NFS server running with the share exported

A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.

PersistentVolumes are a way for users to “claim” durable storage (such as a GCE PersistentDisk or an iSCSI volume) without knowing the details of the particular cloud environment.

A projected volume maps several existing volume sources into the same directory.

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Each projected volume source is listed in the spec under sources

A Container using a projected volume source as a subPath volume mount will not receive updates for those volume sources.

RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed

A secret volume is used to pass sensitive information, such as passwords, to Pods

create a secret in the Kubernetes API before you can use it

StorageOS runs as a Container within your Kubernetes environment, making local or attached storage accessible from any node within the Kubernetes cluster.

StorageOS provides block storage to Containers, accessible via a file system.

A vsphereVolume is used to mount a vSphere VMDK Volume into your Pod.

supports both VMFS and VSAN datastore.

create VMDK using one of the following methods before using with Pod.

share one volume for multiple uses in a single Pod.

volumeMounts: - name: workdir1 mountPath: /logs subPathExpr: $(POD_NAME)

env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name

Use the subPathExpr field to construct subPath directory names from Downward API environment variables

enable the VolumeSubpathEnvExpansion feature gate

emptyDir and hostPath volumes will be able to request a certain amount of space using a resource specification, and to select the type of media to use, for clusters that have several media types.

the Container Storage Interface (CSI) and Flexvolume. They enable storage vendors to create custom storage plugins without adding them to the Kubernetes repository.

Container Storage Interface (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the csi volume type to attach, mount, etc. the volumes exposed by the CSI driver.

This feature requires CSIInlineVolume feature gate to be enabled:--feature-gates=CSIInlineVolume=true

In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the “Types of Volumes” section above.

Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts.

HostToContainer - This volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories.

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the Container will be propagated back to the host and to all Containers of all Pods that use the same volume.

Edit your Docker’s systemd service file. Set MountFlags as follows:MountFlags=shared

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure.

a VE there is no preloaded emulation manager software as in a VM.

LXC will boast bare metal performance characteristics because it only packages the needed applications.

a VM, which packages the entire OS and machine setup, including hard drive, virtual processors and network interfaces. The resulting bloated mass usually takes a long time to boot and consumes a lot of CPU and RAM.

don’t offer some other neat features of VM’s such as IaaS setups and live migration.

LXC as supercharged chroot on Linux. It allows you to not only isolate applications, but even the entire OS.

Libvirt, which allows the use of containers through the LXC driver by connecting to 'lxc:///'.

'LXC', is not compatible with libvirt, but is more flexible with more userspace tools.

Portable deployment across machines

Versioning: Docker includes git-like capabilities for tracking successive versions of a container

Component reuse: Docker allows building or stacking of already created packages.

Shared libraries: There is already a public registry (http://index.docker.io/ ) where thousands have already uploaded the useful containers they have created.

Docker taking the devops world by storm since its launch back in 2013.

LXC, while older, has not been as popular with developers as Docker has proven to be

LXC having a focus on sys admins that’s similar to what solutions like the Solaris operating system, with its Solaris Zones, Linux OpenVZ, and FreeBSD, with its BSD Jails virtualization system

it started out being built on top of LXC, Docker later moved beyond LXC containers to its own execution environment called libcontainer.

LXC tooling sticks close to what system administrators running bare metal servers are used to

The LXC command line provides essential commands that cover routine management tasks, including the creation, launch, and deletion of LXC containers.

Docker containers aim to be even lighter weight in order to support the fast, highly scalable, deployment of applications with microservice architecture.

With backing from Canonical, LXC and LXD have an ecosystem tightly bound to the rest of the open source Linux community.

Docker Swarm

Docker Trusted Registry

Docker Compose

Docker Machine

Kubernetes facilitates the deployment of containers in your data center by representing a cluster of servers as a single system.

Swarm is Docker’s clustering, scheduling and orchestration tool for managing a cluster of Docker hosts. 

rkt is a security minded container engine that uses KVM for VM-based isolation and packs other enhanced security features. 

Apache Mesos can run different kinds of distributed jobs, including containers. 

Elastic Container Service is Amazon’s service for running and orchestrating containerized applications on AWS

LXC offers the advantages of a VE on Linux, mainly the ability to isolate your own private workloads from one another. It is a cheaper and faster solution to implement than a VM, but doing so requires a bit of extra learning and expertise.

for a lot of people, the name “Docker” itself is synonymous with the word “container”.

Docker created a very ergonomic (nice-to-use) tool for working with containers – also called docker.

docker is designed to be installed on a workstation or server and comes with a bunch of tools to make it easy to build and run containers as a developer, or DevOps person.

containerd: This is a daemon process that manages and runs containers.

runc: This is the low-level container runtime (the thing that actually creates and runs containers).

Kubernetes includes a component called dockershim, which allows it to support Docker.

Kubernetes will remove support for Docker directly, and prefer to use only container runtimes that implement its Container Runtime Interface.

Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.

Docker images, are actually images packaged in the Open Container Initiative (OCI) format.

CRI is the API that Kubernetes uses to control the different runtimes that create and manage containers.

CRI makes it easier for Kubernetes to use different container runtimes

containerd is a high-level container runtime that came from Docker, and implements the CRI spec

containerd was separated out of the Docker project, to make Docker more modular.

CRI-O is another high-level container runtime which implements the Container Runtime Interface (CRI).

The idea behind the OCI is that you can choose between different runtimes which conform to the spec.

runc is an OCI-compatible container runtime.

A reference implementation is a piece of software that has implemented all the requirements of a specification or standard.

runc provides all of the low-level functionality for containers, interacting with existing low-level Linux features, like namespaces and control groups.

docker build --cache-from "$CI_REGISTRY_IMAGE:latest" -t "$CI_REGISTRY_IMAGE:new-tag"

But if you maintain a CHANGELOG in this format, and/or your Git tags are also your Docker tags, you can get the previous version and use cache the this image version.

script: - export PREVIOUS_VERSION=$(perl -lne 'print "v${1}" if /^##\s\[(\d\.\d\.\d)\]\s-\s\d{4}(?:-\d{2}){2}\s*$/' CHANGELOG.md | sed -n '2 p') - docker build --cache-from "$CI_REGISTRY_IMAGE:$PREVIOUS_VERSION" -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG" -f ./prod.Dockerfile .

« Docker layer caching » is enough to optimize the build time.

We're building a Docker image, dependencies are installed inside a container.We can't cache a dependencies directory if it doesn't exists in the job workspace.

Dependencies will always be installed from a container but will be extracted by the GitLab Runner in the job workspace. Our goal is to send the cached version in the build context.

- docker cp app:/var/www/html/vendor/ ./vendor

after_script

- docker cp app:/var/www/html/node_modules/ ./node_modules

To avoid old dependencies to be mixed with the new ones, at the risk of keeping unused dependencies in cache, which would make cache and images heavier.

If you need to cache directories in testing jobs, it's easier: use volumes !

version your cache keys !

sharing Docker image between jobs

In every job, we automatically get artifacts from previous stages.

docker save $DOCKER_CI_IMAGE | gzip > app.tar.gz

I personally use the « push / pull » technique,