Group items tagged

A Pod models an application-specific “logical host”

being executed on the same physical or virtual machine would mean being executed on the same logical host.

The shared context of a Pod is a set of Linux namespaces, cgroups, and potentially other facets of isolation

Containers in different Pods have distinct IP addresses and can not communicate by IPC without special configuration. These containers usually communicate with each other via Pod IP addresses.

a Pod is modelled as a group of Docker containers with shared namespaces and shared filesystem volumes

Pods are considered to be relatively ephemeral (rather than durable) entities.

Pods are created, assigned a unique ID (UID), and scheduled to nodes where they remain until termination (according to restart policy) or deletion.

it can be replaced by an identical Pod

When something is said to have the same lifetime as a Pod, such as a volume, that means that it exists as long as that Pod (with that UID) exists.

uses a persistent volume for shared storage between the containers

Pods serve as unit of deployment, horizontal scaling, and replication

flat shared networking space

Containers within the Pod see the system hostname as being the same as the configured name for the Pod.

Volumes enable data to survive container restarts and to be shared among the applications within the Pod.

Individual Pods are not intended to run multiple instances of the same application

The individual containers may be versioned, rebuilt and redeployed independently.

Controllers like StatefulSet can also provide support to stateful Pods.

When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container.

Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server.

grace period

Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers.

When the grace period expires, any processes still running in the Pod are killed with SIGKILL.

By default, all deletes are graceful within 30 seconds.

You must specify an additional flag --force along with --grace-period=0 in order to perform force deletions.

Force deletion of a Pod is defined as deletion of a Pod from the cluster state and etcd immediately.

Processes within the container get almost the same privileges that are available to processes outside a container.

the ELB also performs a vital role in improving the fault tolerance of the services which it fronts.

he Open Systems Interconnection Model, or OSI Model, is a conceptual model which is used to facilitate communications between different computing systems.

Layer 1 is the physical layer, and represents the physical medium across which the request is sent.

Layer 2 describes the data link layer

Layer 3 (the network layer)

Layer 7, which serves the application layer.

A network device, of which the Classic ELB is an example, reads the protocol and port of the incoming request, and then routes it to one or more backend servers.

Whereas a request to a specific URL backed by a Classic ELB would only enable routing to a particular pool of homogeneous servers, the ALB can route based on the content of the URL, and direct to a specific subgroup of backing servers existing in a heterogeneous collection registered with the load balancer.

The Classic ELB is a simple load balancer, is easy to configure

As organizations move towards microservice architecture or adopt a container-based infrastructure, the ability to merely map a single address to a specific service becomes more complicated and harder to maintain.

oute traffic to different services based on either the host or the content of the path contained within that URL.

choosing a leader for a distributed application without an internal member election process

publishing a Service to applications that don't support Kubernetes APIs to discover them

The core of the Operator is code to tell the API server how to make reality match the configured resources.

to deploy an Operator is to add the Custom Resource Definition and its associated Controller to your cluster.

Once you have an Operator deployed, you'd use it by adding, modifying or deleting the kind of resource that the Operator uses.

Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure.

a VE there is no preloaded emulation manager software as in a VM.

LXC will boast bare metal performance characteristics because it only packages the needed applications.

a VM, which packages the entire OS and machine setup, including hard drive, virtual processors and network interfaces. The resulting bloated mass usually takes a long time to boot and consumes a lot of CPU and RAM.

don’t offer some other neat features of VM’s such as IaaS setups and live migration.

LXC as supercharged chroot on Linux. It allows you to not only isolate applications, but even the entire OS.

Libvirt, which allows the use of containers through the LXC driver by connecting to 'lxc:///'.

'LXC', is not compatible with libvirt, but is more flexible with more userspace tools.

Portable deployment across machines

Versioning: Docker includes git-like capabilities for tracking successive versions of a container

Component reuse: Docker allows building or stacking of already created packages.

Shared libraries: There is already a public registry (http://index.docker.io/ ) where thousands have already uploaded the useful containers they have created.

Docker taking the devops world by storm since its launch back in 2013.

LXC, while older, has not been as popular with developers as Docker has proven to be

LXC having a focus on sys admins that’s similar to what solutions like the Solaris operating system, with its Solaris Zones, Linux OpenVZ, and FreeBSD, with its BSD Jails virtualization system

it started out being built on top of LXC, Docker later moved beyond LXC containers to its own execution environment called libcontainer.

LXC tooling sticks close to what system administrators running bare metal servers are used to

The LXC command line provides essential commands that cover routine management tasks, including the creation, launch, and deletion of LXC containers.

Docker containers aim to be even lighter weight in order to support the fast, highly scalable, deployment of applications with microservice architecture.

With backing from Canonical, LXC and LXD have an ecosystem tightly bound to the rest of the open source Linux community.

Docker Swarm

Docker Trusted Registry

Docker Compose

Docker Machine

Kubernetes facilitates the deployment of containers in your data center by representing a cluster of servers as a single system.

Swarm is Docker’s clustering, scheduling and orchestration tool for managing a cluster of Docker hosts. 

rkt is a security minded container engine that uses KVM for VM-based isolation and packs other enhanced security features. 

Apache Mesos can run different kinds of distributed jobs, including containers. 

Elastic Container Service is Amazon’s service for running and orchestrating containerized applications on AWS

LXC offers the advantages of a VE on Linux, mainly the ability to isolate your own private workloads from one another. It is a cheaper and faster solution to implement than a VM, but doing so requires a bit of extra learning and expertise.

for a lot of people, the name “Docker” itself is synonymous with the word “container”.

Docker created a very ergonomic (nice-to-use) tool for working with containers – also called docker.

docker is designed to be installed on a workstation or server and comes with a bunch of tools to make it easy to build and run containers as a developer, or DevOps person.

containerd: This is a daemon process that manages and runs containers.

runc: This is the low-level container runtime (the thing that actually creates and runs containers).

Kubernetes includes a component called dockershim, which allows it to support Docker.

Kubernetes will remove support for Docker directly, and prefer to use only container runtimes that implement its Container Runtime Interface.

Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.

Docker images, are actually images packaged in the Open Container Initiative (OCI) format.

CRI is the API that Kubernetes uses to control the different runtimes that create and manage containers.

CRI makes it easier for Kubernetes to use different container runtimes

containerd is a high-level container runtime that came from Docker, and implements the CRI spec

containerd was separated out of the Docker project, to make Docker more modular.

CRI-O is another high-level container runtime which implements the Container Runtime Interface (CRI).

The idea behind the OCI is that you can choose between different runtimes which conform to the spec.

runc is an OCI-compatible container runtime.

A reference implementation is a piece of software that has implemented all the requirements of a specification or standard.

runc provides all of the low-level functionality for containers, interacting with existing low-level Linux features, like namespaces and control groups.

/sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location.

an userspace library that processes can use to query their memory usage and available memory.

we might need to encourage people to stop using those tools inside containers.

Trunk-based development is a more open model since all developers have access to the main code. This enables teams to iterate quickly and implement CI/CD.

Developers can create short-lived branches with a few small commits compared to other long-lived feature branching strategies.

Gitflow is an alternative Git branching model that uses long-lived feature branches and multiple primary branches.

Gitflow also has separate primary branch lines for development, hotfixes, features, and releases.

Trunk-based development is far more simplified since it focuses on the main branch as the source of fixes and releases.

Trunk-based development eases the friction of code integration.

trunk-based development model reduces these conflicts.

Adding an automated test suite and code coverage monitoring for this stream of commits enables continuous integration.

With continuous integration, developers perform trunk-based development in conjunction with automated tests that run after each committee to a trunk.

Instead of creating a feature branch and waiting to build out the complete specification, developers can instead create a trunk commit that introduces the feature flag and pushes new trunk commits that build out the feature specification within the flag.

Short running unit and integration tests are executed during development and upon code merge.

Automated tests provide a layer of preemptive code review.

A repository with a large amount of active branches has some unfortunate side effects

Merge branches to the trunk at least once a day

The “continuous” in CI/CD implies that updates are constantly flowing.

Pods are immutable, which means that when they die, they are not resurrected. The Kubernetes cluster creates new pods in the same node or in a new node once a pod dies. 

For internal application access within a Kubernetes cluster, ClusterIP is the preferred method

Kubernetes Ingress is an API object that provides routing rules to manage external users' access to the services in a Kubernetes cluster, typically via HTTPS/HTTP.

content-based routing, support for multiple protocols, and authentication.

Ingress is made up of an Ingress API object and the Ingress Controller.

Kubernetes Ingress is an API object that describes the desired state for exposing services to the outside of the Kubernetes cluster.

An Ingress Controller reads and processes the Ingress Resource information and usually runs as pods within the Kubernetes cluster.  

If Kubernetes Ingress is the API object that provides routing rules to manage external access to services, Ingress Controller is the actual implementation of the Ingress API.

Layer 7 (L7) refers to the application level of the OSI stack—external connections load-balanced across pods, based on requests.

if Kubernetes Ingress is a computer, then Ingress Controller is a programmer using the computer and taking action.

the Ingress Controller is an application that runs in a Kubernetes cluster and configures an HTTP load balancer according to Ingress Resources.

ClusterIP is the preferred option for internal service access and uses an internal IP address to access the service

A NodePort is a virtual machine (VM) used to expose a service on a Static Port number.

a NodePort would be used to expose a single service (with no load-balancing requirements for multiple services).

Ingress enables you to consolidate the traffic-routing rules into a single resource and runs as part of a Kubernetes cluster.

An application is accessed from the Internet via Port 80 (HTTP) or Port 443 (HTTPS), and Ingress is an object that allows access to your Kubernetes services from outside the Kubernetes cluster. 

To implement Ingress, you need to configure an Ingress Controller in your cluster—it is responsible for processing Ingress Resource information and allowing traffic based on the Ingress Rules.

Big Bang releases

there is a lengthy suite of tests and checks that run before it is deployed to staging. During this period, which could end up being hours, engineers will likely pick up another task. I’ve seen people merge, and then forget that their changes are on staging, more times than I can count.

only merge code that is ready to go live

written sufficient tests and have validated our changes in development.

All branches are cut from main, and all changes get merged back into main.

If we ever have an issue in production, we always roll forward.

Feature flags can be enabled on a per-user basis so we can monitor performance and gather feedback

Experimental features can be enabled by users in their account settings.

we have monitoring, logging, and alarms around all of our services. We also blue/green deploy, by draining and replacing a percentage of containers.

Dropping your staging environment in favour of true continuous integration and deployment can create a different mindset for shipping software.

Pod-to-Service communications

External-to-Service communications

sharing machines requires ensuring that two applications do not try to use the same ports.

Dynamic port allocation brings a lot of complications to the system

do not need to explicitly create links between Pods

almost never need to deal with mapping container ports to host ports.

pods on a node can communicate with all pods on all nodes without NAT

agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node

pods in the host network of a node can communicate with all pods on all nodes without NAT

containers within a Pod share their network namespaces - including their IP address

containers within a Pod must coordinate port usage

“IP-per-pod” model.

request ports on the Node itself which forward to your Pod (called host ports), but this is a very niche operation

The Pod itself is blind to the existence or non-existence of host ports.

AOS is an Intent-Based Networking system that creates and manages complex datacenter environments from a simple integrated platform.

Cisco Application Centric Infrastructure offers an integrated overlay and underlay SDN solution that supports containers, virtual machines, and bare metal servers.

AOS Reference Design currently supports Layer-3 connected hosts that eliminate legacy Layer-2 switching problems.

The AWS VPC CNI offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters.

users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters.

Using this CNI plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.

The CNI allocates AWS Elastic Networking Interfaces (ENIs) to each Kubernetes node and using the secondary IP range from each ENI for pods on the node.

Big Cloud Fabric is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments.

Cilium is L7/HTTP aware and can enforce network policies on L3-L7 using an identity based security model that is decoupled from network addressing.

CNI-Genie is a CNI plugin that enables Kubernetes to simultaneously have access to different implementations of the Kubernetes network model in runtime.

CNI-Genie also supports assigning multiple IP addresses to a pod, each from a different CNI plugin.

cni-ipvlan-vpc-k8s contains a set of CNI and IPAM plugins to provide a simple, host-local, low latency, high throughput, and compliant networking stack for Kubernetes within Amazon Virtual Private Cloud (VPC) environments by making use of Amazon Elastic Network Interfaces (ENI) and binding AWS-managed IPs into Pods using the Linux kernel’s IPvlan driver in L2 mode.

to be straightforward to configure and deploy within a VPC

Contiv provides configurable networking

Contrail, based on Tungsten Fabric, is a truly open, multi-cloud network virtualization and policy management platform.

DANM is a networking solution for telco workloads running in a Kubernetes cluster.

Flannel is a very simple overlay network that satisfies the Kubernetes requirements.

Any traffic bound for that subnet will be routed directly to the VM by the GCE network fabric.

sysctl net.ipv4.ip_forward=1

Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.

Knitter is a network solution which supports multiple networking in Kubernetes.

Kube-OVN is an OVN-based kubernetes network fabric for enterprises.

Kube-router provides a Linux LVS/IPVS-based service proxy, a Linux kernel forwarding-based pod-to-pod networking solution with no overlays, and iptables/ipset-based network policy enforcer.

If you have a “dumb” L2 network, such as a simple switch in a “bare-metal” environment, you should be able to do something similar to the above GCE setup.

Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.

NSX-T can provide network virtualization for a multi-cloud and multi-hypervisor environment and is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks.

NSX-T Container Plug-in (NCP) provides integration between NSX-T and container orchestrators such as Kubernetes

Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.

OpenVSwitch is a somewhat more mature but also complicated way to build an overlay network

OVN is an opensource network virtualization solution developed by the Open vSwitch community.

Project Calico is an open source container networking provider and network policy engine.

Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet

Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking.

Romana is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network

Weave Net runs as a CNI plug-in or stand-alone. In either version, it doesn’t require any configuration or extra code to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.

The network model is implemented by the container runtime on each node.

Don't need to disable Docker's iptables and let Docker to manage it's network.

The public network cannot access ports that published by Docker.

In a very convenient way to allow/deny public networks to access container ports without additional software and extra configurations

Enable Docker's iptables feature. Remove all changes like --iptables=false , including configuration file /etc/docker/daemon.json

Modify the UFW configuration file /etc/ufw/after.rules

There may be some unknown reasons cause the UFW rules will not take effect after restart UFW, please reboot servers.

If we publish a port by using option -p 8080:80, we should use the container port 80, not the host port 8080

allow the private networks to be able to visit each other.

The following rules block connection requests initiated by all public networks, but allow internal networks to access external networks.

Since the UDP protocol is stateless, it is not possible to block the handshake signal that initiates the connection request as TCP does.

For GNU/Linux we can find the local port range in the file /proc/sys/net/ipv4/ip_local_port_range. The default range is 32768 60999

It not only exposes ports of containers but also exposes ports of the host.

Cannot expose services running on hosts and containers at the same time by the same command.

軟體是達到目標的手段，而不是目標本身。目標是定向的，但手段要可隨時變通。

不需要維護的程式

10 倍工程師是愚蠢神話。有人能在 1 天內完成另一個有能力、努力工作、有類似經驗的工程師在 2 週內完成的工作，這種想法很愚蠢。

。一個人能夠成為 10 倍工程師的唯一方法，是你把他們和 0.1x 的工程師相比。

between a senior engineer and a junior engineer is that they’ve formed opinions about the way things should be

沒有什麼比一個對自己的工具或如何建構軟體沒有意見的資深工程師更讓我擔心了。

如果你正在使用你的工具，而你對它們不愛也不恨，那麼你需要去體驗更多。

你的資料可能會比你的程式長壽。花點精力保持秩序和乾淨

軟體工程師應該定期撰寫部落格、日記、文件，總之做任何需要他們保持書面溝通能力之事。

如果你把一個人從他們的工作成果中抽離，他們就會對他們的工作不那麼關心。這也是跨職能團隊運作良好的主要原因，也是 DevOps 變得如此流行的原因。從頭到尾擁有整個過程，並直接負責交付價值。讓一群充滿熱情的人對設計、建構和交付一個軟體 (或任何東西) 擁有完全的自主，這終將發生令人驚奇之事。

面試最好是用來瞭解某人是誰，以及他們對某一專業領域的興趣如何。嘗試找出他們會成為多好的團隊成員是一個沒有結果的努力。

你在建構一個系統的過程中會學到很多東西，你最終會迭代成一個比你當初設計的更好的系統。

stop sharpening the saw, and just start cutting shit