You can use role-based access control
(RBAC) and other
security mechanisms to make sure that users and workloads can get access to the
resources they need, while keeping workloads, and the cluster itself, secure.
You can set limits on the resources that users and workloads can access
by managing policies and
container resources.
you need to plan how to scale to relieve increased
pressure from more requests to the control plane and worker nodes or scale down to reduce unused
resources.
Managed control plane: Let the provider manage the scale and availability
of the cluster's control plane, as well as handle patches and upgrades.
The simplest Kubernetes cluster has the entire control plane and worker node
services running on the same machine.
You can deploy a control plane using tools such
as kubeadm, kops, and kubespray.
Secure communications between control plane services
are implemented using certificates.
Certificates are automatically generated
during deployment or you can generate them using your own certificate authority.
Separate and backup etcd service: The etcd services can either run on the
same machines as other control plane services or run on separate machines
Create multiple control plane systems: For high availability, the
control plane should not be limited to a single machine
Some deployment tools set up Raft
consensus algorithm to do leader election of Kubernetes services. If the
primary goes away, another service elects itself and take over.
Groups of zones are referred to as regions.
if you installed with kubeadm, there are instructions to help you with
Certificate Management
and Upgrading kubeadm clusters.
Production-quality workloads need to be resilient and anything they rely
on needs to be resilient (such as CoreDNS).
Add nodes to the cluster: If you are managing your own cluster you can
add nodes by setting up your own machines and either adding them manually or
having them register themselves to the cluster’s apiserver.
Set up node health checks: For important workloads, you want to make sure
that the nodes and pods running on those nodes are healthy.
Authentication: The apiserver can authenticate users using client
certificates, bearer tokens, an authenticating proxy, or HTTP basic auth.
Authorization: When you set out to authorize your regular users, you will probably choose
between RBAC and ABAC authorization.
Role-based access control (RBAC): Lets you
assign access to your cluster by allowing specific sets of permissions to authenticated users.
Permissions can be assigned for a specific namespace (Role) or across the entire cluster
(ClusterRole).
Attribute-based access control (ABAC): Lets you
create policies based on resource attributes in the cluster and will allow or deny access
based on those attributes.
Set limits on workload resources
Set namespace limits: Set per-namespace quotas on things like memory and CPU
Prepare for DNS demand: If you expect workloads to massively scale up,
your DNS service must be ready to scale up as well.
run one or two control plane instances per failure zone,
scaling those instances vertically first and then scaling horizontally after reaching
the point of falling returns to (vertical) scale.
Kubernetes
nodes do not automatically steer traffic towards control-plane endpoints that are in the
same failure zone
store Event objects in a separate
dedicated etcd instance.
start and configure additional etcd instance
Kubernetes resource limits
help to minimize the impact of memory leaks and other ways that pods and containers can
impact on other components.
Addons' default limits are typically based on data collected from experience running
each addon on small or medium Kubernetes clusters.
When running on large
clusters, addons often consume more of some resources than their default limits.
Many addons scale horizontally - you add capacity by running more pods
The VerticalPodAutoscaler can run in recommender mode to provide suggested
figures for requests and limits.
Some addons run as one copy per node, controlled by a DaemonSet: for example, a node-level log aggregator.
VerticalPodAutoscaler is a custom resource that you can deploy into your cluster
to help you manage resource requests and limits for pods.
The cluster autoscaler
integrates with a number of cloud providers to help you run the right number of
nodes for the level of resource demand in your cluster.
The addon resizer
helps you in resizing the addons automatically as your cluster's scale changes.
"Summary: On Heroku, Rails scales fairly easily, but there are some important things to consider. We looked at how various dyno and Postgres settings effect overall performance on Heroku."
"Pinpoint is an open source APM (Application Performance Management) tool for large-scale distributed systems written in Java. http://naver.github.io/pinpoint/"
RoleBindings per namespace enable to restrict granted permissions to the very namespaces only that Traefik is watching over, thereby following the least-privileges principle.
The scalability can be much better when using a Deployment
you will have a Single-Pod-per-Node model when using a DaemonSet,
DaemonSets automatically scale to new nodes, when the nodes join the cluster
DaemonSets ensure that only one replica of pods run on any single node.
DaemonSets can be run with the NET_BIND_SERVICE capability, which will allow it to bind to port 80/443/etc on each host. This will allow bypassing the kube-proxy, and reduce traffic hops.
start with the Daemonset
The Deployment has easier up and down scaling possibilities.
The DaemonSet automatically scales to all nodes that meets a specific selector and guarantees to fill nodes one at a time.
Rolling updates are fully supported from Kubernetes 1.7 for DaemonSets as well.
provide the TLS certificate via a Kubernetes secret in the same namespace as the ingress.
If there are any errors while loading the TLS section of an ingress, the whole ingress will be skipped.
create secret generic
Name-based Routing
Path-based Routing
Traefik will merge multiple Ingress definitions for the same host/path pair into one definition.
specify priority for ingress routes
traefik.frontend.priority
When specifying an ExternalName,
Traefik will forward requests to the given host accordingly and use HTTPS when the Service port matches 443.
By default Traefik will pass the incoming Host header to the upstream resource.
traefik.frontend.passHostHeader: "false"
type: ExternalName
By default, Traefik processes every Ingress objects it observes.
It is also possible to set the ingressClass option in Traefik to a particular value. Traefik will only process matching Ingress objects.
It is possible to split Ingress traffic in a fine-grained manner between multiple deployments using service weights.
use case is canary releases where a deployment representing a newer release is to receive an initially small but ever-increasing fraction of the requests over time.
Over time, the ratio may slowly shift towards the canary deployment until it is deemed to replace the previous main application, in steps such as 5%/95%, 10%/90%, 50%/50%, and finally 100%/0%.
Your project will continue to use an alternative
CI/CD configuration file if one is found
Auto DevOps works with any Kubernetes cluster;
using the
Docker or Kubernetes
executor, with
privileged mode enabled.
Base domain (needed for Auto Review Apps and Auto Deploy)
Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)
Prometheus (needed for Auto Monitoring)
scrape your Kubernetes cluster.
project level as a variable: KUBE_INGRESS_BASE_DOMAIN
A wildcard DNS A record matching the base domain(s) is required
Once set up, all requests will hit the load balancer, which in turn will route
them to the Kubernetes pods that run your application(s).
review/ (every environment starting with review/)
staging
production
need to define a separate
KUBE_INGRESS_BASE_DOMAIN variable for all the above
based on the environment.
Continuous deployment to production: Enables Auto Deploy
with master branch directly deployed to production.
Continuous deployment to production using timed incremental rollout
Automatic deployment to staging, manual deployment to production
Auto Build creates a build of the application using an existing Dockerfile or
Heroku buildpacks.
If a project’s repository contains a Dockerfile, Auto Build will use
docker build to create a Docker image.
Each buildpack requires certain files to be in your project’s repository for
Auto Build to successfully build your application.
Auto Test automatically runs the appropriate tests for your application using
Herokuish and Heroku
buildpacks by analyzing
your project to detect the language and framework.
Auto Code Quality uses the
Code Quality image to run
static analysis and other code checks on the current code.
Static Application Security Testing (SAST) uses the
SAST Docker image to run static
analysis on the current code and checks for potential security issues.
Dependency Scanning uses the
Dependency Scanning Docker image
to run analysis on the project dependencies and checks for potential security issues.
License Management uses the
License Management Docker image
to search the project dependencies for their license.
Vulnerability Static Analysis for containers uses
Clair to run static analysis on a
Docker image and checks for potential security issues.
Review Apps are temporary application environments based on the
branch’s code so developers, designers, QA, product managers, and other
reviewers can actually see and interact with code changes as part of the review
process. Auto Review Apps create a Review App for each branch.
Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster
is available, no deployment will occur.
The Review App will have a unique URL based on the project ID, the branch or tag
name, and a unique number, combined with the Auto DevOps base domain.
Review apps are deployed using the
auto-deploy-app chart with
Helm, which can be customized.
Your apps should not be manipulated outside of Helm (using Kubernetes directly).
Dynamic Application Security Testing (DAST) uses the
popular open source tool OWASP ZAProxy
to perform an analysis on the current code and checks for potential security
issues.
Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.
add the paths to a file named .gitlab-urls.txt in the root directory, one per line.
After a branch or merge request is merged into the project’s default branch (usually
master), Auto Deploy deploys the application to a production environment in
the Kubernetes cluster, with a namespace based on the project name and unique
project ID
Auto Deploy doesn’t include deployments to staging or canary by default, but the
Auto DevOps template contains job definitions for these tasks if you want to
enable them.
Apps are deployed using the
auto-deploy-app chart with
Helm.
For internal and private projects a GitLab Deploy Token
will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.
If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is
used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.
If present, DB_INITIALIZE will be run as a shell command within an
application pod as a helm post-install hook.
a post-install hook means that if any deploy succeeds,
DB_INITIALIZE will not be processed thereafter.
DB_MIGRATE will be run as a shell command within an application pod as
a helm pre-upgrade hook.
Once your application is deployed, Auto Monitoring makes it possible to monitor
your application’s server and response metrics right out of the box.
annotate
the NGINX Ingress deployment to be scraped by Prometheus using
prometheus.io/scrape: "true" and prometheus.io/port: "10254"
If you are also using Auto Review Apps and Auto Deploy and choose to provide
your own Dockerfile, make sure you expose your application to port
5000 as this is the port assumed by the
default Helm chart.
While Auto DevOps provides great defaults to get you started, you can customize
almost everything to fit your needs; from custom buildpacks,
to Dockerfiles, Helm charts, or
even copying the complete CI/CD configuration
into your project to enable staging and canary deployments, and more.
If your project has a Dockerfile in the root of the project repo, Auto DevOps
will build a Docker image based on the Dockerfile rather than using buildpacks.
Auto DevOps uses Helm to deploy your application to Kubernetes.
Bundled chart - If your project has a ./chart directory with a Chart.yaml
file in it, Auto DevOps will detect the chart and use it instead of the default
one.
Create a project variable
AUTO_DEVOPS_CHART with the URL of a custom chart to use or create two project variables AUTO_DEVOPS_CHART_REPOSITORY with the URL of a custom chart repository and AUTO_DEVOPS_CHART with the path to the chart.
make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.
specify the use of a custom Helm chart per environment by scoping the environment variable
to the desired environment.
Your additions will be merged with the
Auto DevOps template using the behaviour described for
include
copy and paste the contents of the Auto DevOps
template into your project and edit this as needed.
In order to support applications that require a database,
PostgreSQL is provisioned by default.
Set up the replica variables using a
project variable
and scale your application by just redeploying it!
You should not scale your application using Kubernetes directly.
Some applications need to define secret variables that are
accessible by the deployed application.
Auto DevOps detects variables where the key starts with
K8S_SECRET_ and make these prefixed variables available to the
deployed application, as environment variables.
Auto DevOps pipelines will take your application secret variables to
populate a Kubernetes secret.
Environment variables are generally considered immutable in a Kubernetes
pod.
if you update an application secret without changing any
code then manually create a new pipeline, you will find that any running
application pods will not have the updated secrets.
Variables with multiline values are not currently supported
The normal behavior of Auto DevOps is to use Continuous Deployment, pushing
automatically to the production environment every time a new pipeline is run
on the default branch.
If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to
1 as a CI/CD variable), then the application will be automatically deployed
to a staging environment, and a production_manual job will be created for
you when you’re ready to manually deploy to production.
If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to
1 as a CI/CD variable) then two manual jobs will be created:
canary which will deploy the application to the canary environment
production_manual which is to be used by you when you’re ready to manually
deploy to production.
If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead
of the standard production job, 4 different
manual jobs
will be created:
rollout 10%
rollout 25%
rollout 50%
rollout 100%
The percentage is based on the REPLICAS variable and defines the number of
pods you want to have for your deployment.
To start a job, click on the play icon next to the job’s name.
Once you get to 100%, you cannot scale down, and you’d have to roll
back by redeploying the old version using the
rollback button in the
environment page.
With INCREMENTAL_ROLLOUT_MODE set to manual and with STAGING_ENABLED
not all buildpacks support Auto Test yet
When a project has been marked as private, GitLab’s Container
Registry requires authentication when downloading
containers.
Authentication credentials will be valid while the pipeline is running, allowing
for a successful initial deployment.
After the pipeline completes, Kubernetes will no longer be able to access the
Container Registry.
We strongly advise using GitLab Container Registry with Auto DevOps in order to
simplify configuration and prevent any unforeseen issues.
"This tool helps you to generate beautiful blurry background images that you can use in any project. It doesn't use CSS3 gradients, but a rather unique approach. It takes a stock image, extracts a very small area (sample area) and scales it up to 100%. The browser's image smoothing algorithm takes care of the rest.
You can then use the image as an inline, base64 encoded image in any HTML element's background, just click Generate CSS button at the bottom of the app. Select source images from the gallery or use yours, the possibilities are endless."
"Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The text format (a superset of JSON) is easy to read and author, supporting rapid prototyping. The binary representation is efficient to store, transmit, and skip-scan parse. The rich type system provides unambiguous semantics for long-term preservation of business data which can survive multiple generations of software evolution.
Ion was built to solve the rapid development, decoupling, and efficiency challenges faced every day while engineering large-scale, service-oriented architectures. Ion has been addressing these challenges within Amazon for nearly a decade, and we believe others will benefit as well.
"
"CocoaPods is the dependency manager for Swift and Objective-C Cocoa projects. It has over ten thousand libraries and can help you scale your projects elegantly."
When you run Docker without using swarm mode, you execute container
commands.
When you run the Docker in swarm mode, you orchestrate services.
You can run swarm services and standalone containers on the same Docker instances.
A node is an instance of the Docker engine participating in the swarm
You can run one or more nodes on a single physical computer or cloud server
To deploy your application to a swarm, you submit a service definition to a
manager node.
Manager nodes also perform the orchestration and cluster management functions
required to maintain the desired state of the swarm.
Manager nodes elect a
single leader to conduct orchestration tasks.
Worker nodes receive and execute tasks dispatched from manager nodes.
service is the definition of the tasks to execute on the worker nodes
When you create a service, you specify which container image to use and which
commands to execute inside running containers.
replicated services model, the swarm manager distributes a specific
number of replica tasks among the nodes based upon the scale you set in the
desired state.
global services, the swarm runs one task for the service on every
available node in the cluster.
A task carries a Docker container and the commands to run inside the
container
Manager nodes assign tasks
to worker nodes according to the number of replicas set in the service scale.
Once a task is assigned to a node, it cannot move to another node
If you do not
specify a port, the swarm manager assigns the service a port in the 30000-32767
range.
External components, such as cloud load balancers, can access the service on the
PublishedPort of any node in the cluster whether or not the node is currently
running the task for the service.
Swarm mode has an internal DNS component that automatically assigns each service
in the swarm a DNS entry.
The control plane's components make global decisions about the cluster
Control plane components can be run on any machine in the cluster.
for simplicity, set up scripts typically start all control plane components on
the same machine, and do not run user containers on this machine
The API server is the front end for the Kubernetes control plane.
kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances.
You can run several instances of kube-apiserver and balance traffic between those instances.
Kubernetes cluster uses etcd as its backing store, make sure you have a
back up plan
for those data.
watches for newly created
Pods with no assigned
node, and selects a node for them
to run on.
Factors taken into account for scheduling decisions include:
individual and collective resource requirements, hardware/software/policy
constraints, affinity and anti-affinity specifications, data locality,
inter-workload interference, and deadlines.
each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.
Node controller
Job controller
Endpoints controller
Service Account & Token controllers
The cloud controller manager lets you link your
cluster into your cloud provider's API, and separates out the components that interact
with that cloud platform from components that only interact with your cluster.
If you are running Kubernetes on your own premises, or in a learning environment inside your
own PC, the cluster does not have a cloud controller manager.
An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.
The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.
The kubelet doesn't manage containers which were not created by Kubernetes.
kube-proxy is a network proxy that runs on each
node in your cluster,
implementing part of the Kubernetes
Service concept.
kube-proxy
maintains network rules on nodes. These network rules allow network
communication to your Pods from network sessions inside or outside of
your cluster.
kube-proxy uses the operating system packet filtering layer if there is one
and it's available.
Kubernetes supports several container runtimes: Docker,
containerd, CRI-O,
and any implementation of the Kubernetes CRI (Container Runtime
Interface).
Addons use Kubernetes resources (DaemonSet,
Deployment, etc)
to implement cluster features
namespaced resources
for addons belong within the kube-system namespace.
all Kubernetes clusters should have cluster DNS,
Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.
Containers started by Kubernetes automatically include this DNS server in their DNS searches.
Container Resource Monitoring records generic time-series metrics
about containers in a central database, and provides a UI for browsing that data.
A cluster-level logging mechanism is responsible for
saving container logs to a central log store with search/browsing interface.
"Project Fonos is open-source telecommunications for the cloud. It helps VoIP integrators quickly deploy new networks and benefit from value-added services such as Programmable Voice, Messaging, and Video. This repository assembles the various components needed to deploy a telephony system at scale."
MinIO is purpose-built to take full advantage of the Kubernetes architecture.
MinIO and Kubernetes work together to simplify infrastructure management, providing a way to manage object storage infrastructure within the Kubernetes toolset.
The operator pattern extends Kubernetes's familiar declarative API model with custom resource definitions (CRDs) to perform common operations like resource orchestration, non-disruptive upgrades, cluster expansion and to maintain high-availability
The Operator uses the command set kubectl that the Kubernetes community was already familiar with and adds the kubectl minio plugin . The MinIO Operator and the MinIO kubectl plugin facilitate the deployment and management of MinIO Object Storage on Kubernetes - which is how multi-tenant object storage as a service is delivered.
choosing a leader for a distributed application without an internal member election process
The Operator Console makes Kubernetes object storage easier still. In this graphical user interface, MinIO created something so simple that anyone in the organization can create, deploy and manage object storage as a service.
The primary unit of managing MinIO on Kubernetes is the tenant.
The MinIO Operator can allocate multiple tenants within the same Kubernetes cluster.
Each tenant, in turn, can have different capacity (i.e: a small 500GB tenant vs a 100TB tenant), resources (1000m CPU and 4Gi RAM vs 4000m CPU and 16Gi RAM) and servers (4 pods vs 16 pods), as well a separate configurations regarding Identity Providers, Encryption and versions.
each tenant is a cluster of server pools (independent sets of nodes with their own compute, network, and storage resources), that, while sharing the same physical infrastructure, are fully isolated from each other in their own namespaces.
Each tenant runs their own MinIO cluster, fully isolated from other tenants
Each tenant scales independently by federating clusters across geographies.
Microservices also bring a set of additional benefits, such as easier scaling, the possibility to use multiple programming languages and technologies, and others.
Java is a frequent choice for building a microservices architecture as it is a mature language tested over decades and has a multitude of microservices-favorable frameworks, such as legendary Spring, Jersey, Play, and others.
A monolithic architecture keeps it all simple. An app has just one server and one database.
All the connections between units are inside-code calls.
split our application into microservices and got a set of units completely independent for deployment and maintenance.
Each of microservices responsible for a certain business function communicates either via sync HTTP/REST or async AMQP protocols.
ensure seamless communication between newly created distributed components.
The gateway became an entry point for all clients’ requests.
We also set the Zuul 2 framework for our gateway service so that the application could leverage the benefits of non-blocking HTTP calls.
we've implemented the Eureka server as our server discovery that keeps a list of utilized user profile and order servers to help them discover each other.
We also have a message broker (RabbitMQ) as an intermediary between the notification server and the rest of the servers to allow async messaging in-between.
microservices can definitely help when it comes to creating complex applications that deal with huge loads and need continuous improvement and scaling.
Serverless was first used to describe applications that significantly or fully
depend on 3rd party applications / services (‘in the cloud’) to manage server-side
logic and state.
‘rich client’ applications (think single page
web apps, or mobile apps) that use the vast ecosystem of cloud accessible
databases (like Parse, Firebase), authentication services (Auth0, AWS Cognito),
etc.
Serverless can also mean applications where some amount of server-side logic
is still written by the application developer but unlike traditional architectures
is run in stateless compute containers that are event-triggered, ephemeral (may
only last for one invocation), and fully managed by a 3rd party.
‘Functions as a service
AWS Lambda is one of the most popular implementations of FaaS at present,
A good example is
Auth0 - they started initially with BaaS ‘Authentication
as a Service’, but with Auth0 Webtask they are entering the
FaaS space.
a typical ecommerce app
a backend data-processing service
with zero administration.
FaaS offerings do not require coding to a specific framework or
library.
Horizontal scaling is completely automatic, elastic, and managed by the
provider
Functions in FaaS are triggered by event types defined by the provider.
a FaaS-supported message broker
from a
deployment-unit point of view FaaS functions are stateless.
allowed the client direct access to a
subset of our database
deleted the authentication logic in the original application and have
replaced it with a third party BaaS service
The client is in fact well on its way to becoming a Single Page Application.
implement a FaaS function that responds to http requests via an
API Gateway
port the search code from the Pet Store server to the Pet Store Search
function
replaced a long lived consumer application with a
FaaS function that runs within the event driven context
server
applications - is a key difference when comparing with other modern
architectural trends like containers and PaaS
the only code that needs to
change when moving to FaaS is the ‘main method / startup’ code, in that it is
deleted, and likely the specific code that is the top-level message handler
(the ‘message listener interface’ implementation), but this might only be a change
in method signature
With FaaS you need to write the function ahead of time to assume parallelism
Most providers also allow functions to be triggered as a response to inbound
http requests, typically in some kind of API gateway
you should assume that for any given
invocation of a function none of the in-process or host state that you create
will be available to any subsequent invocation.
FaaS
functions are either naturally stateless
store
state across requests or for further input to handle a request.
certain classes of long lived task are not suited to FaaS
functions without re-architecture
if you were writing a
low-latency trading application you probably wouldn’t want to use FaaS systems
at this time
An
API Gateway is an HTTP server where routes / endpoints are defined in
configuration and each route is associated with a FaaS function.
API
Gateway will allow mapping from http request parameters to inputs arguments
for the FaaS function
API Gateways may also perform authentication, input validation,
response code mapping, etc.
the Serverless Framework makes working
with API Gateway + Lambda significantly easier than using the first principles
provided by AWS.
Apex - a project to
‘Build, deploy, and manage AWS Lambda functions with ease.'
'Serverless'
to mean the union of a couple of other ideas - 'Backend as a Service' and
'Functions as a Service'.