Rails 4 automatically adds the sass-rails, coffee-rails and uglifier
gems to your Gemfile
reduce the number of requests that a browser makes to render a web page
Starting with version 3.1, Rails defaults to concatenating all JavaScript files into one master .js file and all CSS files into one master .css file
In production, Rails inserts an MD5 fingerprint into each filename so that the file is cached by the web browser
The technique sprockets uses for fingerprinting is to insert a hash of the
content into the name, usually at the end.
asset minification or compression
The sass-rails gem is automatically used for CSS compression if included
in Gemfile and no config.assets.css_compressor option is set.
Supported languages include Sass for CSS, CoffeeScript for JavaScript, and ERB for both by default.
When a filename is unique and based on its content, HTTP headers can be set to encourage caches everywhere (whether at CDNs, at ISPs, in networking equipment, or in web browsers) to keep their own copy of the content
asset pipeline is technically no longer a core feature of Rails 4
Rails uses for fingerprinting is to insert a hash of the content into the name, usually at the end
With the asset pipeline, the preferred location for these assets is now the app/assets directory.
Fingerprinting is enabled by default for production and disabled for all other
environments
The files in app/assets are never served directly in production.
Paths are traversed in the order that they occur in the search path
You should use app/assets for
files that must undergo some pre-processing before they are served.
By default .coffee and .scss files will not be precompiled on their own
app/assets is for assets that are owned by the application, such as custom images, JavaScript files or stylesheets.
lib/assets is for your own libraries' code that doesn't really fit into the scope of the application or those libraries which are shared across applications.
vendor/assets is for assets that are owned by outside entities, such as code for JavaScript plugins and CSS frameworks.
Any path under assets/* will be searched
By default these files will be ready to use by your application immediately using the require_tree directive.
By default, this means the files in app/assets take precedence, and will mask corresponding paths in lib and vendor
Sprockets uses files named index (with the relevant extensions) for a special purpose
Rails.application.config.assets.paths
causes turbolinks to check if
an asset has been updated and if so loads it into the page
if you add an erb extension to a CSS asset (for example, application.css.erb), then helpers like asset_path are available in your CSS rules
If you add an erb extension to a JavaScript asset, making it something such as application.js.erb, then you can use the asset_path helper in your JavaScript code
The asset pipeline automatically evaluates ERB
data URI — a method of embedding the image data directly into the CSS file — you can use the asset_data_uri helper.
Sprockets will also look through the paths specified in config.assets.paths,
which includes the standard application paths and any paths added by Rails
engines.
image_tag
the closing tag cannot be of the style -%>
asset_data_uri
app/assets/javascripts/application.js
sass-rails provides -url and -path helpers (hyphenated in Sass,
underscored in Ruby) for the following asset classes: image, font, video, audio,
JavaScript and stylesheet.
Rails.application.config.assets.compress
In JavaScript files, the directives begin with //=
The require_tree directive tells Sprockets to recursively include all JavaScript files in the specified directory into the output.
manifest files contain directives — instructions that tell Sprockets which files to require in order to build a single CSS or JavaScript file.
You should not rely on any particular order among those
Sprockets uses manifest files to determine which assets to include and serve.
the family of require directives prevents files from being included twice in the output
which files to require in order to build a single CSS or JavaScript file
Directives are processed top to bottom, but the order in which files are included by require_tree is unspecified.
In JavaScript files, Sprockets directives begin with //=
If require_self is called more than once, only the last call is respected.
require
directive is used to tell Sprockets the files you wish to require.
You need not supply the extensions explicitly.
Sprockets assumes you are requiring a .js file when done from within a .js
file
paths must be
specified relative to the manifest file
require_directory
Rails 4 creates both app/assets/javascripts/application.js and
app/assets/stylesheets/application.css regardless of whether the
--skip-sprockets option is used when creating a new rails application.
The file extensions used on an asset determine what preprocessing is applied.
app/assets/stylesheets/application.css
Additional layers of preprocessing can be requested by adding other extensions, where each extension is processed in a right-to-left manner
require_self
use the Sass @import rule
instead of these Sprockets directives.
Keep in mind that the order of these preprocessors is important
In development mode, assets are served as separate files in the order they are specified in the manifest file.
when these files are
requested they are processed by the processors provided by the coffee-script
and sass gems and then sent back to the browser as JavaScript and CSS
respectively.
css.scss.erb
js.coffee.erb
Keep in mind the order of these preprocessors is important.
By default Rails assumes that assets have been precompiled and will be served as static assets by your web server
with the Asset Pipeline the :cache and :concat options aren't used anymore
Assets are compiled and cached on the first request after the server is started
Putting this information in a secret
is safer and more flexible than putting it verbatim in a
PodThe smallest and simplest Kubernetes object. A Pod represents a set of running containers on your cluster. definition or in a container imageStored instance of a container that holds a set of software needed to run an application.
.
A Secret is an object that contains a small amount of sensitive data such as
a password, a token, or a key.
Users can create secrets, and the system also creates some secrets.
To use a secret, a pod needs to reference the secret.
A secret can be used with a pod in two ways: as files in a
volumeA directory containing data, accessible to the containers in a pod. mounted on one or more of
its containers, or used by kubelet when pulling images for the pod.
--from-file
You can also create a Secret in a file first, in json or yaml format,
and then create that object.
The
Secret contains two maps:
data and stringData.
The data field is used to store arbitrary data, encoded using
base64.
Kubernetes automatically creates secrets which contain credentials for
accessing the API and it automatically modifies your pods to use this type of
secret.
kubectl get and kubectl describe avoid showing the contents of a secret by
default.
stringData field is provided for convenience, and allows you to provide
secret data as unencoded strings.
where you are deploying an application
that uses a Secret to store a configuration file, and you want to populate
parts of that configuration file during your deployment process.
a field is specified in both data and stringData, the value from stringData
is used.
The keys of data and stringData must consist of alphanumeric characters,
‘-’, ‘_’ or ‘.’.
Newlines are not valid within these strings and must
be omitted.
When using the base64 utility on Darwin/macOS users should avoid
using the -b option to split long lines.
create a Secret from generators and then apply it to create the object on
the Apiserver.
The generated Secrets name has a suffix appended by hashing the contents.
base64 --decode
Secrets can be mounted as data volumes or be exposed as
environment variablesContainer environment variables are name=value pairs that provide useful information into containers running in a Pod.
to be used by a container in a pod.
Multiple pods can reference the same secret.
Each key in the secret data map becomes the filename under mountPath
each container needs its
own volumeMounts block, but only one .spec.volumes is needed per secret
use .spec.volumes[].secret.items field to change target path of each key:
If .spec.volumes[].secret.items is used, only keys specified in items are projected.
To consume all keys from the secret, all of them must be listed in the items field.
You can also specify the permission mode bits files part of a secret will have.
If you don’t specify any, 0644 is used by default.
JSON spec doesn’t support octal notation, so use the value 256 for
0400 permissions.
Inside the container that mounts a secret volume, the secret keys appear as
files and the secret values are base-64 decoded and stored inside these files.
Mounted Secrets are updated automatically
Kubelet is checking whether the mounted secret is fresh on every periodic sync.
cache propagation delay depends on the chosen cache type
A container using a Secret as a
subPath volume mount will not receive
Secret updates.
Inside a container that consumes a secret in an environment variables, the secret keys appear as
normal environment variables containing the base-64 decoded values of the secret data.
An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry
password to the Kubelet so it can pull a private image on behalf of your Pod.
a secret
needs to be created before any pods that depend on it.
Secret API objects reside in a namespaceAn abstraction used by Kubernetes to support multiple virtual clusters on the same physical cluster.
.
They can only be referenced by pods in that same namespace.
Individual secrets are limited to 1MiB in size.
Kubelet only supports use of secrets for Pods it gets from the API server.
Secrets must be created before they are consumed in pods as environment
variables unless they are marked as optional.
References to Secrets that do
not exist will prevent the pod from starting.
References via secretKeyRef to keys that do not exist in a named Secret
will prevent the pod from starting.
Once a pod is scheduled, the kubelet will try to fetch the
secret value.
Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret.
Special characters such as $, \*, and ! require escaping.
If the password you are using has special characters, you need to escape them using the \\ character.
You do not need to escape special characters in passwords from files
make that key begin with a dot
Dotfiles in secret volume
.secret-file
a frontend container
which handles user interaction and business logic, but which cannot see the
private key;
a signer container that can see the private key, and responds
to simple signing requests from the frontend
When deploying applications that interact with the secrets API, access should be
limited using authorization policies such as RBAC
watch and list requests for secrets within a namespace are
extremely powerful capabilities and should be avoided
watch and list all secrets in a cluster should be reserved for only the most
privileged, system-level components.
additional
precautions with secret objects, such as avoiding writing them to disk where
possible.
A secret is only sent to a node if a pod on that node requires it
only the
secrets that a pod requests are potentially visible within its containers
each container in a pod has
to request the secret volume in its volumeMounts for it to be visible within
the container.
In the API server secret data is stored in etcdConsistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.
limit access to etcd to admin users
Base64 encoding is not an
encryption method and is considered the same as plain text.
A user who can create a pod that uses a secret can also see the value of that secret.
anyone with root on any node can read any secret from the apiserver,
by impersonating the kubelet.
A chart is a collection of files
that describe a related set of Kubernetes resources.
A single chart
might be used to deploy something simple, like a memcached pod, or
something complex, like a full web app stack with HTTP servers,
databases, caches, and so on.
Charts are created as files laid out in a particular directory tree,
then they can be packaged into versioned archives to be deployed.
A chart is organized as a collection of files inside of a directory.
values.yaml # The default configuration values for this chart
charts/ # A directory containing any charts upon which this chart depends.
templates/ # A directory of templates that, when combined with values,
# will generate valid Kubernetes manifest files.
version: A SemVer 2 version (required)
apiVersion: The chart API version, always "v1" (required)
Every chart must have a version number. A version must follow the
SemVer 2 standard.
non-SemVer names are explicitly
disallowed by the system.
When generating a
package, the helm package command will use the version that it finds
in the Chart.yaml as a token in the package name.
the appVersion field is not related to the version field. It is
a way of specifying the version of the application.
appVersion: The version of the app that this contains (optional). This needn't be SemVer.
If the latest version of a chart in the
repository is marked as deprecated, then the chart as a whole is considered to
be deprecated.
deprecated: Whether this chart is deprecated (optional, boolean)
one chart may depend on any number of other charts.
dependencies can be dynamically linked through the requirements.yaml
file or brought in to the charts/ directory and managed manually.
the preferred method of declaring dependencies is by using a
requirements.yaml file inside of your chart.
A requirements.yaml file is a simple file for listing your
dependencies.
The repository field is the full URL to the chart repository.
you must also use helm repo add to add that repo locally.
helm dependency update
and it will use your dependency file to download all the specified
charts into your charts/ directory for you.
When helm dependency update retrieves charts, it will store them as
chart archives in the charts/ directory.
Managing charts with requirements.yaml is a good way to easily keep
charts updated, and also share requirements information throughout a
team.
All charts are loaded by default.
The condition field holds one or more YAML paths (delimited by commas).
If this path exists in the top parent’s values and resolves to a boolean value,
the chart will be enabled or disabled based on that boolean value.
The tags field is a YAML list of labels to associate with this chart.
all charts with tags can be enabled or disabled by
specifying the tag and a boolean value.
The --set parameter can be used as usual to alter tag and condition values.
Conditions (when set in values) always override tags.
The first condition path that exists wins and subsequent ones for that chart are ignored.
The keys containing the values to be imported can be specified in the parent chart’s requirements.yaml file
using a YAML list. Each item in the list is a key which is imported from the child chart’s exports field.
specifying the key data in our import list, Helm looks in the exports field of the child
chart for data key and imports its contents.
the parent key data is not contained in the parent’s final values. If you need to specify the
parent key, use the ‘child-parent’ format.
To access values that are not contained in the exports key of the child chart’s values, you will need to
specify the source key of the values to be imported (child) and the destination path in the parent chart’s
values (parent).
To drop a dependency into your charts/ directory, use the
helm fetch command
A dependency can be either a chart archive (foo-1.2.3.tgz) or an
unpacked chart directory.
name cannot start with _ or ..
Such files are ignored by the chart loader.
a single release is created with all the objects for the chart and its dependencies.
Helm Chart templates are written in the
Go template language, with the
addition of 50 or so add-on template
functions from the Sprig library and a
few other specialized functions
When
Helm renders the charts, it will pass every file in that directory
through the template engine.
Chart developers may supply a file called values.yaml inside of a
chart. This file can contain default values.
Chart users may supply a YAML file that contains values. This can be
provided on the command line with helm install.
When a user supplies custom values, these values will override the
values in the chart’s values.yaml file.
Template files follow the standard conventions for writing Go templates
{{default "minio" .Values.storage}}
Values that are supplied via a values.yaml file (or via the --set
flag) are accessible from the .Values object in a template.
pre-defined, are available to every template, and
cannot be overridden
the names are case
sensitive
Release.Name: The name of the release (not the chart)
Release.IsUpgrade: This is set to true if the current operation is an upgrade or rollback.
Release.Revision: The revision number. It begins at 1, and increments with
each helm upgrade
Chart: The contents of the Chart.yaml
Files: A map-like object containing all non-special files in the chart.
Files can be
accessed using {{index .Files "file.name"}} or using the {{.Files.Get name}} or
{{.Files.GetString name}} functions.
.helmignore
access the contents of the file
as []byte using {{.Files.GetBytes}}
Any unknown Chart.yaml fields will be dropped
Chart.yaml cannot be
used to pass arbitrarily structured data into the template.
A values file is formatted in YAML.
A chart may include a default
values.yaml file
be merged into the default
values file.
The default values file included inside of a chart must be named
values.yaml
accessible inside of templates using the
.Values object
Values files can declare values for the top-level chart, as well as for
any of the charts that are included in that chart’s charts/ directory.
Charts at a higher level have access to all of the variables defined
beneath.
lower level charts cannot access things in
parent charts
Values are namespaced, but namespaces are pruned.
the scope of the values has been reduced and the
namespace prefix removed
Helm supports special “global” value.
a way of sharing one top-level variable with all
subcharts, which is useful for things like setting metadata properties
like labels.
If a subchart declares a global variable, that global will be passed
downward (to the subchart’s subcharts), but not upward to the parent
chart.
global variables of parent charts take precedence over the global variables from subcharts.
helm lint
A chart repository is an HTTP server that houses one or more packaged
charts
Any HTTP server that can serve YAML files and tar files and can answer
GET requests can be used as a repository server.
Helm does not provide tools for uploading charts to
remote repository servers.
the only way to add a chart to $HELM_HOME/starters is to manually
copy it there.
Helm provides a hook mechanism to allow chart developers to intervene
at certain points in a release’s life cycle.
Execute a Job to back up a database before installing a new chart,
and then execute a second job after the upgrade in order to restore
data.
Hooks are declared as an annotation in the metadata section of a manifest
Hooks work like regular templates, but they have special annotations
pre-install
post-install: Executes after all resources are loaded into Kubernetes
pre-delete
post-delete: Executes on a deletion request after all of the release’s
resources have been deleted.
pre-upgrade
post-upgrade
pre-rollback
post-rollback: Executes on a rollback request after all resources
have been modified.
crd-install
test-success: Executes when running helm test and expects the pod to
return successfully (return code == 0).
test-failure: Executes when running helm test and expects the pod to
fail (return code != 0).
Hooks allow you, the chart developer, an opportunity to perform
operations at strategic points in a release lifecycle
Tiller then loads the hook with the lowest weight first (negative to positive)
Tiller returns the release name (and other data) to the client
If the resources is a Job kind, Tiller
will wait until the job successfully runs to completion.
if the job
fails, the release will fail. This is a blocking operation, so the
Helm client will pause while the Job is run.
If they
have hook weights (see below), they are executed in weighted order. Otherwise,
ordering is not guaranteed.
good practice to add a hook weight, and set it
to 0 if weight is not important.
The resources that a hook creates are not tracked or managed as part of the
release.
leave the hook resource alone.
To destroy such
resources, you need to either write code to perform this operation in a pre-delete
or post-delete hook or add "helm.sh/hook-delete-policy" annotation to the hook template file.
Hooks are just Kubernetes manifest files with special annotations in the
metadata section
One resource can implement multiple hooks
no limit to the number of different resources that
may implement a given hook.
When subcharts declare hooks, those are also evaluated. There is no way
for a top-level chart to disable the hooks declared by subcharts.
Hook weights can be positive or negative numbers but must be represented as
strings.
sort those hooks in ascending order.
Hook deletion policies
"before-hook-creation" specifies Tiller should delete the previous hook before the new hook is launched.
By default Tiller will wait for 60 seconds for a deleted hook to no longer exist in the API server before timing out.
Custom Resource Definitions (CRDs) are a special kind in Kubernetes.
The crd-install hook is executed very early during an installation, before
the rest of the manifests are verified.
A common reason why the hook resource might already exist is that it was not deleted following use on a previous install/upgrade.
Helm uses Go templates for templating
your resource files.
two special template functions: include and required
include
function allows you to bring in another template, and then pass the results to other
template functions.
The required function allows you to declare a particular
values entry as required for template rendering.
If the value is empty, the template
rendering will fail with a user submitted error message.
When you are working with string data, you are always safer quoting the
strings than leaving them as bare words
Quote Strings, Don’t Quote Integers
when working with integers do not quote the values
env variables values which are expected to be string
to include a template, and then perform an operation
on that template’s output, Helm has a special include function
The above includes a template called toYaml, passes it $value, and
then passes the output of that template to the nindent function.
Go provides a way for setting template options to control behavior
when a map is indexed with a key that’s not present in the map
The required function gives developers the ability to declare a value entry
as required for template rendering.
The tpl function allows developers to evaluate strings as templates inside a template.
Rendering a external configuration file
(.Files.Get "conf/app.conf")
Image pull secrets are essentially a combination of registry, username, and password.
Automatically Roll Deployments When ConfigMaps or Secrets change
configmaps or secrets are injected as configuration
files in containers
a restart may be required should those
be updated with a subsequent helm upgrade
The sha256sum function can be used to ensure a deployment’s
annotation section is updated if another file changes
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
helm upgrade --recreate-pods
"helm.sh/resource-policy": keep
resources that should not be deleted when Helm runs a
helm delete
this resource becomes
orphaned. Helm will no longer manage it in any way.
create some reusable parts in your chart
In the templates/ directory, any file that begins with an
underscore(_) is not expected to output a Kubernetes manifest file.
by convention, helper templates and partials are placed in a
_helpers.tpl file.
The current best practice for composing a complex application from discrete parts
is to create a top-level umbrella chart that
exposes the global configurations, and then use the charts/ subdirectory to
embed each of the components.
SAP’s Converged charts: These charts
install SAP Converged Cloud a full OpenStack IaaS on Kubernetes. All of the charts are collected
together in one GitHub repository, except for a few submodules.
Deis’s Workflow:
This chart exposes the entire Deis PaaS system with one chart. But it’s different
from the SAP chart in that this umbrella chart is built from each component, and
each component is tracked in a different Git repository.
YAML is a superset of JSON
any valid JSON structure ought to be valid in YAML.
As a best practice, templates should follow a YAML-like syntax unless
the JSON syntax substantially reduces the risk of a formatting issue.
There are functions in Helm that allow you to generate random data,
cryptographic keys, and so on.
a chart repository is a location where packaged charts can be
stored and shared.
A chart repository is an HTTP server that houses an index.yaml file and
optionally some packaged charts.
Because a chart repository can be any HTTP server that can serve YAML and tar
files and can answer GET requests, you have a plethora of options when it comes
down to hosting your own chart repository.
It is not required that a chart package be located on the same server as the
index.yaml file.
A valid chart repository must have an index file. The
index file contains information about each chart in the chart repository.
The Helm project provides an open-source Helm repository server called ChartMuseum that you can host yourself.
$ helm repo index fantastic-charts --url https://fantastic-charts.storage.googleapis.com
A repository will not be added if it does not contain a valid
index.yaml
add the repository to their helm client via the helm
repo add [NAME] [URL] command with any name they would like to use to
reference the repository.
Helm has provenance tools which help chart users verify the integrity and origin
of a package.
Integrity is established by comparing a chart to a provenance record
The provenance file contains a chart’s YAML file plus several pieces of
verification information
Chart repositories serve as a centralized collection of Helm charts.
Chart repositories must make it possible to serve provenance files over HTTP via
a specific request, and must make them available at the same URI path as the chart.
We don’t want to be “the certificate authority” for all chart
signers. Instead, we strongly favor a decentralized model, which is part
of the reason we chose OpenPGP as our foundational technology.
The Keybase platform provides a public
centralized repository for trust information.
A chart contains a number of Kubernetes resources and components that work together.
A test in a helm chart lives under the templates/ directory and is a pod definition that specifies a container with a given command to run.
The pod definition must contain one of the helm test hook annotations: helm.sh/hook: test-success or helm.sh/hook: test-failure
helm test
nest your test suite under a tests/ directory like <chart-name>/templates/tests/
load balancing, connection failover and decoupling of the application tier from the underlying database topologies.
ProxySQL as a Kubernetes service (centralized deployment)
running as a service makes ProxySQL pods live independently from the applications and can be easily scaled and clustered together with the help of Kubernetes ConfigMap.
ProxySQL's multi-layer configuration system makes pod clustering possible with ConfigMap.
create ProxySQL pods and attach a Kubernetes service to be accessed by the other pods within the Kubernetes network or externally.
Default to 6033 for MySQL load-balanced connections and 6032 for ProxySQL administration console.
separated by "---" delimiter
deploy two ProxySQL pods as a ReplicaSet that matches containers labelled with "app=proxysql,tier=frontend".
A Kubernetes service is an abstraction layer which defines the logical set of pods and a policy by which to access them
The range of valid ports for NodePort resource is 30000-32767.
ConfigMap - To store ProxySQL configuration file as a volume so it can be mounted to multiple pods and can be remounted again if the pod is being rescheduled to the other Kubernetes node.
FreeIPA DNS integration allows administrator to manage and serve DNS records in a domain using the same CLI or Web UI as when managing identities and policies.
Single-master DNS is error prone, especially for inexperienced admins.
Goal is NOT to provide general-purpose DNS server.
DNS component in FreeIPA is optional and user may choose to manage all DNS records manually in other third party DNS server.
Clients can be configured to automatically run DNS updates (nsupdate) when their IP address changes and thus keeping its DNS record up-to-date. DNS zones can be configured to synchronize client's reverse (PTR) record along with the forward (A, AAAA) DNS record.
It is extremely hard to change DNS domain in existing installations so it is better to think ahead.
You should only use names which are delegated to you by the parent domain.
Not respecting this rule will cause problems sooner or later!
DNSSEC validation.
For internal names you can use arbitrary sub-domain in a DNS sub-tree you own, e.g. int.example.com.. Always respect rules from the previous section.
General advice about DNS views is do not use them because views make DNS deployment harder to maintain and security benefits are questionable (when compared with ACL).
The DNS integration is based on the bind-dyndb-ldap project, which enhances BIND name server to be able to use FreeIPA server LDAP instance as a data backend (data are stored in cn=dns entry, using schema defined by bind-dyndb-ldap
FreeIPA LDAP directory information tree is by default accessible to any user in the network
As DNS data are often considered as sensitive and as having access to cn=dns tree would be basically equal to being able to run zone transfer to all FreeIPA managed DNS zones, contents of this tree in LDAP are hidden by default.
standard system log (/var/log/messages or system journal)
BIND configuration (/etc/named.conf) can be updated to produce a more detailed log.
"FreeIPA DNS integration allows administrator to manage and serve DNS records in a domain using the same CLI or Web UI as when managing identities and policies."
in order to keep master a true record of known working production code the actual deployment to production should happen from the feature branch before merging it into master.
This approach works well if we seldom publish results of our work. (Maybe once every 2 weeks).
Aside from promoting ready to deploy master branch and feature branches (same as GitHub Flow) it introduces three other kinds of branches
The persistent data stored in the backend belongs to a workspace.
Certain backends support multiple named workspaces, allowing multiple states
to be associated with a single configuration.
Terraform starts with a single workspace named "default". This
workspace is special both because it is the default and also because
it cannot ever be deleted.
Within your Terraform configuration, you may include the name of the current
workspace using the ${terraform.workspace} interpolation sequence.
changing behavior based
on the workspace.
Named workspaces allow conveniently switching between multiple instances of
a single configuration within its single backend.
A common use for multiple workspaces is to create a parallel, distinct copy of
a set of infrastructure in order to test a set of changes before modifying the
main production infrastructure.
Non-default workspaces are often related to feature branches in version control.
Workspaces alone
are not a suitable tool for system decomposition, because each subsystem should
have its own separate configuration and backend, and will thus have its own
distinct set of workspaces.
In particular, organizations commonly want to create a strong separation
between multiple deployments of the same infrastructure serving different
development stages (e.g. staging vs. production) or different internal teams.
use one or more re-usable modules to
represent the common elements, and then represent each instance as a separate
configuration that instantiates those common elements in the context of a
different backend.
If a Terraform state for one configuration is stored in a remote backend
that is accessible to other configurations then
terraform_remote_state
can be used to directly consume its root module outputs from those other
configurations.
For server addresses, use a provider-specific resource to create a DNS
record with a predictable name and then either use that name directly or
use the dns provider to retrieve
the published addresses in other configurations.
Workspaces are technically equivalent to renaming your state file.
using a remote backend instead is recommended when there are
multiple collaborators.
learn about is SSHKit and the various methods it provides
SSHKit was actually developed and released with Capistrano 3, and it’s basically a lower-level tool that provides methods for connecting and interacting with remote servers
on(): specifies the server to run on
within(): specifies the directory path to run in
with(): specifies the environment variables to run with
run on the application server
within the path specified
with certain environment variables set
execute(): the workhorse that runs the commands on your server
upload(): uploads a file from your local computer to your remote server
capture(): executes a command and returns its output as a string
upload() has the bang symbol (!) because that’s how it’s defined in SSHKit, and it’s just a convention letting us know that the method will block until it finishes.
But in order to ensure rake runs with the proper environment variables set, we have to use rake as a symbol and pass db:seed as a string
This format will also be necessary whenever you’re running any other Rails-specific commands that rely on certain environment variables being set
I recommend you take a look at SSHKit’s example page to learn more
make sure we pushed all our local changes to the remote master branch
run this task before Capistrano runs its own deploy task
actually creates three separate tasks
I created a namespace called deploy to contain these tasks since that’s what they’re related to.
we’re using the callbacks inside a namespace to make sure Capistrano knows which tasks the callbacks are referencing.
custom recipe (a Capistrano term meaning a series of tasks)
/shared: holds files and directories that persist throughout deploys
When you run cap production deploy, you’re actually calling a Capistrano task called deploy, which then sequentially invokes other tasks
your favorite browser (I hope it’s not Internet Explorer)
Deployment is hard and takes a while to sink in.
the most important thing is to not get discouraged
I didn’t want other people going through the same thing
Once a custom resource is installed, users can create and access its objects using
kubectl
When you combine a custom resource with a custom controller, custom resources
provide a true declarative API.
A declarative API
allows you to declare or specify the desired state of your resource and tries to
keep the current state of Kubernetes objects in sync with the desired state.
Custom controllers can work with any kind of resource,
but they are especially effective when combined with custom resources.
The
Operator pattern combines custom
resources and custom controllers.
the API represents a desired state, not an exact state.
define configuration of applications or infrastructure.
The main operations on the objects are CRUD-y (creating, reading, updating and deleting).
The client says "do this", and then gets an operation ID back, and has to check a separate Operation object to determine completion of the request.
The natural operations on the objects are not CRUD-y.
High bandwidth access (10s of requests per second sustained) needed.
Use a ConfigMap if any of the following apply
You want to put the entire config file into one key of a configMap.
You want to perform rolling updates via Deployment, etc., when the file is updated.
Use a secret for sensitive data, which is similar to a configMap but more secure.
You want to build new automation that watches for updates on the new object, and then CRUD other objects, or vice versa.
You want the object to be an abstraction over a collection of controlled resources, or a summarization of other resources.
CRDs are simple and can be created without any programming.
Aggregated APIs are subordinate API servers that sit behind the primary API server
CRDs allow users to create new types of resources without adding another API server
Defining a CRD object creates a new custom resource with a name and schema that you specify.
The name of a CRD object must be a valid
DNS subdomain name
each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects.
The main API server delegates requests to you for the custom resources that you handle,
making them available to all of its clients.
The new endpoints support CRUD basic operations via HTTP and kubectl
Custom resources consume storage space in the same way that ConfigMaps do.
Aggregated API servers may use the same storage as the main API server
CRDs always use the same authentication, authorization, and audit logging as the built-in resources of your API server.
most RBAC roles will not grant access to the new resources (except the cluster-admin role or any role created with wildcard rules).
CRDs and Aggregated APIs often come bundled with new role definitions for the types they add.
Researching issues felt like bouncing a rubber ball between teams, hard to catch the root cause and harder yet to stop from bouncing between one another.
In the past, Edge Engineering had ops-focused teams and SRE specialists who owned the deploy+operate+support parts of the software life cycle
devs could push code themselves when needed, and also were responsible for off-hours production issues and support requests
What were we trying to accomplish and why weren’t we being successful?
These specialized roles create efficiencies within each segment while potentially creating inefficiencies across the entire life cycle.
Grouping differing specialists together into one team can reduce silos, but having different people do each role adds communication overhead, introduces bottlenecks, and inhibits the effectiveness of feedback loops.
devops principles
develops a system also be responsible for operating and supporting that system
Each development team owns deployment issues, performance bugs, capacity planning, alerting gaps, partner support, and so on.
Those centralized teams act as force multipliers by turning their specialized knowledge into reusable building blocks.
Communication and alignment are the keys to success.
Full cycle developers are expected to be knowledgeable and effective in all areas of the software life cycle.
ramping up on areas they haven’t focused on before
We run dev bootcamps and other forms of ongoing training to impart this knowledge and build up these skills
“how can I automate what is needed to operate this system?”
“what self-service tool will enable my partners to answer their questions without needing me to be involved?”
A full cycle developer thinks and acts like an SWE, SDET, and SRE. At times they create software that solves business problems, at other times they write test cases for that, and still other times they automate operational aspects of that system.
the need for continuous delivery pipelines, monitoring/observability, and so on.
Tooling and automation help to scale expertise, but no tool will solve every problem in the developer productivity and operations space
A replica set in MongoDB is a group of mongod processes
that maintain the same data set.
Replica sets provide redundancy and
high availability, and are the basis for all production
deployments.
With
multiple copies of data on different database servers, replication
provides a level of fault tolerance against the loss of a single
database server.
replication can provide increased read capacity as
clients can send read operations to different servers.
A replica set is a group of mongod instances that maintain
the same data set.
A replica set contains several data bearing nodes
and optionally one arbiter node.
one and
only one member is deemed the primary node, while the other nodes are
deemed secondary nodes.
A replica set can have only one primary capable of
confirming writes with { w: "majority" }
write concern; although in some circumstances, another mongod instance
may transiently believe itself to also be primary.
The secondaries replicate the
primary’s oplog and apply the operations to their data sets such that
the secondaries’ data sets reflect the primary’s data set
add a mongod instance to a replica set as an
arbiter. An arbiter participates in
elections but does not hold data
An arbiter will always be an arbiter
whereas a primary may step down and
become a secondary and a
secondary may become the primary
during an election.
Secondaries replicate the primary’s oplog and apply the operations to
their data sets asynchronously.
These slow oplog messages are logged
for the secondaries in the diagnostic log under the REPL component with the text applied
op: <oplog entry> took <num>ms.
Replication lag refers to the amount of time
that it takes to copy (i.e. replicate) a write operation on the
primary to a secondary.
When a primary does not communicate with the other members of the set
for more than the configured electionTimeoutMillis period
(10 seconds by default), an eligible secondary calls for an election
to nominate itself as the new primary.
The replica set cannot process write operations
until the election completes successfully.
The median time before a cluster elects a new primary should not
typically exceed 12 seconds, assuming default replica
configuration settings.
Factors such as network latency may extend the time required
for replica set elections to complete, which in turn affects the amount
of time your cluster may operate without a primary.
Your application connection logic should include tolerance for automatic
failovers and the subsequent elections.
MongoDB drivers
can detect the loss of the primary and automatically
retry certain write operations a single time,
providing additional built-in handling of automatic failovers and elections
By default, clients read from the primary [1];
however, clients can specify a read preference to send read operations to secondaries.
Keyfiles are bare-minimum forms of security and are best suited for testing or
development environments.
With keyfile authentication, each
mongod instances in the replica set uses the contents of the keyfile as the
shared password for authenticating other members in the deployment.
On UNIX systems, the keyfile must not have group or world
permissions.
MinIO is purpose-built to take full advantage of the Kubernetes architecture.
MinIO and Kubernetes work together to simplify infrastructure management, providing a way to manage object storage infrastructure within the Kubernetes toolset.
The operator pattern extends Kubernetes's familiar declarative API model with custom resource definitions (CRDs) to perform common operations like resource orchestration, non-disruptive upgrades, cluster expansion and to maintain high-availability
The Operator uses the command set kubectl that the Kubernetes community was already familiar with and adds the kubectl minio plugin . The MinIO Operator and the MinIO kubectl plugin facilitate the deployment and management of MinIO Object Storage on Kubernetes - which is how multi-tenant object storage as a service is delivered.
choosing a leader for a distributed application without an internal member election process
The Operator Console makes Kubernetes object storage easier still. In this graphical user interface, MinIO created something so simple that anyone in the organization can create, deploy and manage object storage as a service.
The primary unit of managing MinIO on Kubernetes is the tenant.
The MinIO Operator can allocate multiple tenants within the same Kubernetes cluster.
Each tenant, in turn, can have different capacity (i.e: a small 500GB tenant vs a 100TB tenant), resources (1000m CPU and 4Gi RAM vs 4000m CPU and 16Gi RAM) and servers (4 pods vs 16 pods), as well a separate configurations regarding Identity Providers, Encryption and versions.
each tenant is a cluster of server pools (independent sets of nodes with their own compute, network, and storage resources), that, while sharing the same physical infrastructure, are fully isolated from each other in their own namespaces.
Each tenant runs their own MinIO cluster, fully isolated from other tenants
Each tenant scales independently by federating clusters across geographies.
there is a lengthy suite of tests and checks that run before it is deployed to staging. During this period, which could end up being hours, engineers will likely pick up another task. I’ve seen people merge, and then forget that their changes are on staging, more times than I can count.
only merge code that is ready to go live
written sufficient tests and have validated our changes in development.
All branches are cut from main, and all changes get merged back into main.
If we ever have an issue in production, we always roll forward.
Feature flags can be enabled on a per-user basis so we can monitor performance and gather feedback
Experimental features can be enabled by users in their account settings.
we have monitoring, logging, and alarms around all of our services. We also blue/green deploy, by draining and replacing a percentage of containers.
Dropping your staging environment in favour of true continuous integration and deployment can create a different mindset for shipping software.
Edge routerA router that enforces the firewall policy for your cluster.
Cluster networkA set of links, logical or physical, that facilitate communication within a cluster according to the Kubernetes networking model.
A Kubernetes ServiceA way to expose an application running on a set of Pods as a network service. that identifies a set of Pods using labelTags objects with identifying attributes that are meaningful and relevant to users. selectors.
Services are assumed to have virtual IPs only routable within the cluster network.
Ingress exposes HTTP and HTTPS routes from outside the cluster to
services within the cluster.
Traffic routing is controlled by rules defined on the Ingress resource.
An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.
Exposing services other than HTTP and HTTPS to the internet typically
uses a service of type Service.Type=NodePort or
Service.Type=LoadBalancer.
You must have an ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.
As with all other Kubernetes resources, an Ingress needs apiVersion, kind, and metadata fields
Ingress frequently uses annotations to configure some options depending on the Ingress controller,
Ingress resource only supports rules
for directing HTTP traffic.
An optional host.
A list of paths
A backend is a combination of Service and port names
has an associated backend
Both the host and path must match the content of an incoming request before the
load balancer directs traffic to the referenced Service.
HTTP (and HTTPS) requests to the
Ingress that matches the host and path of the rule are sent to the listed backend.
A default backend is often configured in an Ingress controller to service any requests that do not
match a path in the spec.
An Ingress with no rules sends all traffic to a single default backend.
Ingress controllers and load balancers may take a minute or two to allocate an IP address.
A fanout configuration routes traffic from a single IP address to more than one Service,
based on the HTTP URI being requested.
nginx.ingress.kubernetes.io/rewrite-target: /
describe ingress
get ingress
Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.
route requests based on
the Host header.
an Ingress resource without any hosts defined in the rules, then any
web traffic to the IP address of your Ingress controller can be matched without a name based
virtual host being required.
secure an Ingress by specifying a SecretStores sensitive information, such as passwords, OAuth tokens, and ssh keys.
that contains a TLS private key and certificate.
Currently the Ingress only
supports a single TLS port, 443, and assumes TLS termination.
An Ingress controller is bootstrapped with some load balancing policy settings
that it applies to all Ingress, such as the load balancing algorithm, backend
weight scheme, and others.
persistent sessions, dynamic weights) are not yet exposed through the
Ingress. You can instead get these features through the load balancer used for
a Service.
review the controller
specific documentation to see how they handle health checks
edit ingress
After you save your changes, kubectl updates the resource in the API server, which tells the
Ingress controller to reconfigure the load balancer.
kubectl replace -f on a modified Ingress YAML file.
Node: A worker machine in Kubernetes, part of a cluster.
in most common Kubernetes deployments, nodes in the cluster are not part of the public internet.
Edge router: A router that enforces the firewall policy for your cluster.
a gateway managed by a cloud provider or a physical piece of hardware.
Cluster network: A set of links, logical or physical, that facilitate communication within a cluster according to the Kubernetes networking model.
Service: A Kubernetes Service that identifies a set of Pods using label selectors.
An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.
An Ingress does not expose arbitrary ports or protocols.
You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.
The name of an Ingress object must be a valid
DNS subdomain name
The Ingress spec
has all the information needed to configure a load balancer or proxy server.
Ingress resource only supports rules
for directing HTTP(S) traffic.
An Ingress with no rules sends all traffic to a single default backend and .spec.defaultBackend
is the backend that should handle requests in that case.
If defaultBackend is not set, the handling of requests that do not match any of the rules will be up to the
ingress controller
A common
usage for a Resource backend is to ingress data to an object storage backend
with static assets.
Exact: Matches the URL path exactly and with case sensitivity.
Prefix: Matches based on a URL path prefix split by /. Matching is case
sensitive and done on a path element by element basis.
multiple paths within an Ingress will match a request. In those
cases precedence will be given first to the longest matching path.
Hosts can be precise matches (for example “foo.bar.com”) or a wildcard (for
example “*.foo.com”).
No match, wildcard only covers a single DNS label
Each Ingress should specify a class, a reference to an
IngressClass resource that contains additional configuration including the name
of the controller that should implement the class.
secure an Ingress by specifying a Secret
that contains a TLS private key and certificate.
The Ingress resource only
supports a single TLS port, 443, and assumes TLS termination at the ingress point
(traffic to the Service and its Pods is in plaintext).
TLS will not work on the default rule because the
certificates would have to be issued for all the possible sub-domains.
hosts in the tls section need to explicitly match the host in the rules
section.