Group items tagged

When multiple private IP addresses are masquerading behind a single public IP address, Azure uses port address translation (PAT) to masquerade private IP addresses.

If you want outbound connectivity when working with Standard SKUs, you must explicitly define it either with Standard Public IP addresses or Standard public Load Balancer.

the VM is part of a public Load Balancer backend pool. The VM does not have a public IP address assigned to it.

VM has an Instance Level Public IP (ILPIP) assigned to it. As far as outbound connections are concerned, it doesn't matter whether the VM is load balanced or not.

A public IP assigned to a VM is a 1:1 relationship (rather than 1: many) and implemented as a stateless 1:1 NAT.

Port masquerading (PAT) is not used, and the VM has all ephemeral ports available for use.

When the load-balanced VM creates an outbound flow, Azure translates the private source IP address of the outbound flow to the public IP address of the public Load Balancer frontend.

Ephemeral ports of the load balancer's public IP address frontend are used to distinguish individual flows originated by the VM.

When multiple public IP addresses are associated with Load Balancer Basic, any of these public IP addresses are a candidate for outbound flows, and one is selected at random.

the VM is not part of a public Load Balancer pool (and not part of an internal Standard Load Balancer pool) and does not have an ILPIP address assigned to it.

The public IP address used for this outbound flow is not configurable and does not count against the subscription's public IP resource limit.

Standard Load Balancer uses all candidates for outbound flows at the same time when multiple (public) IP frontends is present.

Load Balancer Basic chooses a single frontend to be used for outbound flows when multiple (public) IP frontends are candidates for outbound flows.

the disableOutboundSnat option defaults to false and signifies that this rule programs outbound SNAT for the associated VMs in the backend pool of the load balancing rule.

Port masquerading SNAT (PAT)

Ephemeral port preallocation for port masquerading SNAT (PAT)

determine the public source IP address of an outbound connection.

To use a secret, a pod needs to reference the secret.

--from-file

You can also create a Secret in a file first, in json or yaml format, and then create that object.

The Secret contains two maps: data and stringData.

Kubernetes automatically creates secrets which contain credentials for accessing the API and it automatically modifies your pods to use this type of secret.

kubectl get and kubectl describe avoid showing the contents of a secret by default.

stringData field is provided for convenience, and allows you to provide secret data as unencoded strings.

where you are deploying an application that uses a Secret to store a configuration file, and you want to populate parts of that configuration file during your deployment process.

a field is specified in both data and stringData, the value from stringData is used.

The keys of data and stringData must consist of alphanumeric characters, ‘-’, ‘_’ or ‘.’.

Newlines are not valid within these strings and must be omitted.

When using the base64 utility on Darwin/macOS users should avoid using the -b option to split long lines.

create a Secret from generators and then apply it to create the object on the Apiserver.

The generated Secrets name has a suffix appended by hashing the contents.

Multiple pods can reference the same secret.

each container needs its own volumeMounts block, but only one .spec.volumes is needed per secret

use .spec.volumes[].secret.items field to change target path of each key:

You can also specify the permission mode bits files part of a secret will have. If you don’t specify any, 0644 is used by default.

Kubelet is checking whether the mounted secret is fresh on every periodic sync.

cache propagation delay depends on the chosen cache type

Multiple pods can reference the same secret.

env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base-64 decoded values of the secret data.

An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry password to the Kubelet so it can pull a private image on behalf of your Pod.

Individual secrets are limited to 1MiB in size.

Kubelet only supports use of secrets for Pods it gets from the API server.

Secrets must be created before they are consumed in pods as environment variables unless they are marked as optional.

Once a pod is scheduled, the kubelet will try to fetch the secret value.

Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret.

volumes: - name: secret-volume secret: secretName: ssh-key-secret

You do not need to escape special characters in passwords from files

make that key begin with a dot

Dotfiles in secret volume

.secret-file

a frontend container which handles user interaction and business logic, but which cannot see the private key;

a signer container that can see the private key, and responds to simple signing requests from the frontend

watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided

watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.

additional precautions with secret objects, such as avoiding writing them to disk where possible.

A secret is only sent to a node if a pod on that node requires it

only the secrets that a pod requests are potentially visible within its containers

each container in a pod has to request the secret volume in its volumeMounts for it to be visible within the container.

In the API server secret data is stored in etcdConsistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

limit access to etcd to admin users

A user who can create a pod that uses a secret can also see the value of that secret.

anyone with root on any node can read any secret from the apiserver, by impersonating the kubelet.

Name servers host a domain’s DNS information in a text file called a zone file.

Start of Authority (SOA) records

specifying DNS records, which match domain names to IP addresses.

Every domain’s zone file contains the domain administrator’s email address, the name servers, and the DNS records.

Your ISP’s DNS resolver queries a root nameserver for the proper TLD nameserver. In other words, it asks the root nameserver, *Where can I find the nameserver for .com domains?*

In actuality, ISPs cache a lot of DNS information after they’ve looked it up the first time.

An A record points your domain or subdomain to your Linode’s IP address,

use an asterisk (*) as your subdomain

An AAAA record is just like an A record, but for IPv6 IP addresses.

An AXFR record is a type of DNS record used for DNS replication

DNS Certification Authority Authorization uses DNS to allow the holder of a domain to specify which certificate authorities are allowed to issue certificates for that domain.

A CNAME record or Canonical Name record matches a domain or subdomain to a different domain.

Some mail servers handle mail oddly for domains with CNAME records, so you should not use a CNAME record for a domain that gets email.

MX records cannot reference CNAME-defined hostnames.

A DKIM record or DomainKeys Identified Mail record displays the public key for authenticating messages that have been signed with the DKIM protocol

DKIM records are implemented as text records.

An MX record or mail exchanger record sets the mail delivery destination for a domain or subdomain.

An MX record should ideally point to a domain that is also the hostname for its server.

Priority allows you to designate a fallback server (or servers) for mail for a particular domain. Lower numbers have a higher priority.

NS records or name server records set the nameservers for a domain or subdomain.

You can also set up different nameservers for any of your subdomains

Primary nameservers get configured at your registrar and secondary subdomain nameservers get configured in the primary domain’s zone file.

A PTR record or pointer record matches up an IP address to a domain or subdomain, allowing reverse DNS queries to function.

opposite service an A record does

PTR records are usually set with your hosting provider. They are not part of your domain’s zone file.

An SOA record or Start of Authority record labels a zone file with the name of the host where it was originally created.

Minimum TTL: The minimum amount of time other servers should keep data cached from this zone file.

An SPF record or Sender Policy Framework record lists the designated mail servers for a domain or subdomain.

An SPF record for your domain tells other receiving mail servers which outgoing server(s) are valid sources of email so they can reject spoofed mail from your domain that has originated from unauthorized servers.

Make sure your SPF records are not too strict.

An SRV record or service record matches up a specific service that runs on your domain or subdomain to a target domain.

Service: The name of the service must be preceded by an underscore (_) and followed by a period (.)

Protocol: The name of the protocol must be proceeded by an underscore (_) and followed by a period (.)

Port: The TCP or UDP port on which the service runs.

Target: The target domain or subdomain. This domain must have an A or AAAA record that resolves to an IP address.

A TXT record or text record provides information about the domain in question to other resources on the internet.

Auto DevOps works with any Kubernetes cluster;

using the Docker or Kubernetes executor, with privileged mode enabled.

Base domain (needed for Auto Review Apps and Auto Deploy)

Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)

Prometheus (needed for Auto Monitoring)

scrape your Kubernetes cluster.

project level as a variable: KUBE_INGRESS_BASE_DOMAIN

A wildcard DNS A record matching the base domain(s) is required

Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).

review/ (every environment starting with review/)

staging

production

need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.

Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.

Continuous deployment to production using timed incremental rollout

Automatic deployment to staging, manual deployment to production

Auto Build creates a build of the application using an existing Dockerfile or Heroku buildpacks.

If a project’s repository contains a Dockerfile, Auto Build will use docker build to create a Docker image.

Each buildpack requires certain files to be in your project’s repository for Auto Build to successfully build your application.

Auto Test automatically runs the appropriate tests for your application using Herokuish and Heroku buildpacks by analyzing your project to detect the language and framework.

Auto Code Quality uses the Code Quality image to run static analysis and other code checks on the current code.

Static Application Security Testing (SAST) uses the SAST Docker image to run static analysis on the current code and checks for potential security issues.

Dependency Scanning uses the Dependency Scanning Docker image to run analysis on the project dependencies and checks for potential security issues.

License Management uses the License Management Docker image to search the project dependencies for their license.

Vulnerability Static Analysis for containers uses Clair to run static analysis on a Docker image and checks for potential security issues.

Review Apps are temporary application environments based on the branch’s code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur.

The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain.

Your apps should not be manipulated outside of Helm (using Kubernetes directly).

Dynamic Application Security Testing (DAST) uses the popular open source tool OWASP ZAProxy to perform an analysis on the current code and checks for potential security issues.

Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.

add the paths to a file named .gitlab-urls.txt in the root directory, one per line.

After a branch or merge request is merged into the project’s default branch (usually master), Auto Deploy deploys the application to a production environment in the Kubernetes cluster, with a namespace based on the project name and unique project ID

Auto Deploy doesn’t include deployments to staging or canary by default, but the Auto DevOps template contains job definitions for these tasks if you want to enable them.

For internal and private projects a GitLab Deploy Token will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.

If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.

If present, DB_INITIALIZE will be run as a shell command within an application pod as a helm post-install hook.

a post-install hook means that if any deploy succeeds, DB_INITIALIZE will not be processed thereafter.

DB_MIGRATE will be run as a shell command within an application pod as a helm pre-upgrade hook.

Once your application is deployed, Auto Monitoring makes it possible to monitor your application’s server and response metrics right out of the box.

annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254"

While Auto DevOps provides great defaults to get you started, you can customize almost everything to fit your needs; from custom buildpacks, to Dockerfiles, Helm charts, or even copying the complete CI/CD configuration into your project to enable staging and canary deployments, and more.

Auto DevOps uses Helm to deploy your application to Kubernetes.

make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.

specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment.

Your additions will be merged with the Auto DevOps template using the behaviour described for include

copy and paste the contents of the Auto DevOps template into your project and edit this as needed.

In order to support applications that require a database, PostgreSQL is provisioned by default.

Set up the replica variables using a project variable and scale your application by just redeploying it!

You should not scale your application using Kubernetes directly.

Some applications need to define secret variables that are accessible by the deployed application.

Auto DevOps pipelines will take your application secret variables to populate a Kubernetes secret.

Environment variables are generally considered immutable in a Kubernetes pod.

If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to 1 as a CI/CD variable), then the application will be automatically deployed to a staging environment, and a production_manual job will be created for you when you’re ready to manually deploy to production.

If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to 1 as a CI/CD variable) then two manual jobs will be created: canary which will deploy the application to the canary environment production_manual which is to be used by you when you’re ready to manually deploy to production.

If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead of the standard production job, 4 different manual jobs will be created: rollout 10% rollout 25% rollout 50% rollout 100%

To start a job, click on the play icon next to the job’s name.

not all buildpacks support Auto Test yet

When a project has been marked as private, GitLab’s Container Registry requires authentication when downloading containers.

Authentication credentials will be valid while the pipeline is running, allowing for a successful initial deployment.

We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.

With respect to permissions, all users on the system except those with a user role of No Access have read access to objects in partition Common, and by default, partition Common is their current partition.

The current partition is the specific partition to which the system is currently set for a logged-in user.

A partition access assignment gives a user some level of access to the specified partition.

assigning partition access to a user does not necessarily give the user full access to all objects in the partition

user account objects also reside in partitions

when you first install the BIG-IP system, every existing user account (root and admin) resides in partition Common

the partition in which a user account object resides does not affect the partition or partitions to which that user is granted access to manage other BIG-IP objects

the object it references resides in partition Common

a referenced object must reside either in the same partition as the object that is referencing it

config/environments/production.rb

Create an additional staging environment that closely resembles the production one

Keep any additional configuration in YAML files under the config/ directory

Rails::Application.config_for(:yaml_file)

Use nested routes to express better the relationship between ActiveRecord models

nest routes more than 1 level deep then use the shallow: true option

namespaced routes to group related actions

Keep the controllers skinny

all the business logic should naturally reside in the model

Share no more than two instance variables between a controller and a view.

using a template

Prefer corresponding symbols to numeric HTTP status codes

without abbreviations

Keep your models for business logic and data-persistence only

Avoid altering ActiveRecord defaults (table names, primary key, etc)

Prefer has_many :through to has_and_belongs_to_many

self[:attribute]

self[:attribute] = value

validates

Consider extracting custom validators to a shared gem

preferable to make a class method instead which serves the same purpose of the named scope

.update_attributes

Override the to_param method of the model

Use the friendly_id gem. It allows creation of human-readable URLs by using some descriptive attribute of the model instead of its id

.find_each

.find_each

Looping through a collection of records from the database (using the all method, for example) is very inefficient since it will try to instantiate all the objects at once

always call before_destroy callbacks that perform validation with prepend: true

When persisting AR objects

Avoid string interpolation in queries

param will be properly escaped

Consider using named placeholders instead of positional placeholders

use of find over where when you need to retrieve a single record by id

use of find_by over where and find_by_attribute

use of where.not over SQL

use heredocs with squish

Use rake db:schema:load instead of rake db:migrate to initialize an empty database

change_column_default

imposing data integrity from the Rails app is impossible

use the change method instead of up and down methods.

constructive migrations

use models in migrations, make sure you define them so that you don't end up with broken migrations in the future

In this case, block will be used by create_table in rollback

Never make complex formatting in the views, export the formatting to a method in the view helper or the model.

When the labels of an ActiveRecord model need to be translated, use the activerecord scope

Separate the texts used in the views from translations of ActiveRecord attributes

config/application.rb config.i18n.load_path += Dir[Rails.root.join('config', 'locales', '**', '*.{rb,yml}')]

I18n.t

I18n.l

Use "lazy" lookup for the texts used in views.

Use the dot-separated keys in the controllers and models

Reserve app/assets for custom stylesheets, javascripts, or images

Third party code such as jQuery or bootstrap should be placed in vendor/assets

Provide both HTML and plain-text view templates

config.action_mailer.raise_delivery_errors = true

Use a local SMTP server like Mailcatcher in the development environment

Provide default settings for the host name

The _url methods include the host name and the _path methods don't

Format the from and to addresses properly

default from:

sending html emails all styles should be inline

Sending emails while generating page response should be avoided. It causes delays in loading of the page and request can timeout if multiple email are sent.

.start_with?

.end_with?

&.

Config your timezone accordingly in application.rb

config.active_record.default_timezone = :local

it can be only :utc or :local

Time.zone.parse

Time.zone.now

learn about is SSHKit and the various methods it provides

SSHKit was actually developed and released with Capistrano 3, and it’s basically a lower-level tool that provides methods for connecting and interacting with remote servers

on(): specifies the server to run on

within(): specifies the directory path to run in

with(): specifies the environment variables to run with

run on the application server

within the path specified

with certain environment variables set

execute(): the workhorse that runs the commands on your server

upload(): uploads a file from your local computer to your remote server

capture(): executes a command and returns its output as a string

upload() has the bang symbol (!) because that’s how it’s defined in SSHKit, and it’s just a convention letting us know that the method will block until it finishes.

I recommend you take a look at SSHKit’s example page to learn more

make sure we pushed all our local changes to the remote master branch

run this task before Capistrano runs its own deploy task

actually creates three separate tasks

I created a namespace called deploy to contain these tasks since that’s what they’re related to.

we’re using the callbacks inside a namespace to make sure Capistrano knows which tasks the callbacks are referencing.

custom recipe (a Capistrano term meaning a series of tasks)

/shared: holds files and directories that persist throughout deploys

When you run cap production deploy, you’re actually calling a Capistrano task called deploy, which then sequentially invokes other tasks

Deployment is hard and takes a while to sink in.

globalLock.totalTime: If this is higher than the total database uptime, the database has been in a lock state for too long.

Unlike relational databases such as MySQL or PostgreSQL, MongoDB uses JSON-like documents for storing data.

Databases operate in an environment that consists of numerous reads, writes, and updates.

When a lock occurs, no other operation can read or modify the data until the operation that initiated the lock is finished.

locks.deadlockCount: Number of times the lock acquisitions have encountered deadlocks

Is the database frequently locking from queries? This might indicate issues with the schema design, query structure, or system architecture.

For version 3.2 on, WiredTiger is the default.

MMAPv1 locks whole collections, not individual documents.

When the MMAPv1 storage engine is in use, MongoDB will use memory-mapped files to store data.

db.serverStatus().mem

mem.resident: Roughly equivalent to the amount of RAM in megabytes that the database process uses

If mem.resident exceeds the value of system memory and there’s a large amount of unmapped data on disk, we’ve most likely exceeded system capacity.

If the value of mem.mapped is greater than the amount of system memory, some operations will experience page faults.

The WiredTiger storage engine is a significant improvement over MMAPv1 in performance and concurrency.

By default, MongoDB will reserve 50 percent of the available memory for the WiredTiger data cache.

wiredTiger.cache.bytes currently in the cache – This is the size of the data currently in the cache.

wiredTiger.cache.tracked dirty bytes in the cache – This is the size of the dirty data in the cache.

we can look at the wiredTiger.cache.bytes read into cache value for read-heavy applications. If this value is consistently high, increasing the cache size may improve overall read performance.

check whether the application is read-heavy. If it is, increase the size of the replica set and distribute the read operations to secondary members of the set.

Replication is the propagation of data from one node to another

Replication sets handle this replication.

Sometimes, data isn’t replicated as quickly as we’d like.

use the db.printSlaveReplicationInfo() or the rs.printSlaveReplicationInfo() command to see the status of a replica set from the perspective of the secondary member of the set.

shows how far behind the secondary members are from the primary. This number should be as low as possible.

monitor this metric closely.

watch for any spikes in replication delay.

One replica set is primary. All others are secondary.

use the profiler to gain a deeper understanding of the database’s behavior.

Enabling the profiler can affect system performance, due to the additional activity.

The upgrade procedure on control plane nodes should be executed one node at a time.

Manually upgrade your CNI provider plugin.

sudo systemctl daemon-reload sudo systemctl restart kubelet

If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again.

To recover from a bad state, you can also run kubeadm upgrade apply --force without changing the version that your cluster is running.

kubeadm-backup-etcd contains a backup of the local etcd member data for this control plane Node.

the contents of this folder can be manually restored in /var/lib/etcd

kubeadm-backup-manifests contains a backup of the static Pod manifest files for this control plane Node.

the contents of this folder can be manually restored in /etc/kubernetes/manifests

Enforces the version skew policies.

Upgrades the control plane components or rollbacks if any of them fails to come up.