"Pinpoint is an open source APM (Application Performance Management) tool for large-scale distributed systems written in Java. http://naver.github.io/pinpoint/"
The generate attribute is used to inform Terragrunt to generate the Terraform code for configuring the backend.
The find_in_parent_folders() helper will automatically search up the directory tree to find the root terragrunt.hcl and inherit the remote_state configuration from it.
Unlike the backend configurations, provider configurations support variables,
if you needed to modify the configuration to expose another parameter (e.g
session_name), you would have to then go through each of your modules to make this change.
instructs Terragrunt to create the file provider.tf in the working directory (where Terragrunt calls terraform)
before it calls any of the Terraform commands
large modules should be considered harmful.
it is a Bad Idea to define all of your environments (dev, stage, prod, etc), or even a large amount of infrastructure (servers, databases, load balancers, DNS, etc), in a single Terraform module.
Large modules are slow, insecure, hard to update, hard to code review, hard to test, and brittle (i.e., you have all your eggs in one basket).
Terragrunt allows you to define your Terraform code once and to promote a versioned, immutable “artifact” of that exact same code from environment to environment.
run one or two control plane instances per failure zone,
scaling those instances vertically first and then scaling horizontally after reaching
the point of falling returns to (vertical) scale.
Kubernetes
nodes do not automatically steer traffic towards control-plane endpoints that are in the
same failure zone
store Event objects in a separate
dedicated etcd instance.
start and configure additional etcd instance
Kubernetes resource limits
help to minimize the impact of memory leaks and other ways that pods and containers can
impact on other components.
Addons' default limits are typically based on data collected from experience running
each addon on small or medium Kubernetes clusters.
When running on large
clusters, addons often consume more of some resources than their default limits.
Many addons scale horizontally - you add capacity by running more pods
The VerticalPodAutoscaler can run in recommender mode to provide suggested
figures for requests and limits.
Some addons run as one copy per node, controlled by a DaemonSet: for example, a node-level log aggregator.
VerticalPodAutoscaler is a custom resource that you can deploy into your cluster
to help you manage resource requests and limits for pods.
The cluster autoscaler
integrates with a number of cloud providers to help you run the right number of
nodes for the level of resource demand in your cluster.
The addon resizer
helps you in resizing the addons automatically as your cluster's scale changes.
"Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The text format (a superset of JSON) is easy to read and author, supporting rapid prototyping. The binary representation is efficient to store, transmit, and skip-scan parse. The rich type system provides unambiguous semantics for long-term preservation of business data which can survive multiple generations of software evolution.
Ion was built to solve the rapid development, decoupling, and efficiency challenges faced every day while engineering large-scale, service-oriented architectures. Ion has been addressing these challenges within Amazon for nearly a decade, and we believe others will benefit as well.
"
"Nmap ("Network Mapper") is a free and open source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. It was designed to rapidly scan large networks, but works fine against single hosts. Nmap runs on all major computer operating systems, and official binary packages are available for Linux, Windows, and Mac OS X. In addition to the classic command-line Nmap executable, the Nmap suite includes an advanced GUI and results viewer (Zenmap), a flexible data transfer, redirection, and debugging tool (Ncat), a utility for comparing scan results (Ndiff), and a packet generation and response analysis tool (Nping)."
the abstraction of the infrastructure layer, which is now considered code. Deployment of a new application may require the deployment of new infrastructure code as well.
"big bang" deployments update whole or large parts of an application in one fell swoop.
Big bang deployments required the business to conduct extensive development and testing before release, often associated with the "waterfall model" of large sequential releases.
Rollbacks are often costly, time-consuming, or even impossible.
In a rolling deployment, an application’s new version gradually replaces the old one.
new and old versions will coexist without affecting functionality or user experience.
Each container is modified to download the latest image from the app vendor’s site.
two identical production environments work in parallel.
Once the testing results are successful, application traffic is routed from blue to green.
In a blue-green deployment, both systems use the same persistence layer or database back end.
You can use the primary database by blue for write operations and use the secondary by green for read operations.
Blue-green deployments rely on traffic routing.
long TTL values can delay these changes.
The main challenge of canary deployment is to devise a way to route some users to the new application.
Using an application logic to unlock new features to specific users and groups.
With CD, the CI-built code artifact is packaged and always ready to be deployed in one or more environments.
Use Build Automation tools to automate environment builds
Use configuration management tools
Enable automated rollbacks for deployments
An application performance monitoring (APM) tool can help your team monitor critical performance metrics including server response times after deployments.
Helm will figure out where to install Tiller by reading your Kubernetes
configuration file (usually $HOME/.kube/config). This is the same file
that kubectl uses.
By default, when Tiller is installed, it does not have authentication enabled.
helm repo update
Without a max history set the history is kept indefinitely, leaving a large number of records for helm and tiller to maintain.
helm init --upgrade
Whenever you install a chart, a new release is created.
one chart can
be installed multiple times into the same cluster. And each can be
independently managed and upgraded.
helm list function will show you a list of all deployed releases.
helm delete
helm status
you
can audit a cluster’s history, and even undelete a release (with helm
rollback).
the Helm
server (Tiller).
The Helm client (helm)
brew install kubernetes-helm
Tiller, the server portion of Helm, typically runs inside of your
Kubernetes cluster.
it can also be run locally, and
configured to talk to a remote Kubernetes cluster.
Role-Based Access Control - RBAC for short
create a service account for Tiller with the right roles and permissions to access resources.
run Tiller in an RBAC-enabled Kubernetes cluster.
run kubectl get pods --namespace
kube-system and see Tiller running.
helm inspect
Helm will look for Tiller in the kube-system namespace unless
--tiller-namespace or TILLER_NAMESPACE is set.
For development, it is sometimes easier to work on Tiller locally, and
configure it to connect to a remote Kubernetes cluster.
even when running locally, Tiller will store release
configuration in ConfigMaps inside of Kubernetes.
helm version should show you both
the client and server version.
Tiller stores its data in Kubernetes ConfigMaps, you can safely
delete and re-install Tiller without worrying about losing any data.
helm reset
The --node-selectors flag allows us to specify the node labels required
for scheduling the Tiller pod.
--override allows you to specify properties of Tiller’s
deployment manifest.
helm init --override manipulates the specified properties of the final
manifest (there is no “values” file).
The --output flag allows us skip the installation of Tiller’s deployment
manifest and simply output the deployment manifest to stdout in either
JSON or YAML format.
By default, tiller stores release information in ConfigMaps in the namespace
where it is running.
switch from the default backend to the secrets
backend, you’ll have to do the migration for this on your own.
a beta SQL storage backend that stores release
information in an SQL database (only postgres has been tested so far).
Once you have the Helm Client and Tiller successfully installed, you can
move on to using Helm to manage charts.
Helm requires that kubelet have access to a copy of the socat program to proxy connections to the Tiller API.
A Release is an instance of a chart running in a Kubernetes cluster.
One chart can often be installed many times into the same cluster.
helm init --client-only
helm init --dry-run --debug
A panic in Tiller is almost always the result of a failure to negotiate with the
Kubernetes API server
Tiller and Helm have to negotiate a common version to make sure that they can safely
communicate without breaking API assumptions
helm delete --purge
Helm stores some files in $HELM_HOME, which is
located by default in ~/.helm
A Chart is a Helm package. It contains all of the resource definitions
necessary to run an application, tool, or service inside of a Kubernetes
cluster.
it like the Kubernetes equivalent of a Homebrew formula,
an Apt dpkg, or a Yum RPM file.
A Repository is the place where charts can be collected and shared.
Set the $HELM_HOME environment variable
each time it is installed, a new release is created.
Helm installs charts into Kubernetes, creating a new release for
each installation. And to find new charts, you can search Helm chart
repositories.
chart repository is named
stable by default
helm search shows you all of the available charts
helm inspect
To install a new package, use the helm install command. At its
simplest, it takes only one argument: The name of the chart.
If you want to use your own release name, simply use the
--name flag on helm install
additional configuration steps you can or
should take.
Helm does not wait until all of the resources are running before it
exits. Many charts require Docker images that are over 600M in size, and
may take a long time to install into the cluster.
helm status
helm inspect
values
helm inspect values stable/mariadb
override any of these settings in a YAML formatted file,
and then pass that file during installation.
helm install -f config.yaml stable/mariadb
--values (or -f): Specify a YAML file with overrides.
--set (and its variants --set-string and --set-file): Specify overrides on the command line.
Values that have been --set can be cleared by running helm upgrade with --reset-values
specified.
Chart
designers are encouraged to consider the --set usage when designing the format
of a values.yaml file.
--set-file key=filepath is another variant of --set.
It reads the file and use its content as a value.
inject a multi-line text into values without dealing with indentation in YAML.
An unpacked chart directory
When a new version of a chart is released, or when you want to change
the configuration of your release, you can use the helm upgrade
command.
Kubernetes charts can be large and
complex, Helm tries to perform the least invasive upgrade.
It will only
update things that have changed since the last release
If both are used, --set values are merged into --values with higher precedence.
The helm get command is a useful tool for looking at a release in the
cluster.
helm rollback
A release version is an incremental revision. Every time an install,
upgrade, or rollback happens, the revision number is incremented by 1.
helm history
a release name cannot be
re-used.
you can rollback a
deleted resource, and have it re-activate.
helm repo list
helm repo add
helm repo update
The Chart Development Guide explains how to develop your own
charts.
helm create
helm lint
helm package
Charts that are archived can be loaded into chart repositories.
chart repository server
Tiller can be installed into any namespace.
Limiting Tiller to only be able to install into specific namespaces and/or resource types is controlled by Kubernetes RBAC roles and rolebindings
Release names are unique PER TILLER INSTANCE
Charts should only contain resources that exist in a single namespace.
not recommended to have multiple Tillers configured to manage resources in the same namespace.
a client-side Helm plugin. A plugin is a
tool that can be accessed through the helm CLI, but which is not part of the
built-in Helm codebase.
Helm plugins are add-on tools that integrate seamlessly with Helm. They provide
a way to extend the core feature set of Helm, but without requiring every new
feature to be written in Go and added to the core tool.
Helm plugins live in $(helm home)/plugins
The Helm plugin model is partially modeled on Git’s plugin model
helm referred to as the porcelain layer, with
plugins being the plumbing.
command is the command that this plugin will
execute when it is called.
Environment variables are interpolated before the plugin
is executed.
The command itself is not executed in a shell. So you can’t oneline a shell script.
Helm is able to fetch Charts using HTTP/S
Variables like KUBECONFIG are set for the plugin if they are set in the
outer environment.
In Kubernetes, granting a role to an application-specific service account is a best practice to ensure that your application is operating in the scope that you have specified.
restrict Tiller’s capabilities to install resources to certain namespaces, or to grant a Helm client running access to a Tiller instance.
Service account with cluster-admin role
The cluster-admin role is created by default in a Kubernetes cluster
Deploy Tiller in a namespace, restricted to deploying resources only in that namespace
Deploy Tiller in a namespace, restricted to deploying resources in another namespace
When running a Helm client in a pod, in order for the Helm client to talk to a Tiller instance, it will need certain privileges to be granted.
SSL Between Helm and Tiller
The Tiller authentication model uses client-side SSL certificates.
creating an internal CA, and using both the
cryptographic and identity functions of SSL.
Helm is a powerful and flexible package-management and operations tool for Kubernetes.
default installation applies no security configurations
with a cluster that is well-secured in a private network with no data-sharing or no other users or teams.
With great power comes great responsibility.
Choose the Best Practices you should apply to your helm installation
Role-based access control, or RBAC
Tiller’s gRPC endpoint and its usage by Helm
Kubernetes employ a role-based access control (or RBAC) system (as do modern operating systems) to help mitigate the damage that can be done if credentials are misused or bugs exist.
In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied.
Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.
release information
charts
charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.
As with all shared software, in a controlled or shared environment you must validate all software you install yourself before you install it.
Helm’s provenance tools to ensure the provenance and integrity of charts
"Helm will figure out where to install Tiller by reading your Kubernetes configuration file (usually $HOME/.kube/config). This is the same file that kubectl uses."
AS3 manages topology records globally in /Common, it is required that records only be managed through AS3, as it will treat the records declaratively.
If a record is added outside of AS3, it will be removed if it is not included in the next AS3 declaration for topology records (AS3 completely overwrites non-AS3 topologies when a declaration is submitted).
using AS3 to delete a tenant (for example, sending DELETE to the /declare/<TENANT> endpoint) that contains GSLB topologies will completely remove ALL GSLB topologies from the BIG-IP.
When posting a large declaration (hundreds of application services in a single declaration), you may experience a 500 error stating that the save sys config operation failed.
Even if you have asynchronous mode set to false, after 45 seconds AS3 sets asynchronous mode to true (API swap), and returns an async response.
When creating a new tenant using AS3, it must not use the same name as a
partition you separately create on the target BIG-IP system.
If you use the
same name and then post the declaration, AS3 overwrites (or removes) the
existing partition completely, including all configuration objects in that
partition.
use AS3 to create a tenant (which creates a BIG-IP partition),
manually adding configuration objects to the partition created by AS3 can
have unexpected results
When you delete the
Tenant using AS3, the system deletes both virtual servers.
if a Firewall_Address_List contains zero addresses, a dummy IPv6 address of ::1:5ee:bad:c0de is added in order to maintain a valid Firewall_Address_List. If an address is added to the list, the dummy address is removed.
use /mgmt/shared/appsvcs/declare?async=true if you have a particularly large declaration which will take a long time to process.
reviewing the Sizing BIG-IP Virtual Editions section (page 7) of Deploying BIG-IP VEs in a Hyper-Converged Infrastructure
To test whether your system has AS3 installed or not, use GET with the /mgmt/shared/appsvcs/info URI.
You may find it more convenient to put multi-line texts such as iRules into
AS3 declarations by first encoding them in Base64.
no matter your BIG-IP user account name, audit logs show all
messages from admin and not the specific user name.
A
certificate is considered a duplicate of an earlier certificate if they contain
the exact same set of hostnames, ignoring capitalization and ordering of
hostnames.
We also have a Duplicate Certificate limit of 5 certificates per week.
a
Renewal Exemption to the Certificates per Registered Domain limit.
The Duplicate Certificate limit and the Renewal Exemption ignore the public key
and extensions requested
You can issue 20 certificates in
week 1, 20 more certificates in week 2, and so on, while not interfering with
renewals of existing certificates.
Revoking certificates does not reset rate limits
If you’ve hit a rate limit, we don’t have a way to temporarily reset it.
get a list of certificates
issued for your registered domain by searching on crt.sh
Revoking certificates does not reset rate limits
If you have a large number of pending authorization objects and are getting a
rate limiting error, you can trigger a validation attempt for those
authorization objects by submitting a JWS-signed POST to one of its challenges, as
described in the
ACME spec.
If you do not
have logs containing the relevant authorization URLs, you need to wait for the
rate limit to expire.
having a large number of pending authorizations is generally the
result of a buggy client
globalLock.currentQueue.total: This number can indicate a possible concurrency issue if it’s consistently high. This can happen if a lot of requests are waiting for a lock to be released.
globalLock.totalTime: If this is higher than the total database uptime, the database has been in a lock state for too long.
Unlike relational databases such as MySQL or PostgreSQL, MongoDB uses JSON-like documents for storing data.
Databases operate in an environment that consists of numerous reads, writes, and updates.
When a lock occurs, no other operation can read or modify the data until the operation that initiated the lock is finished.
locks.deadlockCount: Number of times the lock acquisitions have encountered deadlocks
Is the database frequently locking from queries? This might indicate issues with the schema design, query structure, or system architecture.
For version 3.2 on, WiredTiger is the default.
MMAPv1 locks whole collections, not individual documents.
WiredTiger performs locking at the document level.
When the MMAPv1 storage engine is in use, MongoDB will use memory-mapped files to store data.
All available memory will be allocated for this usage if the data set is large enough.
db.serverStatus().mem
mem.resident: Roughly equivalent to the amount of RAM in megabytes that the database process uses
If mem.resident exceeds the value of system memory and there’s a large amount of unmapped data on disk, we’ve most likely exceeded system capacity.
If the value of mem.mapped is greater than the amount of system memory, some operations will experience page faults.
The WiredTiger storage engine is a significant improvement over MMAPv1 in performance and concurrency.
By default, MongoDB will reserve 50 percent of the available memory for the WiredTiger data cache.
wiredTiger.cache.bytes currently in the cache – This is the size of the data currently in the cache.
wiredTiger.cache.tracked dirty bytes in the cache – This is the size of the dirty data in the cache.
we can look at the wiredTiger.cache.bytes read into cache value for read-heavy applications. If this value is consistently high, increasing the cache size may improve overall read performance.
check whether the application is read-heavy. If it is, increase the size of the replica set and distribute the read operations to secondary members of the set.
write-heavy, use sharding within a sharded cluster to distribute the load.
Replication is the propagation of data from one node to another
Replication sets handle this replication.
Sometimes, data isn’t replicated as quickly as we’d like.
a particularly thorny problem if the lag between a primary and secondary node is high and the secondary becomes the primary
use the db.printSlaveReplicationInfo() or the rs.printSlaveReplicationInfo() command to see the status of a replica set from the perspective of the secondary member of the set.
shows how far behind the secondary members are from the primary. This number should be as low as possible.
monitor this metric closely.
watch for any spikes in replication delay.
Always investigate these issues to understand the reasons for the lag.
One replica set is primary. All others are secondary.
it’s not normal for nodes to change back and forth between primary and secondary.
use the profiler to gain a deeper understanding of the database’s behavior.
Enabling the profiler can affect system performance, due to the additional activity.
"globalLock.currentQueue.total: This number can indicate a possible concurrency issue if it's consistently high. This can happen if a lot of requests are waiting for a lock to be released."
rails dbconsole figures out which database you're using and drops you into whichever command line interface you would use with it
The console command lets you interact with your Rails application from the command line. On the underside, rails console uses IRB
rake about gives information about version numbers for Ruby, RubyGems, Rails, the Rails subcomponents, your application's folder, the current Rails environment name, your app's database adapter, and schema version
You can precompile the assets in app/assets using rake assets:precompile and remove those compiled assets using rake assets:clean.
rake db:version is useful when troubleshooting
The doc: namespace has the tools to generate documentation for your app, API documentation, guides.
rake notes will search through your code for comments beginning with FIXME, OPTIMIZE or TODO.
You can also use custom annotations in your code and list them using rake notes:custom by specifying the annotation using an environment variable ANNOTATION.
rake routes will list all of your defined routes, which is useful for tracking down routing problems in your app, or giving you a good overview of the URLs in an app you're trying to get familiar with.
rake secret will give you a pseudo-random key to use for your session secret.
Custom rake tasks have a .rake extension and are placed in
Rails.root/lib/tasks.
rails new . --git --database=postgresql
All commands can run with -h or --help to list more information
The rails server command launches a small web server named WEBrick which comes bundled with Ruby
rails server -e production -p 4000
You can run a server as a daemon by passing a -d option
The rails generate command uses templates to create a whole lot of things.
Using generators will save you a large amount of time by writing boilerplate code, code that is necessary for the app to work.
With a normal, plain-old Rails application, your URLs will generally follow the pattern of http://(host)/(controller)/(action), and a URL like http://(host)/(controller) will hit the index action of that controller.
A scaffold in Rails is a full set of model, database migration for that model, controller to manipulate it, views to view and manipulate the data, and a test suite for each of the above.
Unit tests are code that tests and makes assertions about code.
Unit tests are your friend.
rails console --sandbox
rails db
Each task has a description, and should help you find the thing you need.
rake tmp:clear clears all the three: cache, sessions and sockets.
Each subnet
must
reside entirely within one Availability Zone and cannot span zones.
Availability
Zones
are distinct locations that are engineered to be isolated from failures in other
Availability Zones
If a subnet's traffic is routed to an internet gateway, the subnet is known as a
public subnet.
If a subnet doesn't have a route to the internet gateway, the subnet is known as a
private subnet.
If a subnet doesn't have a route to the internet gateway, but has its traffic routed
to a
virtual private gateway for a VPN connection, the subnet is known as a
VPN-only subnet.
By default, all VPCs and subnets must have IPv4 CIDR blocks—you can't change
this behavior.
The allowed
block size is between a /16 netmask (65,536 IP addresses) and
/28 netmask (16 IP addresses).
The first four IP addresses and the last IP address in each subnet CIDR block are
not
available for you to use
The allowed block size is between a /28 netmask and
/16 netmask
The CIDR block must not overlap with any existing CIDR block that's associated with
the
VPC.
Each subnet must be associated with a route table
Every subnet that you create is automatically
associated with the main route table for the VPC
Security groups control inbound and outbound traffic for your
instances
network ACLs control inbound and outbound traffic for your subnets
each subnet must be associated with a network ACL
You can create a flow log on your VPC or subnet to capture the traffic that flows
to and
from the network interfaces in your VPC or subnet.
A VPC peering connection enables you to route traffic between the VPCs
using
private IP addresses
you cannot create a VPC peering connection between
VPCs that
have overlapping CIDR blocks
recommend that you create a VPC with a CIDR range large enough for expected
future growth, but not one that overlaps with current or expected future subnets anywhere
in
your corporate or home network, or that overlaps with current or future VPCs
PHP processes run as child processes of Apache, started on demand as needed by request volume.
Java processes take the opposite approach, with the JVM providing one massive uberprocess that reserves a large block of system resources (CPU and memory) on startup, with concurrency managed internally via threads
Processes in the twelve-factor app take strong cues from the unix process model for running service daemons.
allow Nginx to pass requests off to backend http servers for further processing
Nginx is often set up as a reverse proxy solution to help scale out infrastructure or to pass requests to other servers that are not designed to handle large client loads
explore buffering and caching to improve the performance of proxying operations for clients
Nginx is built to handle many concurrent connections at the same time.
provides you with flexibility in easily adding backend servers or taking them down as needed for maintenance
Proxying in Nginx is accomplished by manipulating a request aimed at the Nginx server and passing it to other servers for the actual processing
The servers that Nginx proxies requests to are known as upstream servers.
Nginx can proxy requests to servers that communicate using the http(s), FastCGI, SCGI, and uwsgi, or memcached protocols through separate sets of directives for each type of proxy
When a request matches a location with a proxy_pass directive inside, the request is forwarded to the URL given by the directive
For example, when a request for /match/here/please is handled by this block, the request URI will be sent to the example.com server as http://example.com/match/here/please
The request coming from Nginx on behalf of a client will look different than a request coming directly from a client
Nginx gets rid of any empty headers
Nginx, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request
The "Host" header is re-written to the value defined by the $proxy_host variable.
The upstream should not expect this connection to be persistent
Headers with empty values are completely removed from the passed request.
if your backend application will be processing non-standard headers, you must make sure that they do not have underscores
by default, this will be set to the value of $proxy_host, a variable that will contain the domain name or IP address and port taken directly from the proxy_pass definition
This is selected by default as it is the only address Nginx can be sure the upstream server responds to
(as it is pulled directly from the connection info)
$http_host: Sets the "Host" header to the "Host" header from the client request.
The headers sent by the client are always available in Nginx as variables. The variables will start with an $http_ prefix, followed by the header name in lowercase, with any dashes replaced by underscores.
preference to: the host name from the request line itself
set the "Host" header to the $host variable. It is the most flexible and will usually provide the proxied servers with a "Host" header filled in as accurately as possible
sets the "Host" header to the $host variable, which should contain information about the original host being requested
This variable takes the value of the original X-Forwarded-For header retrieved from the client and adds the Nginx server's IP address to the end.
The upstream directive must be set in the http context of your Nginx configuration.
http context
Once defined, this name will be available for use within proxy passes as if it were a regular domain name
By default, this is just a simple round-robin selection process (each request will be routed to a different host in turn)
Specifies that new connections should always be given to the backend that has the least number of active connections.
distributes requests to different servers based on the client's IP address.
mainly used with memcached proxying
As for the hash method, you must provide the key to hash against
Server Weight
Nginx's buffering and caching capabilities
Without buffers, data is sent from the proxied server and immediately begins to be transmitted to the client.
With buffers, the Nginx proxy will temporarily store the backend's response and then feed this data to the client
Nginx defaults to a buffering design
can be set in the http, server, or location contexts.
the sizing directives are configured per request, so increasing them beyond your need can affect your performance
When buffering is "off" only the buffer defined by the proxy_buffer_size directive will be used
A high availability (HA) setup is an infrastructure without a single point of failure, and your load balancers are a part of this configuration.
multiple load balancers (one active and one or more passive) behind a static IP address that can be remapped from one server to another.
Nginx also provides a way to cache content from backend servers
The proxy_cache_path directive must be set in the http context.
The proxy_cache_bypass directive is set to the $http_cache_control variable. This will contain an indicator as to whether the client is explicitly requesting a fresh, non-cached version of the resource
any user-related data should not be cached
For private content, you should set the Cache-Control header to "no-cache", "no-store", or "private" depending on the nature of the data
MongoDB uses a locking system to ensure data set consistency. If
certain operations are long-running or a queue forms, performance
will degrade as requests and operations wait for the lock.
performance limitations as a result of inadequate
or inappropriate indexing strategies, or as a consequence of poor schema
design patterns.
performance issues may be temporary and related to
abnormal traffic load.
If globalLock.currentQueue.total is consistently high,
then there is a chance that a large number of requests are waiting for
a lock.
If globalLock.totalTime is
high relative to uptime, the database has
existed in a lock state for a significant amount of time.
For write-heavy applications, deploy sharding and add one or more
shards to a sharded cluster to distribute load among
mongod instances.
Unless constrained by system-wide limits, the maximum number of
incoming connections supported by MongoDB is configured with the
maxIncomingConnections setting.
When logLevel is set to 0, MongoDB records slow
operations to the diagnostic log at a rate determined by
slowOpSampleRate.
At higher logLevel settings, all operations appear in
the diagnostic log regardless of their latency with the following
exception
Full Time Diagnostic Data Collection (FTDC) mechanism. FTDC data files
are compressed, are not human-readable, and inherit the same file access
permissions as the MongoDB data files.
mongod processes store FTDC data files in a
diagnostic.data directory under the instances
storage.dbPath.
"MongoDB uses a locking system to ensure data set consistency. If certain operations are long-running or a queue forms, performance will degrade as requests and operations wait for the lock."
In this case, Elasticsearch. And because Elasticsearch can be down or struggling, or the network can be down, the shipper would ideally be able to buffer and retry
Logstash is typically used for collecting, parsing, and storing logs for future use as part of log management.
Logstash’s biggest con or “Achille’s heel” has always been performance and resource consumption (the default heap size is 1GB).
This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.
Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.
differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.
Filebeat is just a tiny binary with no dependencies.
For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.
For example, the apache module will point Filebeat to default access.log and error.log paths
Filebeat’s scope is very limited,
Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.
Filebeat can parse JSON
you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.
You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off
For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well
The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.
It can tail files, parse them, buffer (on disk and in memory) and ship to a number of destinations, including Elasticsearch.
rsyslog is the fastest shipper
Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).
use it as a simple router/shipper, any decent machine will be limited by network bandwidth
It’s also one of the lightest parsers you can find, depending on the configured memory buffers.
rsyslog requires more work to get the configuration right
the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.
rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).
rsyslog also works well when you need that ultimate performance.
syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).
a modular syslog daemon, that can do much more than just syslog
Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.
Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.
syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance
Fluentd was built on the idea of logging in JSON wherever possible (which is a practice we totally agree with) so that log shippers down the line don’t have to guess which substring is which field of which type.
Fluentd plugins are in Ruby and very easy to write.
structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).
Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.
Fluentd is a good fit when you have diverse or exotic sources and destinations for your logs, because of the number of plugins.
Splunk isn’t a log shipper, it’s a commercial logging solution
Graylog is another complete logging solution, an open-source alternative to Splunk.
everything goes through graylog-server, from authentication to queries.
Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.