Group items tagged

must specify the public subnet in which the NAT gateway should reside

update the route table associated with one or more of your private subnets to point Internet-bound traffic to the NAT gateway.

NAT gateway is created in a specific Availability Zone and implemented with redundancy in that zone.

ensure that resources use the NAT gateway in the same Availability Zone

A NAT gateway supports 5 Gbps of bandwidth and automatically scales up to 45 Gbps

You can associate exactly one Elastic IP address with a NAT gateway

A NAT gateway supports the following protocols: TCP, UDP, and ICMP

cannot associate a security group with a NAT gateway.

create a NAT gateway in the same subnet as your NAT instance, and then replace the existing route in your route table that points to the NAT instance with a route that points to the NAT gateway

A NAT gateway cannot send traffic over VPC endpoints, VPN connections, AWS Direct Connect, or VPC peering connections.

Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout,

Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality.

Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

Wrk is so damn fast

being fast and measuring correctly is about all that Wrk does

k6 is scriptable in plain Javascript

k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Jmeter is a huge beast compared to most other tools.

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly.

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing.

use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through.

If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that.

Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU.

When you combine a custom resource with a custom controller, custom resources provide a true declarative API.

A declarative API allows you to declare or specify the desired state of your resource and tries to keep the current state of Kubernetes objects in sync with the desired state.

Custom controllers can work with any kind of resource, but they are especially effective when combined with custom resources.

the API represents a desired state, not an exact state.

define configuration of applications or infrastructure.

The main operations on the objects are CRUD-y (creating, reading, updating and deleting).

You want to put the entire config file into one key of a configMap.

You want to perform rolling updates via Deployment, etc., when the file is updated.

You want to build new automation that watches for updates on the new object, and then CRUD other objects, or vice versa.

You want the object to be an abstraction over a collection of controlled resources, or a summarization of other resources.

CRDs are simple and can be created without any programming.

CRDs allow users to create new types of resources without adding another API server

Defining a CRD object creates a new custom resource with a name and schema that you specify.

The name of a CRD object must be a valid DNS subdomain name

each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects.

The main API server delegates requests to you for the custom resources that you handle, making them available to all of its clients.

The new endpoints support CRUD basic operations via HTTP and kubectl

Custom resources consume storage space in the same way that ConfigMaps do.

Aggregated API servers may use the same storage as the main API server

CRDs always use the same authentication, authorization, and audit logging as the built-in resources of your API server.

most RBAC roles will not grant access to the new resources (except the cluster-admin role or any role created with wildcard rules).

This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.

Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.

differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.

Filebeat is just a tiny binary with no dependencies.

For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

For example, the apache module will point Filebeat to default access.log and error.log paths

Filebeat’s scope is very limited,

Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.

Filebeat can parse JSON

you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.

You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off

For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well

The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.

rsyslog is the fastest shipper

Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).

It’s also one of the lightest parsers you can find, depending on the configured memory buffers.

rsyslog requires more work to get the configuration right

the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.

rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).

rsyslog also works well when you need that ultimate performance.

syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).

a modular syslog daemon, that can do much more than just syslog

Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.

Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.

syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance

Fluentd plugins are in Ruby and very easy to write.

structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).

Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.

Splunk isn’t a log shipper, it’s a commercial logging solution

everything goes through graylog-server, from authentication to queries.

Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.

it depends

The Terraform Registry is the main directory of publicly available Terraform providers, and hosts providers for most major infrastructure platforms.

Dependency Lock File documents an additional HCL file that can be included with a configuration, which tells Terraform to always use a specific set of provider versions.

Terraform CLI finds and installs providers when initializing a working directory. It can automatically download providers from a Terraform registry, or load them from a local mirror or cache.

To save time and bandwidth, Terraform CLI supports an optional plugin cache. You can enable the cache using the plugin_cache_dir setting in the CLI configuration file.

you can use Terraform CLI to create a dependency lock file and commit it to version control along with your configuration.