Skip to main content

Home/ Larvata/ Contents contributed and discussions participated by 張 旭

Contents contributed and discussions participated by 張 旭

Simple Middle Filter: All | Bookmarks | Topics
張 旭

How to Write a Git Commit Message - 1 views

chris.beams.io/git-commit

programming git code

shared by 張 旭 on 04 Apr 20 - No Cached
  • a well-crafted Git commit message is the best way to communicate context about a change to fellow developers (and indeed to their future selves).
  • A diff will tell you what changed, but only the commit message can properly tell you why.
  • a commit message shows whether a developer is a good collaborator
  • ...22 more annotations...
  • a well-cared for log is a beautiful and useful thing
  • Reviewing others’ commits and pull requests becomes something worth doing, and suddenly can be done independently.
  • Understanding why something happened months or years ago becomes not only possible but efficient.
  • how to write an individual commit message.
  • Markup syntax, wrap margins, grammar, capitalization, punctuation.
  • What should it not contain?
  • issue tracking IDs
  • pull request numbers
  • The seven rules of a great Git commit message
  • Use the body to explain what and why vs. how
  • Use the imperative mood in the subject line
  • it’s a good idea to begin the commit message with a single short (less than 50 character) line summarizing the change, followed by a blank line and then a more thorough description.
  • forces the author to think for a moment about the most concise way to explain what’s going on.
  • If you’re having a hard time summarizing, you might be committing too many changes at once.
  • shoot for 50 characters, but consider 72 the hard limit
  • Imperative mood just means “spoken or written as if giving a command or instruction”.
  • Git itself uses the imperative whenever it creates a commit on your behalf.
  • when you write your commit messages in the imperative, you’re following Git’s own built-in conventions.
  • A properly formed Git commit subject line should always be able to complete the following sentence: If applied, this commit will your subject line here
  • explaining what changed and why
  • Code is generally self-explanatory in this regard (and if the code is so complex that it needs to be explained in prose, that’s what source comments are for).
  • there are tab completion scripts that take much of the pain out of remembering the subcommands and switches.
張 旭

Introduction to GitLab Flow | GitLab - 0 views

docs.gitlab.com/...gitlab_flow.html

programming git gitlab flow

shared by 張 旭 on 14 Mar 20 - No Cached
  • GitLab flow as a clearly defined set of best practices. It combines feature-driven development and feature branches with issue tracking.
  • In Git, you add files from the working copy to the staging area. After that, you commit them to your local repo. The third step is pushing to a shared remote repository.
  • branching model
  • ...68 more annotations...
  • The biggest problem is that many long-running branches emerge that all contain part of the changes.
  • It is a convention to call your default branch master and to mostly branch from and merge to this.
  • Nowadays, most organizations practice continuous delivery, which means that your default branch can be deployed.
  • Continuous delivery removes the need for hotfix and release branches, including all the ceremony they introduce.
  • Merging everything into the master branch and frequently deploying means you minimize the amount of unreleased code, which is in line with lean and continuous delivery best practices.
  • GitHub flow assumes you can deploy to production every time you merge a feature branch.
  • You can deploy a new version by merging master into the production branch. If you need to know what code is in production, you can just checkout the production branch to see.
  • Production branch
  • Environment branches
  • have an environment that is automatically updated to the master branch.
  • deploy the master branch to staging.
  • To deploy to pre-production, create a merge request from the master branch to the pre-production branch.
  • Go live by merging the pre-production branch into the production branch.
  • Release branches
  • work with release branches if you need to release software to the outside world.
  • each branch contains a minor version
  • After announcing a release branch, only add serious bug fixes to the branch.
  • merge these bug fixes into master, and then cherry-pick them into the release branch.
  • Merging into master and then cherry-picking into release is called an “upstream first” policy
  • Tools such as GitHub and Bitbucket choose the name “pull request” since the first manual action is to pull the feature branch.
  • Tools such as GitLab and others choose the name “merge request” since the final action is to merge the feature branch.
  • If you work on a feature branch for more than a few hours, it is good to share the intermediate result with the rest of the team.
  • the merge request automatically updates when new commits are pushed to the branch.
  • If the assigned person does not feel comfortable, they can request more changes or close the merge request without merging.
  • In GitLab, it is common to protect the long-lived branches, e.g., the master branch, so that most developers can’t modify them.
  • if you want to merge into a protected branch, assign your merge request to someone with maintainer permissions.
  • After you merge a feature branch, you should remove it from the source control software.
  • Having a reason for every code change helps to inform the rest of the team and to keep the scope of a feature branch small.
  • If there is no issue yet, create the issue
  • The issue title should describe the desired state of the system.
  • For example, the issue title “As an administrator, I want to remove users without receiving an error” is better than “Admin can’t remove users.”
  • create a branch for the issue from the master branch
  • If you open the merge request but do not assign it to anyone, it is a “Work In Progress” merge request.
  • Start the title of the merge request with [WIP] or WIP: to prevent it from being merged before it’s ready.
  • When they press the merge button, GitLab merges the code and creates a merge commit that makes this event easily visible later on.
  • Merge requests always create a merge commit, even when the branch could be merged without one. This merge strategy is called “no fast-forward” in Git.
  • Suppose that a branch is merged but a problem occurs and the issue is reopened. In this case, it is no problem to reuse the same branch name since the first branch was deleted when it was merged.
  • At any time, there is at most one branch for every issue.
  • It is possible that one feature branch solves more than one issue.
  • GitLab closes these issues when the code is merged into the default branch.
  • If you have an issue that spans across multiple repositories, create an issue for each repository and link all issues to a parent issue.
  • use an interactive rebase (rebase -i) to squash multiple commits into one or reorder them.
  • you should never rebase commits you have pushed to a remote server.
  • Rebasing creates new commits for all your changes, which can cause confusion because the same change would have multiple identifiers.
  • if someone has already reviewed your code, rebasing makes it hard to tell what changed since the last review.
  • never rebase commits authored by other people.
  • it is a bad idea to rebase commits that you have already pushed.
  • If you revert a merge commit and then change your mind, revert the revert commit to redo the merge.
  • Often, people avoid merge commits by just using rebase to reorder their commits after the commits on the master branch.
  • Using rebase prevents a merge commit when merging master into your feature branch, and it creates a neat linear history.
  • every time you rebase, you have to resolve similar conflicts.
  • Sometimes you can reuse recorded resolutions (rerere), but merging is better since you only have to resolve conflicts once.
  • A good way to prevent creating many merge commits is to not frequently merge master into the feature branch.
  • keep your feature branches short-lived.
  • Most feature branches should take less than one day of work.
  • If your feature branches often take more than a day of work, try to split your features into smaller units of work.
  • You could also use feature toggles to hide incomplete features so you can still merge back into master every day.
  • you should try to prevent merge commits, but not eliminate them.
  • Your codebase should be clean, but your history should represent what actually happened.
  • If you rebase code, the history is incorrect, and there is no way for tools to remedy this because they can’t deal with changing commit identifiers
  • Commit often and push frequently
  • You should push your feature branch frequently, even when it is not yet ready for review.
  • A commit message should reflect your intention, not just the contents of the commit.
  • each merge request must be tested before it is accepted.
  • test the master branch after each change.
  • If new commits in master cause merge conflicts with the feature branch, merge master back into the branch to make the CI server re-run the tests.
  • When creating a feature branch, always branch from an up-to-date master.
  • Do not merge from upstream again if your code can work and merge cleanly without doing so.
張 旭

Understanding the GitHub flow · GitHub Guides - 0 views

guides.github.com/...flow

git programming github flow

shared by 張 旭 on 14 Mar 20 - No Cached
  • anything in the master branch is always deployable.
  • Your branch name should be descriptive
  • Commits also create a transparent history of your work that others can follow to understand what you've done and why.
  • ...9 more annotations...
  • each commit is considered a separate unit of change.
  • By writing clear commit messages, you can make it easier for other people to follow along and provide feedback.
  • Pull Requests initiate discussion about your commits.
  • If you're using a Fork & Pull Model, Pull Requests provide a way to notify project maintainers about the changes you'd like them to consider.
  • Pull Requests are designed to encourage and capture this type of conversation.
  • You can also continue to push to your branch in light of discussion and feedback about your commits.
  • If your branch causes issues, you can roll it back by deploying the existing master into production.
  • With GitHub, you can deploy from a branch for final testing in production before merging to master.
  • your changes have been verified in production, it is time to merge your code into the master branch.
  • 張 旭 on 14 Mar 20
     
    "anything in the master branch is always deployable."
張 旭

git - What is the difference between GitHub Flow and GitLab Flow? - Stack Overflow - 0 views

stackoverflow.com/...en-github-flow-and-gitlab-flow

programming git

shared by 張 旭 on 14 Mar 20 - No Cached
  • in order to keep master a true record of known working production code the actual deployment to production should happen from the feature branch before merging it into master.
  • This approach works well if we seldom publish results of our work. (Maybe once every 2 weeks).
  • Aside from promoting ready to deploy master branch and feature branches (same as GitHub Flow) it introduces three other kinds of branches
張 旭

What is a CAA record? - DNSimple Help - 0 views

support.dnsimple.com/...caa-record

networking network dns

shared by 張 旭 on 10 Mar 20 - No Cached
  • The purpose of the CAA record is to allow domain owners to declare which certificate authorities are allowed to issue a certificate for a domain.
  • If a CAA record is present, only the CAs listed in the record(s) are allowed to issue certificates for that hostname.
  • CAA records can set policy for the entire domain, or for specific hostnames.
  • ...4 more annotations...
  • The CAA record consists of a flags byte and a tag-value pair referred to as a ‘property’.
  • example.com. CAA 0 issue "letsencrypt.org"
  • each CAA record contains only one tag-value pair
  • dig google.com type257
張 旭

Understanding Nginx Server and Location Block Selection Algorithms | DigitalOcean - 0 views

www.digitalocean.com/...ion-block-selection-algorithms

web server nginx config

shared by 張 旭 on 03 Feb 20 - No Cached
  • A server block is a subset of Nginx’s configuration that defines a virtual server used to handle requests of a defined type. Administrators often configure multiple server blocks and decide which block should handle which connection based on the requested domain name, port, and IP address.
  • A location block lives within a server block and is used to define how Nginx should handle requests for different resources and URIs for the parent server. The URI space can be subdivided in whatever way the administrator likes using these blocks. It is an extremely flexible model.
  • Nginx logically divides the configurations meant to serve different content into blocks, which live in a hierarchical structure. Each time a client request is made, Nginx begins a process of determining which configuration blocks should be used to handle the request.
  • ...37 more annotations...
  • Nginx is one of the most popular web servers in the world. It can successfully handle high loads with many concurrent client connections, and can easily function as a web server, a mail server, or a reverse proxy server.
  • The main server block directives that Nginx is concerned with during this process are the listen directive, and the server_name directive.
  • The listen directive typically defines which IP address and port that the server block will respond to.
  • 0.0.0.0:8080 if Nginx is being run by a normal, non-root user
  • Nginx translates all “incomplete” listen directives by substituting missing values with their default values so that each block can be evaluated by its IP address and port.
  • In any case, the port must be matched exactly.
  • If there are multiple server blocks with the same level of specificity matching, Nginx then begins to evaluate the server_name directive of each server block.
  • Nginx will only evaluate the server_name directive when it needs to distinguish between server blocks that match to the same level of specificity in the listen directive.
  • Nginx checks the request’s “Host” header. This value holds the domain or IP address that the client was actually trying to reach.
  • Nginx will first try to find a server block with a server_name that matches the value in the “Host” header of the request exactly.
  • If no exact match is found, Nginx will then try to find a server block with a server_name that matches using a leading wildcard (indicated by a * at the beginning of the name in the config).
  • If no match is found using a leading wildcard, Nginx then looks for a server block with a server_name that matches using a trailing wildcard (indicated by a server name ending with a * in the config)
  • If no match is found using a trailing wildcard, Nginx then evaluates server blocks that define the server_name using regular expressions (indicated by a ~ before the name).
  • If no regular expression match is found, Nginx then selects the default server block for that IP address and port.
  • There can be only one default_server declaration per each IP address/port combination.
  • Location blocks live within server blocks (or other location blocks) and are used to decide how to process the request URI (the part of the request that comes after the domain name or IP address/port).
  • If no modifiers are present, the location is interpreted as a prefix match.
  • =: If an equal sign is used, this block will be considered a match if the request URI exactly matches the location given.
  • ~: If a tilde modifier is present, this location will be interpreted as a case-sensitive regular expression match.
  • ~*: If a tilde and asterisk modifier is used, the location block will be interpreted as a case-insensitive regular expression match.
  • ^~: If a carat and tilde modifier is present, and if this block is selected as the best non-regular expression match, regular expression matching will not take place.
  • Keep in mind that if this block is selected and the request is fulfilled using an index page, an internal redirect will take place to another location that will be the actual handler of the request
  • Keeping in mind the types of location declarations we described above, Nginx evaluates the possible location contexts by comparing the request URI to each of the locations.
  • Nginx begins by checking all prefix-based location matches (all location types not involving a regular expression).
  • First, Nginx looks for an exact match.
  • If no exact (with the = modifier) location block matches are found, Nginx then moves on to evaluating non-exact prefixes.
  • After the longest matching prefix location is determined and stored, Nginx moves on to evaluating the regular expression locations (both case sensitive and insensitive).
  • by default, Nginx will serve regular expression matches in preference to prefix matches.
  • regular expression matches within the longest prefix match will “jump the line” when Nginx evaluates regex locations.
  • The exceptions to the “only one location block” rule may have implications on how the request is actually served and may not align with the expectations you had when designing your location blocks.
  • The index directive always leads to an internal redirect if it is used to handle the request.
  • In the case above, if you really need the execution to stay in the first block, you will have to come up with a different method of satisfying the request to the directory.
  • one way of preventing an index from switching contexts, but it’s probably not useful for most configurations
  • the try_files directive. This directive tells Nginx to check for the existence of a named set of files or directories.
  • the rewrite directive. When using the last parameter with the rewrite directive, or when using no parameter at all, Nginx will search for a new matching location based on the results of the rewrite.
  • The error_page directive can lead to an internal redirect similar to that created by try_files.
  • when certain status codes are encountered.
張 旭

A visual guide on troubleshooting Kubernetes deployments - 0 views

learnk8s.io/troubleshooting-deployments

kubernetes pod container service HA proxy

shared by 張 旭 on 06 Dec 19 - No Cached
  • Service and Deployment aren't connected at all.
  • the Service points to the Pods directly and skips the Deployment altogether.
張 旭

[Kubernetes] Taints and Tolerations | 小信豬的原始部落 - 0 views

godleon.github.io/...k8s-Taints-and-Tolerations

kubernetes deploy pod container

shared by 張 旭 on 06 Dec 19 - No Cached
  • 如果有特定的 node 被加上了 taint(汙點)，pod 就不會被分派到上面，除非 pod spec 有設定 toleration(容忍) 來接受這些 taint (必須全部 taint 都接受才行)
  • 假設某個 node 被設定了 effect 為 NoExecute 的 taint，那 k8s 還會把已經存在該 node 上的 pod 趕走，也不會把該 pod 分派到該 node 上。
  • taint 機制設計的目的，就是不要讓 pod 被分派到某個 node 上
  • ...1 more annotation...
  • 當 node 發生問題時(或是任何其他會造成該 node 無法繼續提供服務的情況)，管理者需要考慮驅逐目前在上面運行中的 pod，可以透過加上 taint(Effect=NoExecute) 的方式達成
張 旭

Memory inside Linux containers | Fabio Kung - 0 views

fabiokung.com/...memory-inside-linux-containers

linux docker system process

shared by 張 旭 on 05 Dec 19 - No Cached
  • /sys/fs/cgroup/ is the recommended location for cgroup hierarchies, but it is not a standard.
  • most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others.
  • cat /sys/fs/cgroup/memory/memory.stat
  • ...3 more annotations...
  • /sys/fs/cgroup is just an umbrella for all cgroup hierarchies, there is no recommendation or standard for my own cgroup location.
  • an userspace library that processes can use to query their memory usage and available memory.
  • we might need to encourage people to stop using those tools inside containers.
張 旭

HowTo/LDAP - FreeIPA - 0 views

www.freeipa.org/LDAP

ldap authorization authentication auth security

shared by 張 旭 on 03 Dec 19 - No Cached
  • The basedn in an IPA installation consists of a set of domain components (dc) for the initial domain that IPA was configured with.
  • You will only ever have one basedn, the one defined during installation.
  • find your basedn, and other interesting things, in /etc/ipa/default.conf
  • ...8 more annotations...
  • IPA uses a flat structure, storing like objects in what we call containers.
  • Users: cn=users,cn=accounts,$SUFFIX Groups: cn=groups,cn=accounts,$SUFFIX
  • Do not use the Directory Manager account to authenticate remote services to the IPA LDAP server. Use a system account
  • The reason to use an account like this rather than creating a normal user account in IPA and using that is that the system account exists only for binding to LDAP. It is not a real POSIX user, can't log into any systems and doesn't own any files.
  • This use also has no special rights and is unable to write any data in the IPA LDAP server, only read.
  • When possible, configure your LDAP client to communicate over SSL/TLS.
  • The IPA CA certificate can be found in /etc/ipa/ca.crt
  • /etc/openldap/ldap.conf
張 旭

The Backup Cycle - Full Backups - 0 views

www.percona.com/...full_backup.html

database mysql backup percona xtradb

shared by 張 旭 on 02 Dec 19 - No Cached
  • xtrabackup will not overwrite existing files, it will fail with operating system error 17, file exists.
  • Log copying thread checks the transactional log every second to see if there were any new log records written that need to be copied, but there is a chance that the log copying thread might not be able to keep up with the amount of writes that go to the transactional logs, and will hit an error when the log records are overwritten before they could be read.
  • It is safe to cancel at any time, because xtrabackup does not modify the database.
  • ...15 more annotations...
  • need to prepare it in order to restore it.
  • Data files are not point-in-time consistent until they are prepared, because they were copied at different times as the program ran, and they might have been changed while this was happening.
  • You can run the prepare operation on any machine; it does not need to be on the originating server or the server to which you intend to restore.
  • you simply run xtrabackup with the --prepare option and tell it which directory to prepare,
  • All following prepares will not change the already prepared data files
  • It is not recommended to interrupt xtrabackup process while preparing backup
  • Backup validity is not guaranteed if prepare process was interrupted.
  • If you intend the backup to be the basis for further incremental backups, you should use the --apply-log-only option when preparing the backup, or you will not be able to apply incremental backups to it.
  • Backup needs to be prepared before it can be restored.
  • xtrabackup --copy-back --target-dir=/data/backups/
  • The datadir must be empty before restoring the backup.
  • MySQL server needs to be shut down before restore is performed.
  • You cannot restore to a datadir of a running mysqld instance (except when importing a partial backup).
  • rsync -avrP /data/backup/ /var/lib/mysql/
  • chown -R mysql:mysql /var/lib/mysql
張 旭

Incremental Backup - 0 views

www.percona.com/...incremental_backup.html

mysql database backup percona xtradb

shared by 張 旭 on 02 Dec 19 - No Cached
  • xtrabackup supports incremental backups, which means that they can copy only the data that has changed since the last backup.
  • You can perform many incremental backups between each full backup, so you can set up a backup process such as a full backup once a week and an incremental backup every day, or full backups every day and incremental backups every hour.
  • each InnoDB page contains a log sequence number, or LSN. The LSN is the system version number for the entire database. Each page’s LSN shows how recently it was changed.
  • ...18 more annotations...
  • In full backups, two types of operations are performed to make the database consistent: committed transactions are replayed from the log file against the data files, and uncommitted transactions are rolled back.
  • You should use the --apply-log-only option to prevent the rollback phase.
  • An incremental backup copies each page whose LSN is newer than the previous incremental or full backup’s LSN.
  • Incremental backups do not actually compare the data files to the previous backup’s data files.
  • you can use --incremental-lsn to perform an incremental backup without even having the previous backup, if you know its LSN
  • Incremental backups simply read the pages and compare their LSN to the last backup’s LSN.
  • without a full backup to act as a base, the incremental backups are useless.
  • The xtrabackup binary writes a file called xtrabackup_checkpoints into the backup’s target directory. This file contains a line showing the to_lsn, which is the database’s LSN at the end of the backup.
  • from_lsn is the starting LSN of the backup and for incremental it has to be the same as to_lsn (if it is the last checkpoint) of the previous/base backup.
  • If you do not use the --apply-log-only option to prevent the rollback phase, then your incremental backups will be useless.
  • run --prepare as usual, but prevent the rollback phase
  • If you restore it and start MySQL, InnoDB will detect that the rollback phase was not performed, and it will do that in the background, as it usually does for a crash recovery upon start.
  • xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc1
  • The final data is in /data/backups/base, not in the incremental directory.
  • Do not run xtrabackup --prepare with the same incremental backup directory (the value of –incremental-dir) more than once.
  • xtrabackup --prepare --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc2
  • --apply-log-only should be used when merging all incrementals except the last one.
  • Even if the --apply-log-only was used on the last step, backup would still be consistent but in that case server would perform the rollback phase.
張 旭

Connection and Privileges Needed - 0 views

www.percona.com/...privileges.html

mysql database backup percona xtradb

shared by 張 旭 on 02 Dec 19 - No Cached
  • Percona XtraBackup needs to be able to connect to the database server and perform operations on the server and the datadir when creating a backup, when preparing in some scenarios and when restoring it.
  • When xtrabackup is used, there are two actors involved: the user invoking the program - a system user - and the user performing action in the database server - a database user.
  • these are different users in different places, even though they may have the same username.
  • ...1 more annotation...
  • Once connected to the server, in order to perform a backup you will need READ and EXECUTE permissions at a filesystem level in the server’s datadir.
  • 張 旭 on 02 Dec 19
     
    "Percona XtraBackup needs to be able to connect to the database server and perform operations on the server and the datadir when creating a backup, when preparing in some scenarios and when restoring it. "
張 旭

How Percona XtraBackup Works - 0 views

www.percona.com/...how_xtrabackup_works.html

mysql database backup percona xtradb

shared by 張 旭 on 30 Nov 19 - No Cached
  • Percona XtraBackup is based on InnoDB‘s crash-recovery functionality.
  • it performs crash recovery on the files to make them a consistent, usable database again
  • InnoDB maintains a redo log, also called the transaction log. This contains a record of every change to InnoDB data.
  • ...14 more annotations...
  • When InnoDB starts, it inspects the data files and the transaction log, and performs two steps. It applies committed transaction log entries to the data files, and it performs an undo operation on any transactions that modified data but did not commit.
  • Percona XtraBackup works by remembering the log sequence number (LSN) when it starts, and then copying away the data files.
  • Percona XtraBackup runs a background process that watches the transaction log files, and copies changes from it.
  • Percona XtraBackup needs to do this continually
  • Percona XtraBackup needs the transaction log records for every change to the data files since it began execution.
  • Percona XtraBackup uses Backup locks where available as a lightweight alternative to FLUSH TABLES WITH READ LOCK.
  • Locking is only done for MyISAM and other non-InnoDB tables after Percona XtraBackup finishes backing up all InnoDB/XtraDB data and logs.
  • xtrabackup tries to avoid backup locks and FLUSH TABLES WITH READ LOCK when the instance contains only InnoDB tables. In this case, xtrabackup obtains binary log coordinates from performance_schema.log_status
  • When backup locks are supported by the server, xtrabackup first copies InnoDB data, runs the LOCK TABLES FOR BACKUP and then copies the MyISAM tables.
  • the STDERR of xtrabackup is not written in any file. You will have to redirect it to a file, e.g., xtrabackup OPTIONS 2> backupout.log
  • During the prepare phase, Percona XtraBackup performs crash recovery against the copied data files, using the copied transaction log file. After this is done, the database is ready to restore and use.
  • the tools enable you to do operations such as streaming and incremental backups with various combinations of copying the data files, copying the log files, and applying the logs to the data.
  • To restore a backup with xtrabackup you can use the --copy-back or --move-back options.
  • you may have to change the files’ ownership to mysql before starting the database server, as they will be owned by the user who created the backup.
  • 張 旭 on 30 Nov 19
     
    "Percona XtraBackup is based on InnoDB's crash-recovery functionality."
張 旭

Load balancing with ProxySQL - 0 views

www.percona.com/...proxysql.html

proxysql mysql database HA

shared by 張 旭 on 07 Nov 19 - No Cached
  • accepts incoming traffic from MySQL clients and forwards it to backend MySQL servers.
張 旭

Logstash Alternatives: Pros & Cons of 5 Log Shippers [2019] - Sematext - 0 views

sematext.com/...logstash-alternatives

log monitor system logstash fluentd

shared by 張 旭 on 05 Nov 19 - No Cached
  • In this case, Elasticsearch. And because Elasticsearch can be down or struggling, or the network can be down, the shipper would ideally be able to buffer and retry
  • Logstash is typically used for collecting, parsing, and storing logs for future use as part of log management.
  • Logstash’s biggest con or “Achille’s heel” has always been performance and resource consumption (the default heap size is 1GB).
  • ...37 more annotations...
  • This can be a problem for high traffic deployments, when Logstash servers would need to be comparable with the Elasticsearch ones.
  • Filebeat was made to be that lightweight log shipper that pushes to Logstash or Elasticsearch.
  • differences between Logstash and Filebeat are that Logstash has more functionality, while Filebeat takes less resources.
  • Filebeat is just a tiny binary with no dependencies.
  • For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.
  • For example, the apache module will point Filebeat to default access.log and error.log paths
  • Filebeat’s scope is very limited,
  • Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis, and in 5.x it also gains filtering capabilities.
  • Filebeat can parse JSON
  • you can push directly from Filebeat to Elasticsearch, and have Elasticsearch do both parsing and storing.
  • You shouldn’t need a buffer when tailing files because, just as Logstash, Filebeat remembers where it left off
  • For larger deployments, you’d typically use Kafka as a queue instead, because Filebeat can talk to Kafka as well
  • The default syslog daemon on most Linux distros, rsyslog can do so much more than just picking logs from the syslog socket and writing to /var/log/messages.
  • It can tail files, parse them, buffer (on disk and in memory) and ship to a number of destinations, including Elasticsearch.
  • rsyslog is the fastest shipper
  • Its grammar-based parsing module (mmnormalize) works at constant speed no matter the number of rules (we tested this claim).
  • use it as a simple router/shipper, any decent machine will be limited by network bandwidth
  • It’s also one of the lightest parsers you can find, depending on the configured memory buffers.
  • rsyslog requires more work to get the configuration right
  • the main difference between Logstash and rsyslog is that Logstash is easier to use while rsyslog lighter.
  • rsyslog fits well in scenarios where you either need something very light yet capable (an appliance, a small VM, collecting syslog from within a Docker container).
  • rsyslog also works well when you need that ultimate performance.
  • syslog-ng as an alternative to rsyslog (though historically it was actually the other way around).
  • a modular syslog daemon, that can do much more than just syslog
  • Unlike rsyslog, it features a clear, consistent configuration format and has nice documentation.
  • Similarly to rsyslog, you’d probably want to deploy syslog-ng on boxes where resources are tight, yet you do want to perform potentially complex processing.
  • syslog-ng has an easier, more polished feel than rsyslog, but likely not that ultimate performance
  • Fluentd was built on the idea of logging in JSON wherever possible (which is a practice we totally agree with) so that log shippers down the line don’t have to guess which substring is which field of which type.
  • Fluentd plugins are in Ruby and very easy to write.
  • structured data through Fluentd, it’s not made to have the flexibility of other shippers on this list (Filebeat excluded).
  • Fluent Bit, which is to Fluentd similar to how Filebeat is for Logstash.
  • Fluentd is a good fit when you have diverse or exotic sources and destinations for your logs, because of the number of plugins.
  • Splunk isn’t a log shipper, it’s a commercial logging solution
  • Graylog is another complete logging solution, an open-source alternative to Splunk.
  • everything goes through graylog-server, from authentication to queries.
  • Graylog is nice because you have a complete logging solution, but it’s going to be harder to customize than an ELK stack.
  • it depends
張 旭

Operator pattern - Kubernetes - 1 views

kubernetes.io/...operator

kubernetes system

shared by 張 旭 on 31 Oct 19 - No Cached
  • The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services
  • Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components
  • The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides.
  • ...7 more annotations...
  • Operators are clients of the Kubernetes API that act as controllers for a Custom Resource.
  • choosing a leader for a distributed application without an internal member election process
  • publishing a Service to applications that don't support Kubernetes APIs to discover them
  • The core of the Operator is code to tell the API server how to make reality match the configured resources.
  • If you add a new SampleDB, the operator sets up PersistentVolumeClaims to provide durable database storage, a StatefulSet to run SampleDB and a Job to handle initial configuration.If you delete it, the Operator takes a snapshot, then makes sure that the StatefulSet and Volumes are also removed.
  • to deploy an Operator is to add the Custom Resource Definition and its associated Controller to your cluster.
  • Once you have an Operator deployed, you'd use it by adding, modifying or deleting the kind of resource that the Operator uses.
« First ‹ Previous 161 - 180 of 596 Next › Last »
Showing 20▼ items per page