Skip to main content

Home/ Larvata/ Group items tagged high

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
crazylion lee

Keepalived for Linux - 0 views

www.keepalived.org

clustering HA failover keepalived

shared by crazylion lee on 15 Aug 14 - Cached
  • crazylion lee on 15 Aug 14
     
    Keepalived is a routing software written in C. The main goal of this project is to provide simple and robust facilities for loadbalancing and high-availability to Linux system and Linux based infrastructures. Loadbalancing framework relies on well-known and widely used Linux Virtual Server (IPVS) kernel module providing Layer4 loadbalancing. Keepalived implements a set of checkers to dynamically and adaptively maintain and manage loadbalanced server pool according their health. On the other hand high-availability is achieved by VRRP protocol. VRRP is a fundamental brick for router failover. In addition, Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions. Keepalived frameworks can be used independently or all together to provide resilient infrastructures.
張 旭

MongoDB Performance Tuning: Everything You Need to Know - Stackify - 0 views

stackify.com/mongodb-performance-tuning

database mongodb

shared by 張 旭 on 15 Apr 21 - No Cached
  • db.serverStatus().globalLock
  • db.serverStatus().locks
  • globalLock.currentQueue.total: This number can indicate a possible concurrency issue if it’s consistently high. This can happen if a lot of requests are waiting for a lock to be released.
  • ...35 more annotations...
  • globalLock.totalTime: If this is higher than the total database uptime, the database has been in a lock state for too long.
  • Unlike relational databases such as MySQL or PostgreSQL, MongoDB uses JSON-like documents for storing data.
  • Databases operate in an environment that consists of numerous reads, writes, and updates.
  • When a lock occurs, no other operation can read or modify the data until the operation that initiated the lock is finished.
  • locks.deadlockCount: Number of times the lock acquisitions have encountered deadlocks
  • Is the database frequently locking from queries? This might indicate issues with the schema design, query structure, or system architecture.
  • For version 3.2 on, WiredTiger is the default.
  • MMAPv1 locks whole collections, not individual documents.
  • WiredTiger performs locking at the document level.
  • When the MMAPv1 storage engine is in use, MongoDB will use memory-mapped files to store data.
  • All available memory will be allocated for this usage if the data set is large enough.
  • db.serverStatus().mem
  • mem.resident: Roughly equivalent to the amount of RAM in megabytes that the database process uses
  • If mem.resident exceeds the value of system memory and there’s a large amount of unmapped data on disk, we’ve most likely exceeded system capacity.
  • If the value of mem.mapped is greater than the amount of system memory, some operations will experience page faults.
  • The WiredTiger storage engine is a significant improvement over MMAPv1 in performance and concurrency.
  • By default, MongoDB will reserve 50 percent of the available memory for the WiredTiger data cache.
  • wiredTiger.cache.bytes currently in the cache – This is the size of the data currently in the cache.
  • wiredTiger.cache.tracked dirty bytes in the cache – This is the size of the dirty data in the cache.
  • we can look at the wiredTiger.cache.bytes read into cache value for read-heavy applications. If this value is consistently high, increasing the cache size may improve overall read performance.
  • check whether the application is read-heavy. If it is, increase the size of the replica set and distribute the read operations to secondary members of the set.
  • write-heavy, use sharding within a sharded cluster to distribute the load.
  • Replication is the propagation of data from one node to another
  • Replication sets handle this replication.
  • Sometimes, data isn’t replicated as quickly as we’d like.
  • a particularly thorny problem if the lag between a primary and secondary node is high and the secondary becomes the primary
  • use the db.printSlaveReplicationInfo() or the rs.printSlaveReplicationInfo() command to see the status of a replica set from the perspective of the secondary member of the set.
  • shows how far behind the secondary members are from the primary. This number should be as low as possible.
  • monitor this metric closely.
  • watch for any spikes in replication delay.
  • Always investigate these issues to understand the reasons for the lag.
  • One replica set is primary. All others are secondary.
  • it’s not normal for nodes to change back and forth between primary and secondary.
  • use the profiler to gain a deeper understanding of the database’s behavior.
  • Enabling the profiler can affect system performance, due to the additional activity.
  • 張 旭 on 15 Apr 21
     
    "globalLock.currentQueue.total: This number can indicate a possible concurrency issue if it's consistently high. This can happen if a lot of requests are waiting for a lock to be released."
張 旭

Setup ProxySQL for High Availability (not a Single Point of Failure) - Percona Database... - 0 views

www.percona.com/...ility-not-single-point-failure

mysql database cluster mariadb proxy

shared by 張 旭 on 18 Nov 18 - No Cached
  • ProxySQL doesn’t natively support any high availability solution
  • most common solution is setting up ProxySQL as part of a tile architecture, where Application/ProxySQL are deployed together.
    • 張 旭
      張 旭 on 18 Nov 18
       
      直接把 ProxySQL 跟 App 捆綁發佈
  • If we have 400 instances of ProxySQL, we end up keeping our databases busy just performing the checks.
  • ...5 more annotations...
  • Another possible approach is to have two layers of ProxySQL, one close to the application and another in the middle to connect to the database.
  • creates additional complexity in the management of the platform, and it adds network hops.
  • combining existing solutions and existing blocks: KeepAlived + ProxySQl + MySQL.
  • Keepalived implements a set of checkers to dynamically and adaptively maintain and manage load-balanced server pool according to their health.
  • Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions.
crazylion lee

twitter/distributedlog: A high performance replicated log service. - 0 views

github.com/...distributedlog

twitter log distributed

shared by crazylion lee on 26 Jun 16 - No Cached
  • crazylion lee on 26 Jun 16
     
    "A high performance replicated log service. http://distributedlog.io"
crazylion lee

GitHub - joyieldInc/predixy: A high performance and full features proxy for redis, supp... - 0 views

github.com/predixy

redis proxy

shared by crazylion lee on 29 Aug 17 - No Cached
  • crazylion lee on 29 Aug 17
     
    "A high performance and full features proxy for redis, support redis sentinel and redis cluster"
張 旭

An Introduction to HAProxy and Load Balancing Concepts | DigitalOcean - 0 views

www.digitalocean.com/...xy-and-load-balancing-concepts

server haproxy loading HA architecture

shared by 張 旭 on 07 Dec 17 - No Cached
  • HAProxy, which stands for High Availability Proxy
  • improve the performance and reliability of a server environment by distributing the workload across multiple servers (e.g. web, application, database).
  • ACLs are used to test some condition and perform an action (e.g. select a server, or block a request) based on the test result.
  • ...28 more annotations...
  • Access Control List (ACL)
  • ACLs allows flexible network traffic forwarding based on a variety of factors like pattern-matching and the number of connections to a backend
  • A backend is a set of servers that receives forwarded requests
  • adding more servers to your backend will increase your potential load capacity by spreading the load over multiple servers
  • mode http specifies that layer 7 proxying will be used
  • specifies the load balancing algorithm
  • health checks
  • A frontend defines how requests should be forwarded to backends
  • use_backend rules, which define which backends to use depending on which ACL conditions are matched, and/or a default_backend rule that handles every other case
  • A frontend can be configured to various types of network traffic
  • Load balancing this way will forward user traffic based on IP range and port
  • Generally, all of the servers in the web-backend should be serving identical content--otherwise the user might receive inconsistent content.
  • Using layer 7 allows the load balancer to forward requests to different backend servers based on the content of the user's request.
  • allows you to run multiple web application servers under the same domain and port
  • acl url_blog path_beg /blog matches a request if the path of the user's request begins with /blog.
  • Round Robin selects servers in turns
  • Selects the server with the least number of connections--it is recommended for longer sessions
  • This selects which server to use based on a hash of the source IP
  • ensure that a user will connect to the same server
  • require that a user continues to connect to the same backend server. This persistence is achieved through sticky sessions, using the appsession parameter in the backend that requires it.
  • HAProxy uses health checks to determine if a backend server is available to process requests.
  • The default health check is to try to establish a TCP connection to the server
  • If a server fails a health check, and therefore is unable to serve requests, it is automatically disabled in the backend
  • For certain types of backends, like database servers in certain situations, the default health check is insufficient to determine whether a server is still healthy.
  • However, your load balancer is a single point of failure in these setups; if it goes down or gets overwhelmed with requests, it can cause high latency or downtime for your service.
  • A high availability (HA) setup is an infrastructure without a single point of failure
  • a static IP address that can be remapped from one server to another.
  • If that load balancer fails, your failover mechanism will detect it and automatically reassign the IP address to one of the passive servers.
張 旭

Cluster Networking - Kubernetes - 0 views

kubernetes.io/...networking

kubernetes networking network system

shared by 張 旭 on 08 Jul 19 - No Cached
  • Networking is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work
  • Highly-coupled container-to-container communications
  • Pod-to-Pod communications
  • ...57 more annotations...
  • this is the primary focus of this document
    • 張 旭
      張 旭 on 27 Aug 21
       
      Cluster Networking 所關注處理的是： Pod 到 Pod 之間的連線
  • Pod-to-Service communications
  • External-to-Service communications
  • Kubernetes is all about sharing machines between applications.
  • sharing machines requires ensuring that two applications do not try to use the same ports.
  • Dynamic port allocation brings a lot of complications to the system
  • Every Pod gets its own IP address
  • do not need to explicitly create links between Pods
  • almost never need to deal with mapping container ports to host ports.
  • Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.
  • pods on a node can communicate with all pods on all nodes without NAT
  • agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node
  • pods in the host network of a node can communicate with all pods on all nodes without NAT
  • If your job previously ran in a VM, your VM had an IP and could talk to other VMs in your project. This is the same basic model.
  • containers within a Pod share their network namespaces - including their IP address
  • containers within a Pod can all reach each other’s ports on localhost
  • containers within a Pod must coordinate port usage
  • “IP-per-pod” model.
  • request ports on the Node itself which forward to your Pod (called host ports), but this is a very niche operation
  • The Pod itself is blind to the existence or non-existence of host ports.
  • AOS is an Intent-Based Networking system that creates and manages complex datacenter environments from a simple integrated platform.
  • Cisco Application Centric Infrastructure offers an integrated overlay and underlay SDN solution that supports containers, virtual machines, and bare metal servers.
  • AOS Reference Design currently supports Layer-3 connected hosts that eliminate legacy Layer-2 switching problems.
  • The AWS VPC CNI offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters.
  • users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters.
  • Using this CNI plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.
  • The CNI allocates AWS Elastic Networking Interfaces (ENIs) to each Kubernetes node and using the secondary IP range from each ENI for pods on the node.
  • Big Cloud Fabric is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments.
  • Cilium is L7/HTTP aware and can enforce network policies on L3-L7 using an identity based security model that is decoupled from network addressing.
  • CNI-Genie is a CNI plugin that enables Kubernetes to simultaneously have access to different implementations of the Kubernetes network model in runtime.
  • CNI-Genie also supports assigning multiple IP addresses to a pod, each from a different CNI plugin.
  • cni-ipvlan-vpc-k8s contains a set of CNI and IPAM plugins to provide a simple, host-local, low latency, high throughput, and compliant networking stack for Kubernetes within Amazon Virtual Private Cloud (VPC) environments by making use of Amazon Elastic Network Interfaces (ENI) and binding AWS-managed IPs into Pods using the Linux kernel’s IPvlan driver in L2 mode.
  • to be straightforward to configure and deploy within a VPC
  • Contiv provides configurable networking
  • Contrail, based on Tungsten Fabric, is a truly open, multi-cloud network virtualization and policy management platform.
  • DANM is a networking solution for telco workloads running in a Kubernetes cluster.
  • Flannel is a very simple overlay network that satisfies the Kubernetes requirements.
  • Any traffic bound for that subnet will be routed directly to the VM by the GCE network fabric.
  • sysctl net.ipv4.ip_forward=1
  • Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.
  • Knitter is a network solution which supports multiple networking in Kubernetes.
  • Kube-OVN is an OVN-based kubernetes network fabric for enterprises.
  • Kube-router provides a Linux LVS/IPVS-based service proxy, a Linux kernel forwarding-based pod-to-pod networking solution with no overlays, and iptables/ipset-based network policy enforcer.
  • If you have a “dumb” L2 network, such as a simple switch in a “bare-metal” environment, you should be able to do something similar to the above GCE setup.
  • Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.
  • NSX-T can provide network virtualization for a multi-cloud and multi-hypervisor environment and is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks.
  • NSX-T Container Plug-in (NCP) provides integration between NSX-T and container orchestrators such as Kubernetes
  • Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.
  • OpenVSwitch is a somewhat more mature but also complicated way to build an overlay network
  • OVN is an opensource network virtualization solution developed by the Open vSwitch community.
  • Project Calico is an open source container networking provider and network policy engine.
  • Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet
  • Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking.
  • Calico can also be run in policy enforcement mode in conjunction with other networking solutions such as Flannel, aka canal, or native GCE, AWS or Azure networking.
  • Romana is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network
  • Weave Net runs as a CNI plug-in or stand-alone. In either version, it doesn’t require any configuration or extra code to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.
  • The network model is implemented by the container runtime on each node.
crazylion lee

GNU Octave - 0 views

www.gnu.org/octave

matlab opensource math

shared by crazylion lee on 10 Jun 16 - No Cached
  • crazylion lee on 10 Jun 16
     
    "GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. Octave is normally used through its interactive command line interface, but it can also be used to write non-interactive programs. The Octave language is quite similar to Matlab so that most programs are easily portable."
crazylion lee

YubiKey 4 and YubiKey 4 Nano | U2F, OTP, PIV | Yubico - 0 views

www.yubico.com/...yubikey4

key lock hardware

shared by crazylion lee on 26 Apr 16 - No Cached
  • crazylion lee on 26 Apr 16
     
    "The YubiKey 4 is the strong authentication bullseye the industry has been aiming at for years, enabling one single key to secure an unlimited number of applications. Yubico's 4th generation YubiKey is built on high-performance secure elements. It includes the same range of one-time password and public key authentication protocols as in the YubiKey NEO, excluding NFC, but with stronger public/private keys, faster crypto operations and the world's first touch-to-sign feature. With the YubiKey 4 platform, we have further improved our manufacturing and ordering process, enabling customers to order exactly what functions they want in 500+ unit volumes, with no secrets stored at Yubico or shared with a third-party organization. The best part? An organization can securely customize 1,000 YubiKeys in less than 10 minutes. For customers who require NFC, the YubiKey NEO is our full-featured key with both contact (USB) and contactless (NFC, MIFARE) communications."
crazylion lee

Overview - DistributedLog 1.0 documentation - 0 views

distributedlog.io

twitter log server

shared by crazylion lee on 27 Jun 16 - No Cached
  • crazylion lee on 27 Jun 16
     
    "DistributedLog (DL) is a high-performance, replicated log service, offering durability, replication and strong consistency as essentials for building reliable distributed systems. "
crazylion lee

Computer Science Field Guide - Computer Science Field Guide - 0 views

www.csfieldguide.org.nz/...index.html

computer computer science science guide field google algorithm

shared by crazylion lee on 21 Jun 16 - No Cached
  • crazylion lee on 21 Jun 16
     
    "An online interactive resource for high school students learning about computer science"
crazylion lee

Apache NiFi - 0 views

nifi.apache.org/index.html

apache workflow file

shared by crazylion lee on 24 Sep 15 - No Cached
  • crazylion lee on 24 Sep 15
     
    "Apache nifi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:"
crazylion lee

Five Open-Source Slack Alternatives » okTurtles Blog - 0 views

blog.okturtles.com/...open-source-slack-alternatives

slack opensource

shared by crazylion lee on 08 Nov 15 - No Cached
  • crazylion lee on 08 Nov 15
     
    " Slack is a popular team communications application for organizations that offers group chat and direct messaging for mobile, web, and desktop platforms. While Slack offers many benefits to customers, there are also downsides to using the platform, including high subscription fees and the risk of a massive leak of private data if Slack's servers are ever breached (again)."
張 旭

Basics - Træfik - 0 views

docs.traefik.io/basics

docker proxy networking

shared by 張 旭 on 28 Aug 18 - No Cached
  • Modifier rules only modify the request. They do not have any impact on routing decisions being made.
  • A frontend consists of a set of rules that determine how incoming requests are forwarded from an entrypoint to a backend.
  • Entrypoints are the network entry points into Træfik
  • ...27 more annotations...
  • Modifiers and matchers
  • Matcher rules determine if a particular request should be forwarded to a backend
  • if any rule matches
  • if all rules match
  • In order to use regular expressions with Host and Path matchers, you must declare an arbitrarily named variable followed by the colon-separated regular expression, all enclosed in curly braces.
  • Use a *Prefix* matcher if your backend listens on a particular base path but also serves requests on sub-paths. For instance, PathPrefix: /products would match /products but also /products/shoes and /products/shirts. Since the path is forwarded as-is, your backend is expected to listen on /products
  • Use Path if your backend listens on the exact path only. For instance, Path: /products would match /products but not /products/shoes.
  • Modifier rules ALWAYS apply after the Matcher rules.
  • A backend is responsible to load-balance the traffic coming from one or more frontends to a set of http servers
  • wrr: Weighted Round Robin
  • drr: Dynamic Round Robin: increases weights on servers that perform better than others.
  • A circuit breaker can also be applied to a backend, preventing high loads on failing servers.
  • To proactively prevent backends from being overwhelmed with high load, a maximum connection limit can also be applied to each backend.
  • Sticky sessions are supported with both load balancers.
  • When sticky sessions are enabled, a cookie is set on the initial request.
  • The check is defined by a path appended to the backend URL and an interval (given in a format understood by time.ParseDuration) specifying how often the health check should be executed (the default being 30 seconds). Each backend must respond to the health check within 5 seconds.
  • The static configuration is the global configuration which is setting up connections to configuration backends and entrypoints.
  • We only need to enable watch option to make Træfik watch configuration backend changes and generate its configuration automatically.
  • Separate the regular expression and the replacement by a space.
  • a comma-separated key/value pair where both key and value must be literals.
  • namespacing of your backends happens on the basis of hosts in addition to paths
  • Modifiers will be applied in a pre-determined order regardless of their order in the rule configuration section.
  • customize priority
  • Custom headers can be configured through the frontends, to add headers to either requests or responses that match the frontend's rules.
  • Security related headers (HSTS headers, SSL redirection, Browser XSS filter, etc) can be added and configured per frontend in a similar manner to the custom headers above.
  • Servers are simply defined using a url. You can also apply a custom weight to each server (this will be used by load-balancing).
  • Maximum connections can be configured by specifying an integer value for maxconn.amount and maxconn.extractorfunc which is a strategy used to determine how to categorize requests in order to evaluate the maximum connections.
crazylion lee

ProxySQL - 0 views

www.proxysql.com

mysql proxy

shared by crazylion lee on 10 Mar 18 - No Cached
  • crazylion lee on 10 Mar 18
     
    "ProxySQL High-performance MySQL proxy "
crazylion lee

grpc / grpc.io - 0 views

www.grpc.io

google grpc rpc

shared by crazylion lee on 27 Aug 16 - No Cached
  • crazylion lee on 27 Aug 16
     
    "A high performance, open-source universal RPC framework"
crazylion lee

Laptop USB Console Adapter - CV211, ATEN KVM Cables - 0 views

www.aten.com/...cv211

usb console kvm

shared by crazylion lee on 14 Mar 17 - No Cached
  • crazylion lee on 14 Mar 17
     
    " 'Turn your laptop into a mobile console in a few seconds.' The CV211 Laptop USB Console Adapter provides a direct Laptop-to-Computer connection for fast and easy remote desktop access with no software to install. High efficiency in a compact design, the CV211 offers bi-directional file transfers, hotkey macros, video recording and screenshots through a USB 2.0 and VGA composite cable."
crazylion lee

QMachine - 0 views

www.qmachine.org

bio

shared by crazylion lee on 14 May 17 - No Cached
  • crazylion lee on 14 May 17
     
    "Mix High Performance Computing with the World Wide Web and you'll get QMachine, a web service that can incorporate ordinary browsers into a World Wide Computer - without installing anything."
crazylion lee

Cello * High Level C - 0 views

libcello.org

c library high level

shared by crazylion lee on 16 Apr 17 - No Cached
  • crazylion lee on 16 Apr 17
     
    "Cello is a library that brings higher level programming to C. By acting as a modern, powerful runtime system Cello makes many things easy that were previously impractical or awkward in C such as: Generic Data Structures Polymorphic Functions Interfaces / Type Classes Constructors / Destructors Optional Garbage Collection Exceptions Reflection And because Cello works seamlessly alongside standard C you get all the other benefits such as great performance, powerful tooling, and extensive libraries."
張 旭

MongoDB Performance - MongoDB Manual - 0 views

docs.mongodb.com/...analyzing-mongodb-performance

database mongodb performance

shared by 張 旭 on 20 Apr 21 - No Cached
  • MongoDB uses a locking system to ensure data set consistency. If certain operations are long-running or a queue forms, performance will degrade as requests and operations wait for the lock.
  • performance limitations as a result of inadequate or inappropriate indexing strategies, or as a consequence of poor schema design patterns.
  • performance issues may be temporary and related to abnormal traffic load.
  • ...9 more annotations...
  • Lock-related slowdowns can be intermittent.
  • If globalLock.currentQueue.total is consistently high, then there is a chance that a large number of requests are waiting for a lock.
  • If globalLock.totalTime is high relative to uptime, the database has existed in a lock state for a significant amount of time.
  • For write-heavy applications, deploy sharding and add one or more shards to a sharded cluster to distribute load among mongod instances.
  • Unless constrained by system-wide limits, the maximum number of incoming connections supported by MongoDB is configured with the maxIncomingConnections setting.
  • When logLevel is set to 0, MongoDB records slow operations to the diagnostic log at a rate determined by slowOpSampleRate.
  • At higher logLevel settings, all operations appear in the diagnostic log regardless of their latency with the following exception
  • Full Time Diagnostic Data Collection (FTDC) mechanism. FTDC data files are compressed, are not human-readable, and inherit the same file access permissions as the MongoDB data files.
  • mongod processes store FTDC data files in a diagnostic.data directory under the instances storage.dbPath.
  • 張 旭 on 20 Apr 21
     
    "MongoDB uses a locking system to ensure data set consistency. If certain operations are long-running or a queue forms, performance will degrade as requests and operations wait for the lock."
1 - 20 of 39 Next ›
Showing 20▼ items per page