Skip to main content

Home/ Larvata/ Contents contributed and discussions participated by 張 旭

Contents contributed and discussions participated by 張 旭

Simple Middle Filter: All | Bookmarks | Topics
張 旭

Docker ARG, ENV and .env - a Complete Guide · vsupalov.com - 1 views

vsupalov.com/docker-arg-env-variable-guide

docker development system server environment

shared by 張 旭 on 10 Dec 18 - No Cached
  • understand and use Docker build-time variables, environment variables and docker-compose templating the right way.
  • ARG is only available during the build of a Docker image (RUN etc), not after the image is created and containers are started from it (ENTRYPOINT, CMD).
  • ENV values are available to containers, but also RUN-style commands during the Docker build starting with the line where they are introduced.
  • ...20 more annotations...
  • set an environment variable in an intermediate container using bash (RUN export VARI=5 && …) it will not persist in the next command.
  • An env_file, is a convenient way to pass many environment variables to a single command in one batch.
  • not be confused with a .env file
  • the dot in front of env - .env, not an “env_file”.
  • If you have a file named .env in your project, it’s only used to put values into the docker-compose.yml file which is in the same folder. Those are used with Docker Compose and Docker Stack.
  • Just type docker-compose config. This way you’ll see how the docker-compose.yml file content looks after the substitution step has been performed without running anything else.
  • ARG are also known as build-time variables. They are only available from the moment they are ‘announced’ in the Dockerfile with an ARG instruction up to the moment when the image is built.
  • Running containers can’t access values of ARG variables.
  • ENV variables are also available during the build, as soon as you introduce them with an ENV instruction. However, unlike ARG, they are also accessible by containers started from the final image.
  • ENV values can be overridden when starting a container,
  • If you don’t provide a value to expected ARG variables which don’t have a default, you’ll get an error message.
  • args block
  • You can use ARG to set the default values of ENV vars.
  • dynamic on-build env values
  • 2. Pass environment variable values from your host
  • 1. Provide values one by one
  • 3. Take values from a file (env_file)
  • for each RUN statement, a new container is launched from an intermediate image.
  • An image is saved by the end of the command, but environment variables do not persist that way.
  • The precedence is, from stronger to less-strong: stuff the containerized application sets, values from single environment entries, values from the env_file(s) and finally Dockerfile defaults.
張 旭

Sorting for Humans : Natural Sort Order - 0 views

blog.codinghorror.com/-for-humans-natural-sort-order

python programming sort file

shared by 張 旭 on 07 Dec 18 - No Cached
  • It's a completely nonsensical ordering to anyone who doesn't have the ASCII chart committed to memory (and by the way, uppercase A is decimal 65).
  • it's hard to find information on natural sorting
  • Implementing a natural sort is more complex than it seems
  • ...3 more annotations...
  • a basic natural sort shouldn't require the 40+ lines of code it takes in most languages.
  • ASCIIbetical does not equal alphabetical
  • a more human-friendly natural sort option should be built into mainstream programming languages
張 旭

| Docker Documentation - 0 views

docs.docker.com/...builder

docker volume

shared by 張 旭 on 06 Dec 18 - No Cached
  • The host directory is declared at container run-time: The host directory (the mountpoint) is, by its nature, host-dependent. This is to preserve image portability, since a given host directory can’t be guaranteed to be available on all hosts.
  • This Dockerfile results in an image that causes docker run to create a new mount point at /myvol and copy the greeting file into the newly created volume.
  • 張 旭 on 06 Dec 18
     
    "The host directory is declared at container run-time: The host directory (the mountpoint) is, by its nature, host-dependent. This is to preserve image portability, since a given host directory can't be guaranteed to be available on all hosts."
張 旭

Bash Reference Manual: Shell Parameter Expansion - 1 views

www.gnu.org/...Shell-Parameter-Expansion.html

shell script programming

shared by 張 旭 on 30 Nov 18 - No Cached
  • parameter expansion
  • command substitution
  • arithmetic expansion
  • ...16 more annotations...
  • The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name.
  • When braces are used, the matching ending brace is the first ‘}’ not escaped by a backslash or within a quoted string, and not within an embedded arithmetic expansion, command substitution, or parameter expansion.
  • ${parameter}
  • braces are required
  • If the first character of parameter is an exclamation point (!), and parameter is not a nameref, it introduces a level of variable indirection.
  • ${parameter:-word} If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.
  • ${parameter:=word} If parameter is unset or null, the expansion of word is assigned to parameter.
  • ${parameter:?word} If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits.
  • ${parameter:+word} If parameter is null or unset, nothing is substituted, otherwise the expansion of word is substituted.
  • ${parameter:offset} ${parameter:offset:length}
  • Substring expansion applied to an associative array produces undefined results.
  • ${parameter/pattern/string} The pattern is expanded to produce a pattern just as in filename expansion.
  • If pattern begins with ‘/’, all matches of pattern are replaced with string.
  • Normally only the first match is replaced
  • The ‘^’ operator converts lowercase letters matching pattern to uppercase
  • the ‘,’ operator converts matching uppercase letters to lowercase.
張 旭

Ruby on Rails 實戰聖經 | 網站效能 - 0 views

ihower.tw/performance.html

rails programming mysql performance

shared by 張 旭 on 28 Nov 18 - No Cached
  • 依照慣例是_count結尾，型別是integer，有預設值0。
  • lol_dba提供了Rake任務可以幫忙找忘記加的索引。
  • Bullet是一個外掛可以在開發時偵測N+1 queries問題。
  • ...19 more annotations...
  • 存取資料庫是一種相對很慢的I/O的操作：每一條SQL query都得耗上時間、執行回傳的結果也會被轉成ActiveRecord物件全部放進記憶體
  • 如果需要撈出全部的資料做處理，強烈建議最好不要用all方法，因為這樣會把全部的資料一次放進記憶體中，如果資料有成千上萬筆的話，效能就墜毀了。
  • .find_each( :batch_size => 100 )
  • .find_in_batches( :batch_size => 100 )
  • 在Transaction交易範圍內的SQL效能會加快，因為最後只需要COMMIT一次即可
  • Elasticsearch全文搜尋引擎和elasticsearch-rails gem
  • QueryReviewer這個套件透過SQL EXPLAIN分析SQL query的效率
  • 必要時可以採用逆正規化的設計。犧牲空間，增加修改的麻煩，但是讓讀取這事件變得更快更簡單。
  • 將成本轉嫁到寫入，而最佳化了讀取時間
  • 在效能還沒有造成問題前，就為了優化效能而修改程式和架構，只會讓程式更混亂不好維護
  • 當效能還不會造成問題時，程式的維護性比考慮效能重要
  • 會拖慢整體效能的程式，只佔全部程式的一小部分而已，所以我們只最佳化會造成問題的程式。
  • 善用分析工具找效能瓶頸，最佳化前需要測量，最佳化後也要測量比較。
  • rack-mini-profiler在頁面的左上角顯示花了多少時間，並且提供報表，推薦安裝
  • 如果是不需要權限控管的靜態檔案，可以直接放在public目錄下讓使用者下載。
  • Web伺服器得先安裝好x_sendfile功能
  • 如果要讓你的Assets例如CSS, JavaScript, Images也讓使用者透過CDN下載，只要修改config/environments/production.rb的config.action_controller.asset_host為CDN網址即可。
  • 有時候「執行速度較快」的程式碼不代表好維護、好除錯的程式碼
  • Ruby不是萬能，有時候直接呼叫外部程式是最快的作法
張 旭

Pre-Built CircleCI Docker Images - CircleCI - 0 views

circleci.com/...circleci-images

circle CI docker deploy

shared by 張 旭 on 26 Nov 18 - No Cached
  • typically extensions of official Docker images and include tools especially useful for CI/CD.
  • Convenience images are based on the most recently built versions of upstream images, so it is best practice to use the most specific image possible.
  • add -jessie or -stretch to the end of each of those containers to ensure you’re only using that version of the Debian base OS.
  • ...12 more annotations...
  • language images
  • service images
  • All images add a circleci user as a system user
  • A language image should be listed first under the docker key in your configuration, making it the primary container during execution.
  • For example, if you want to add browsers to the circleci/golang:1.9 image, use the circleci/golang:1.9-browsers image.
  • Service images are convenience images for services like databases
  • should be listed after language images so they become secondary service containers.
  • To speed up builds using RAM volume, add the -ram suffix to the end of a service image tag
  • All convenience images have been extended with additional tools.
  • all images include the following packages, installed via apt-get
  • Most CircleCI convenience images are Debian Jessie- or Stretch-based images, however some extend Ubuntu-based images.
  • The following packages are installed via curl
張 旭

Reusing Config - CircleCI - 0 views

circleci.com/...reusing-config

circle CI config deploy devops

shared by 張 旭 on 26 Nov 18 - No Cached
  • Change the version key to 2.1 in your .circleci/config.yml file and commit the changes to test your build.
  • Reusable commands are invoked with specific parameters as steps inside a job.
  • Commands can use other commands in the scope of execution
  • ...19 more annotations...
  • Executors define the environment in which the steps of a job will be run.
  • Executor declarations in config outside of jobs can be used by all jobs in the scope of that declaration, allowing you to reuse a single executor definition across multiple jobs.
  • It is also possible to allow an orb to define the executor used by all of its commands.
  • When invoking an executor in a job any keys in the job itself will override those of the executor invoked.
  • Steps are used when you have a job or command that needs to mix predefined and user-defined steps.
  • Use the enum parameter type when you want to enforce that the value must be one from a specific set of string values.
  • Use an executor parameter type to allow the invoker of a job to decide what executor it will run on
  • invoke the same job more than once in the workflows stanza of config.yml, passing any necessary parameters as subkeys to the job.
  • If a job is declared inside an orb it can use commands in that orb or the global commands.
  • To use parameters in executors, define the parameters under the given executor.
  • Parameters are in-scope only within the job or command that defined them.
  • A single configuration may invoke a job multiple times.
  • Every job invocation may optionally accept two special arguments: pre-steps and post-steps.
  • Pre and post steps allow you to execute steps in a given job without modifying the job.
  • conditions are checked before a workflow is actually run
  • you cannot use a condition to check an environment variable.
  • Conditional steps may be located anywhere a regular step could and may only use parameter values as inputs.
  • A conditional step consists of a step with the key when or unless. Under this conditional key are the subkeys steps and condition
  • A condition is a single value that evaluates to true or false at the time the config is processed, so you cannot use environment variables as conditions
張 旭

Enabling Build Processing - CircleCI - 0 views

circleci.com/...build-processing

circleci deploy CI

shared by 張 旭 on 25 Nov 18 - No Cached
  • If you use << in your shell commands (most commonly found in use of heredocs) you will need to escape them using backslash \ as in \<< in order to use version 2.1 or higher of configuration.
  • Jobs named build will be wrapped in a workflows stanza by the processor
  • Use of configuration version 2.1 or higher requires build processing to be on
  • ...1 more annotation...
  • New API endpoint to trigger builds, including running all workflows in the build
  • 張 旭 on 25 Nov 18
     
    "If you use << in your shell commands (most commonly found in use of heredocs) you will need to escape them using backslash \ as in \<< in order to use version 2.1 or higher of configuration."
張 旭

Persisting Data in Workflows: When to Use Caching, Artifacts, and Workspaces - CircleCI - 0 views

circleci.com/...ching-artifacts-and-workspaces

circleci deploy system CI volume share

shared by 張 旭 on 25 Nov 18 - No Cached
  • Repeatability is also important
  • When a CI process isn’t repeatable you’ll find yourself wasting time re-running jobs to get them to go green.
  • Workspaces persist data between jobs in a single Workflow.
  • ...9 more annotations...
  • Caching persists data between the same job in different Workflow builds.
  • Artifacts persist data after a Workflow has finished
  • When a Workspace is declared in a job, one or more files or directories can be added. Each addition creates a new layer in the Workspace filesystem. Downstreams jobs can then use this Workspace for its own needs or add more layers on top.
  • Unlike caching, Workspaces are not shared between runs as they no longer exists once a Workflow is complete.
  • Caching lets you reuse the data from expensive fetch operations from previous jobs.
  • A prime example is package dependency managers such as Yarn, Bundler, or Pip.
  • Caches are global within a project, a cache saved on one branch will be used by others so they should only be used for data that is OK to share across Branches
  • Artifacts are used for longer-term storage of the outputs of your build process.
  • If your project needs to be packaged in some form or fashion, say an Android app where the .apk file is uploaded to Google Play, that’s a great example of an artifact.
  • 張 旭 on 25 Nov 18
     
    "CircleCI 2.0 provides a number of different ways to move data into and out of jobs, persist data, and with the introduction of Workspaces, move data between jobs"
張 旭

Using Workflows to Schedule Jobs - CircleCI - 1 views

circleci.com/workflows

circleci CI deploy system

shared by 張 旭 on 25 Nov 18 - No Cached
  • A workflow is a set of rules for defining a collection of jobs and their run order.
  • Schedule workflows for jobs that should only run periodically.
  • run multiple jobs in parallel
  • ...37 more annotations...
  • rerun just the failed job
  • Builds without workflows require a build job.
  • Refer the YAML Anchors/Aliases documentation for information about how to alias and reuse syntax to keep your .circleci/config.yml file small.
  • workflow orchestration with two parallel jobs
  • jobs run according to configured requirements, each job waiting to start until the required job finishes successfully
  • requires: key
  • fans-out to run a set of acceptance test jobs in parallel, and finally fans-in to run a common deploy job.
  • Holding a Workflow for a Manual Approval
  • Workflows can be configured to wait for manual approval of a job before continuing to the next job
  • add a job to the jobs list with the key type: approval
  • approval is a special job type that is only available to jobs under the workflow key
  • The name of the job to hold is arbitrary - it could be wait or pause, for example, as long as the job has a type: approval key in it.
  • schedule a workflow to run at a certain time for specific branches.
  • The triggers key is only added under your workflows key
  • using cron syntax to represent Coordinated Universal Time (UTC) for specified branches.
  • By default, a workflow is triggered on every git push
  • the commit workflow has no triggers key and will run on every git push
  • The nightly workflow has a triggers key and will run on the specified schedule
  • Cron step syntax (for example, */1, */20) is not supported.
  • use a context to share environment variables
  • use the same shared environment variables when initiated by a user who is part of the organization.
  • CircleCI does not run workflows for tags unless you explicitly specify tag filters.
  • CircleCI branch and tag filters support the Java variant of regex pattern matching.
  • Each workflow has an associated workspace which can be used to transfer files to downstream jobs as the workflow progresses.
  • The workspace is an additive-only store of data.
  • Jobs can persist data to the workspace
  • Downstream jobs can attach the workspace to their container filesystem.
  • Attaching the workspace downloads and unpacks each layer based on the ordering of the upstream jobs in the workflow graph.
  • Workflows that include jobs running on multiple branches may require data to be shared using workspaces
  • To persist data from a job and make it available to other jobs, configure the job to use the persist_to_workspace key.
  • Files and directories named in the paths: property of persist_to_workspace will be uploaded to the workflow’s temporary workspace relative to the directory specified with the root key.
  • Configure a job to get saved data by configuring the attach_workspace key.
  • persist_to_workspace
  • attach_workspace
  • To rerun only a workflow’s failed jobs, click the Workflows icon in the app and select a workflow to see the status of each job, then click the Rerun button and select Rerun from failed.
  • if you do not see your workflows triggering, a configuration error is preventing the workflow from starting.
  • check your Workflows page of the CircleCI app (not the Job page)
  • 張 旭 on 25 Nov 18
     
    "A workflow is a set of rules for defining a collection of jobs and their run order."
張 旭

Glossary - CircleCI - 0 views

circleci.com/glossary

circleci CI system deploy

shared by 張 旭 on 25 Nov 18 - No Cached
  • User authentication may use LDAP for an instance of the CircleCI application that is installed on your private server or cloud
  • The first user to log into a private installation of CircleCI
  • Contexts provide a mechanism for securing and sharing environment variables across projects.
  • ...22 more annotations...
  • The environment variables are defined as name/value pairs and are injected at runtime.
  • The CircleCI Docker Layer Caching feature allows builds to reuse Docker image layers
  • from previous builds.
  • Image layers are stored in separate volumes in the cloud and are not shared between projects.
  • Layers may only be used by builds from the same project.
  • Environment variables store customer data that is used by a project.
  • Defines the underlying technology to run a job.
  • machine to run your job inside a full virtual machine.
  • docker to run your job inside a Docker container with a specified image
  • A job is a collection of steps.
  • The first image listed in config.yml
  • A CircleCI project shares the name of the code repository for which it automates workflows, tests, and deployment.
  • must be added with the Add Project button
  • Following a project enables a user to subscribe to email notifications for the project build status and adds the project to their CircleCI dashboard.
  • A step is a collection of executable commands
  • Users must be added to a GitHub or Bitbucket org to view or follow associated CircleCI projects.
  • Users may not view project data that is stored in environment variables. &nbsp;
  • A Workflow is a set of rules for defining a collection of jobs and their run order.
  • Workflows are implemented as a directed acyclic graph (DAG) of jobs for greatest flexibility.
  • referred to as Pipelines
  • A workspace is a workflows-aware storage mechanism.
  • A workspace stores data unique to the job, which may be needed in downstream jobs.
張 旭

Intro to deployment strategies: blue-green, canary, and more - DEV Community - 0 views

dev.to/...blue-green-canary-and-more-3a3

devops deploy system micro service

shared by 張 旭 on 23 Nov 18 - No Cached
  • using a service-oriented architecture and microservices approach, developers can design a code base to be modular.
  • Modern applications are often distributed and cloud-based
  • different release cycles for different components
  • ...20 more annotations...
  • the abstraction of the infrastructure layer, which is now considered code. Deployment of a new application may require the deployment of new infrastructure code as well.
  • "big bang" deployments update whole or large parts of an application in one fell swoop.
  • Big bang deployments required the business to conduct extensive development and testing before release, often associated with the "waterfall model" of large sequential releases.
  • Rollbacks are often costly, time-consuming, or even impossible.
  • In a rolling deployment, an application’s new version gradually replaces the old one.
  • new and old versions will coexist without affecting functionality or user experience.
  • Each container is modified to download the latest image from the app vendor’s site.
  • two identical production environments work in parallel.
  • Once the testing results are successful, application traffic is routed from blue to green.
  • In a blue-green deployment, both systems use the same persistence layer or database back end.
  • You can use the primary database by blue for write operations and use the secondary by green for read operations.
  • Blue-green deployments rely on traffic routing.
  • long TTL values can delay these changes.
  • The main challenge of canary deployment is to devise a way to route some users to the new application.
  • Using an application logic to unlock new features to specific users and groups.
  • With CD, the CI-built code artifact is packaged and always ready to be deployed in one or more environments.
  • Use Build Automation tools to automate environment builds
  • Use configuration management tools
  • Enable automated rollbacks for deployments
  • An application performance monitoring (APM) tool can help your team monitor critical performance metrics including server response times after deployments.
張 旭

A few ways to execute commands remotely using SSH · zaiste.net - 0 views

zaiste.net/te_commands_remotely_using_ssh

ssh command system

shared by 張 旭 on 20 Nov 18 - No Cached
  • Using heredoc is probably the most convenient way to execute multi-line commands on a remote machine
張 旭

Home · sysown/proxysql Wiki - 0 views

github.com/wiki

mysql database cluster mariadb proxy

shared by 張 旭 on 19 Nov 18 - No Cached
  • bear in mind that the best way to configure ProxySQL is through its admin interface.
  • llow you to control the list of the backend servers, how traffic is routed to them, and other important settings (such as caching, access control, etc)
  • Once you've made modifications to the in-memory data structure, you must load the new configuration to the runtime, or persist the new settings to disk
  • ...4 more annotations...
  • mysql_variables: contains global variables that control the functionality for handling the incoming MySQL traffic.
  • mysql_users: contains rows for the mysql_users table from the admin interface. Basically, these define the users which can connect to the proxy, and the users with which the proxy can connect to the backend servers.
  • mysql_servers: contains rows for the mysql_servers table from the admin interface. Basically, these define the backend servers towards which the incoming MySQL traffic is routed.
  • mysql_query_rules: contains rows for the mysql_query_rules table from the admin interface. Basically, these define the rules used to classify and route the incoming MySQL traffic, according to various criteria (patterns matched, user used to run the query, etc.).
張 旭

ProxySQL Series : Percona Cluster/MariaDB Cluster (Galera) Read-write Split - Mydbops - 0 views

mydbops.wordpress.com/...luster-galera-read-write-split

mysql database cluster mariadb proxy

shared by 張 旭 on 18 Nov 18 - No Cached
  • PXC / MariaDB Clusters really works better with writes on single ode than multi node writes.
  • proxySQL setup for a cluster in Single-writer mode, Which is the most recommended for Cluster to avoid of conflicts of writes and split-Brain scenarios.
  • listening on ports 6032 for proxysql admin interface and 6033 for MySQL interface by default
  • 張 旭 on 18 Nov 18
     
    "PXC / MariaDB Clusters really works better with writes on single ode than multi node writes. "
張 旭

Setup ProxySQL for High Availability (not a Single Point of Failure) - Percona Database... - 0 views

www.percona.com/...ility-not-single-point-failure

mysql database cluster mariadb proxy

shared by 張 旭 on 18 Nov 18 - No Cached
  • ProxySQL doesn’t natively support any high availability solution
  • most common solution is setting up ProxySQL as part of a tile architecture, where Application/ProxySQL are deployed together.
    • 張 旭
      張 旭 on 18 Nov 18
       
      直接把 ProxySQL 跟 App 捆綁發佈
  • If we have 400 instances of ProxySQL, we end up keeping our databases busy just performing the checks.
  • ...5 more annotations...
  • Another possible approach is to have two layers of ProxySQL, one close to the application and another in the middle to connect to the database.
  • creates additional complexity in the management of the platform, and it adds network hops.
  • combining existing solutions and existing blocks:&nbsp;KeepAlived + ProxySQl + MySQL.
  • Keepalived implements a set of checkers to dynamically and adaptively maintain and manage load-balanced server pool according to their health.
  • Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions.
張 旭

Running Docker Commands - CircleCI - 0 views

circleci.com/...building-docker-images

circle CI

shared by 張 旭 on 14 Nov 18 - No Cached
  • To build Docker images for deployment, you must use a special setup_remote_docker key which creates a separate environment for each build for security.
  • When setup_remote_docker executes, a remote environment will be created, and your current primary container will be configured to use it.
  • Once setup_remote_docker is called, a new remote environment is created, and your primary container is configured to use it.
  • ...8 more annotations...
  • but building/pushing images and running containers happens in the remote Docker Engine
  • use a primary image that already has Docker (recommended)
  • installs Docker and has Git, use 17.05.0-ce-git
  • The job and remote docker run in separate environments.
  • It is not possible to start a service in remote docker and ping it directly from a primary container or to start a primary container that can ping a service in remote docker.
  • It is not possible to mount a folder from your job space into a container in Remote Docker (and vice versa).
    • 張 旭
      張 旭 on 15 Nov 18
       
      等於是 docker client 跟 docker server 是兩台不同的主機就對了。
  • use https://github.com/outstand/docker-dockup or a similar image for backup and restore to spin up a container
  • 張 旭 on 14 Nov 18
     
    "To build Docker images for deployment, you must use a special setup_remote_docker key which creates a separate environment for each build for security. "
« First ‹ Previous 301 - 320 of 596 Next › Last »
Showing 20▼ items per page