Skip to main content

Home/ Larvata/ Group items tagged container

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
張 旭

GitLab Auto DevOps 深入淺出，自動部署，連設定檔不用？！ | 五倍紅寶石・專業程式教育 - 0 views

5xruby.tw/...gitlab-auto-devops

devops gitlab auto

shared by 張 旭 on 15 Apr 21 - No Cached
  • 一個 K8S 的 Cluster，Auto DevOps 將會把網站部署到這個 Cluster
  • 需要有一個 wildcard 的 DNS 讓部署在這個環境的網站有 Domain name
  • 一個可以跑 Docker 的 GitLab Runner，將會為由它來執行 CI / CD 的流程。
  • ...37 more annotations...
  • 其實 Auto DevOps 就是一份官方寫好的 gitlab-ci.yml，在啟動 Auto DevOps 的專案裡，如果找不到 gitlab-ci.yml 檔，那就會直接用官方 gitlab-ci.yml 去跑 CI / CD 流程。
  • Pod 是 K8S 中可以被部署的最小元件，一個 Pod 是由一到多個 Container 組成，同個 Pod 的不同 Container 之間彼此共享網路資源。
  • 每個 Pod 都會有它的 yaml 檔，用以描述 Pod 會使用的 Image 還有連接的 Port 等資訊。
  • Node 又分成 Worker Node 和 Master Node 兩種
  • Helm 透過參數 (parameter) 跟模板 (template) 的方式，讓我們可以在只修改參數的方式重複利用模板。
  • 為了要有 CI CD 的功能我們會把 .gitlab-ci.yml 放在專案的根目錄裡， GitLab 會依造 .gitlab-ci.yml 的設定產生 CI/CD Pipeline，每個 Pipeline 裡面可能有多個 Job，這時候就會需要有 GitLab Runner 來執行這些 Job 並把執行的結果回傳給 GitLab 讓它知道這個 Job 是否有正常執行。
  • 把專案打包成 Docker Image 這工作又或是 helm 的操作都會在 Container 內執行
  • CI/CD Pipeline 是由 stage 還有 job 組成的，stage 是有順序性的，前面的 stage 完成後才會開始下一個 stage。
  • 每個 stage 裡面包含一到多個 Job
  • Auto Devops 裡也會大量用到這種在指定 Container 內運行的工作。
  • 可以通過 health checks
  • 開 private 的話還要注意使用 Container Registry 的權限問題
  • 申請好的 wildcard 的 DNS
  • Auto Devops 也提供只要設定環境變數就能一定程度客製化的選項
  • 特別注意 namespace 有沒有設定對，不然會找不到資料喔
  • Auto Devops，如果想要進一步的客製化，而且是改 GitLab 環境變數都無法實現的客製化，這時候還是得回到 .gitlab-ci.yml 設定檔
  • 在 Docker in Docker 的環境用 Dockerfile 打包 Image
  • 用 helm upgrade 把 chart 部署到 K8S 上
  • GitLab CI 的環境變數主要有三個來源，優先度高到低依序為Settings > CI/CD 介面定義的變數gitlab_ci.yml 定義環境變數GitLab 預設環境變數
  • 把專案打包成 Docker Image 首先需要在專案下新增一份 Dockerfile
  • Auto Devops 裡面的做法，用 herokuish 提供的 Image 來打包專案
  • 在 Runner 的環境中是沒有 docker 指令可以用的，所以這邊啟動一個 Docker Container 在裡面執行就可以用 docker 指令了。
  • 其中 $CI_COMMIT_SHA $CI_COMMIT_BEFORE_SHA 這兩個都是 GitLab 預設環境變數，代表這次 commit 還有上次 commit 的 SHA 值。
  • dind 則是直接啟動 docker daemon，此外 dind 還會自動產生 TLS certificates
  • 為了在 Docker Container 內運行 Docker，會把 Host 上面的 Docker API 分享給 Container。
  • docker:stable 有執行 docker 需要的執行檔，他裡面也包含要啟動 docker 的程式(docker daemon)，但啟動 Container 的 entrypoint 是 sh
  • docker:dind 繼承自 docker:stable，而且它 entrypoint 就是啟動 docker 的腳本，此外還會做完 TLS certificates
  • Container 要去連 Host 上的 Docker API 。但現在連線失敗卻是找 http://docker:2375，現在的 dind 已經不是被當做 services 來用了，而是要直接在裡面跑 Docker，所以他應該是要 unix:///var/run/docker.sock 用這種連線，於是把環境變數 DOCKER_HOST 從 tcp://docker:2375 改成空字串，讓 docker daemon 走預設連線就能成功囉！
  • auto-deploy preparationhelm init 建立 helm 專案設定 tiller 在背景執行設定 cluster 的 namespace
  • auto-deploy deploy使用 helm upgrade 部署 chart 到 K8S 上透過 --set 來設定要注入 template 的參數
  • set -x，這樣就能在執行前，顯示指令內容。
  • 用 helm repo list 看看現在有註冊哪些 Chart Repository
  • helm fetch gitlab/auto-deploy-app --untar
  • nohup 可以讓你在離線或登出系統後，還能夠讓工作繼續進行
  • 在不特別設定 CI_APPLICATION_REPOSITORY 的情況下，image_repository 的值就是預設環境變數 CI_REGISTRY_IMAGE/CI_COMMIT_REF_SLUG
  • A:-B 的意思是如果有 A 就用它，沒有就用 B
  • 研究 Auto Devops 難度最高的地方就是太多工具整合在一起，搞不清楚他們之間的關係，出錯也不知道從何查起
張 旭

Run your CI/CD jobs in Docker containers | GitLab - 0 views

docs.gitlab.com/...using_docker_images.html

docker cicd gitlab

shared by 張 旭 on 20 Jan 22 - No Cached
  • If you run Docker on your local machine, you can run tests in the container, rather than testing on a dedicated CI/CD server.
  • Run other services, like MySQL, in containers. Do this by specifying services in your .gitlab-ci.yml file.
  • By default, the executor pulls images from Docker Hub
  • ...10 more annotations...
  • Maps must contain at least the name option, which is the same image name as used for the string setting.
  • When a CI job runs in a Docker container, the before_script, script, and after_script commands run in the /builds/<project-path>/ directory. Your image may have a different default WORKDIR defined. To move to your WORKDIR, save the WORKDIR as an environment variable so you can reference it in the container during the job’s runtime.
  • The runner starts a Docker container using the defined entrypoint. The default from Dockerfile that may be overridden in the .gitlab-ci.yml file.
  • attaches itself to a running container.
  • sends the script to the container’s shell stdin and receives the output.
  • To override the entrypoint of a Docker image, define an empty entrypoint in the .gitlab-ci.yml file, so the runner does not start a useless shell layer. However, that does not work for all Docker versions. For Docker 17.06 and later, the entrypoint can be set to an empty value. For Docker 17.03 and earlier, the entrypoint can be set to /bin/sh -c, /bin/bash -c, or an equivalent shell available in the image.
  • The runner expects that the image has no entrypoint or that the entrypoint is prepared to start a shell command.
  • entrypoint: [""]
  • entrypoint: ["/bin/sh", "-c"]
  • A DOCKER_AUTH_CONFIG CI/CD variable
  • 張 旭 on 20 Jan 22
     
    "If you run Docker on your local machine, you can run tests in the container, rather than testing on a dedicated CI/CD server. "
張 旭

Choosing an Executor Type - CircleCI - 0 views

circleci.com/...executor-types

circle CI

shared by 張 旭 on 14 Nov 18 - No Cached
  • Containers are an instance of the Docker Image you specify and the first image listed in your configuration is the primary container image in which all steps run.
  • In this example, all steps run in the container created by the first image listed under the build job
  • If you experience increases in your run times due to installing additional tools during execution, it is best practice to use the Building Custom Docker Images Documentation to create a custom image with tools that are pre-loaded in the container to meet the job requirements.
  • ...9 more annotations...
  • The machine option runs your jobs in a dedicated, ephemeral VM
  • Using the machine executor gives your application full access to OS resources and provides you with full control over the job environment.
  • Using machine may require additional fees in a future pricing update.
  • Using the macos executor allows you to run your job in a macOS environment on a VM.
  • In a multi-image configuration job, all steps are executed in the container created by the first image listed.
  • All containers run in a common network and every exposed port will be available on localhost from a primary container.
  • If you want to work with private images/registries, please refer to Using Private Images.
  • Docker also has built-in image caching and enables you to build, run, and publish Docker images via Remote Docker.
  • if you require low-level access to the network or need to mount external volumes consider using machine
張 旭

Running Docker Commands - CircleCI - 0 views

circleci.com/...building-docker-images

circle CI

shared by 張 旭 on 14 Nov 18 - No Cached
  • To build Docker images for deployment, you must use a special setup_remote_docker key which creates a separate environment for each build for security.
  • When setup_remote_docker executes, a remote environment will be created, and your current primary container will be configured to use it.
  • Once setup_remote_docker is called, a new remote environment is created, and your primary container is configured to use it.
  • ...8 more annotations...
  • but building/pushing images and running containers happens in the remote Docker Engine
  • use a primary image that already has Docker (recommended)
  • installs Docker and has Git, use 17.05.0-ce-git
  • The job and remote docker run in separate environments.
  • It is not possible to start a service in remote docker and ping it directly from a primary container or to start a primary container that can ping a service in remote docker.
  • It is not possible to mount a folder from your job space into a container in Remote Docker (and vice versa).
    • 張 旭
      張 旭 on 15 Nov 18
       
      等於是 docker client 跟 docker server 是兩台不同的主機就對了。
  • use https://github.com/outstand/docker-dockup or a similar image for backup and restore to spin up a container
  • 張 旭 on 14 Nov 18
     
    "To build Docker images for deployment, you must use a special setup_remote_docker key which creates a separate environment for each build for security. "
張 旭

Boosting your kubectl productivity ♦︎ Learnk8s - 0 views

learnk8s.io/...kubectl-productivity

devops kubernetes kubectl

shared by 張 旭 on 09 Apr 19 - No Cached
  • kubectl is your cockpit to control Kubernetes.
  • kubectl is a client for the Kubernetes API
  • Kubernetes API is an HTTP REST API.
  • ...75 more annotations...
  • This API is the real Kubernetes user interface.
  • Kubernetes is fully controlled through this API
  • every Kubernetes operation is exposed as an API endpoint and can be executed by an HTTP request to this endpoint.
  • the main job of kubectl is to carry out HTTP requests to the Kubernetes API
  • Kubernetes maintains an internal state of resources, and all Kubernetes operations are CRUD operations on these resources.
  • Kubernetes is a fully resource-centred system
  • Kubernetes API reference is organised as a list of resource types with their associated operations.
  • This is how kubectl works for all commands that interact with the Kubernetes cluster.
  • kubectl simply makes HTTP requests to the appropriate Kubernetes API endpoints.
  • it's totally possible to control Kubernetes with a tool like curl by manually issuing HTTP requests to the Kubernetes API.
  • Kubernetes consists of a set of independent components that run as separate processes on the nodes of a cluster.
  • components on the master nodes
  • Storage backend: stores resource definitions (usually etcd is used)
  • API server: provides Kubernetes API and manages storage backend
  • Controller manager: ensures resource statuses match specifications
  • Scheduler: schedules Pods to worker nodes
  • component on the worker nodes
  • Kubelet: manages execution of containers on a worker node
  • triggers the ReplicaSet controller, which is a sub-process of the controller manager.
  • the scheduler, who watches for Pod definitions that are not yet scheduled to a worker node.
  • creating and updating resources in the storage backend on the master node.
  • The kubelet of the worker node your ReplicaSet Pods have been scheduled to instructs the configured container runtime (which may be Docker) to download the required container images and run the containers.
  • Kubernetes components (except the API server and the storage backend) work by watching for resource changes in the storage backend and manipulating resources in the storage backend.
  • However, these components do not access the storage backend directly, but only through the Kubernetes API.
    • 張 旭
      張 旭 on 09 Apr 19
       
      很精彩，相互之間都是使用 API call 溝通，良好的微服務行為。
  • double usage of the Kubernetes API for internal components as well as for external users is a fundamental design concept of Kubernetes.
  • All other Kubernetes components and users read, watch, and manipulate the state (i.e. resources) of Kubernetes through the Kubernetes API
  • The storage backend stores the state (i.e. resources) of Kubernetes.
  • command completion is a shell feature that works by the means of a completion script.
  • A completion script is a shell script that defines the completion behaviour for a specific command. Sourcing a completion script enables completion for the corresponding command.
  • kubectl completion zsh
  • /etc/bash_completion.d directory (create it, if it doesn't exist)
  • source <(kubectl completion bash)
  • source <(kubectl completion zsh)
  • autoload -Uz compinit compinit
  • the API reference, which contains the full specifications of all resources.
  • kubectl api-resources
  • displays the resource names in their plural form (e.g. deployments instead of deployment). It also displays the shortname (e.g. deploy) for those resources that have one. Don't worry about these differences. All of these name variants are equivalent for kubectl.
  • .spec
  • custom columns output format comes in. It lets you freely define the columns and the data to display in them. You can choose any field of a resource to be displayed as a separate column in the output
  • kubectl get pods -o custom-columns='NAME:metadata.name,NODE:spec.nodeName'
  • kubectl explain pod.spec.
  • kubectl explain pod.metadata.
  • browse the resource specifications and try it out with any fields you like!
  • JSONPath is a language to extract data from JSON documents (it is similar to XPath for XML).
  • with kubectl explain, only a subset of the JSONPath capabilities is supported
  • Many fields of Kubernetes resources are lists, and this operator allows you to select items of these lists. It is often used with a wildcard as [*] to select all items of the list.
  • kubectl get pods -o custom-columns='NAME:metadata.name,IMAGES:spec.containers[*].image'
  • a Pod may contain more than one container.
  • The availability zones for each node are obtained through the special failure-domain.beta.kubernetes.io/zone label.
  • kubectl get nodes -o yaml kubectl get nodes -o json
  • The default kubeconfig file is ~/.kube/config
  • with multiple clusters, then you have connection parameters for multiple clusters configured in your kubeconfig file.
  • Within a cluster, you can set up multiple namespaces (a namespace is kind of "virtual" clusters within a physical cluster)
  • overwrite the default kubeconfig file with the --kubeconfig option for every kubectl command.
  • Namespace: the namespace to use when connecting to the cluster
  • a one-to-one mapping between clusters and contexts.
  • When kubectl reads a kubeconfig file, it always uses the information from the current context.
  • just change the current context in the kubeconfig file
  • to switch to another namespace in the same cluster, you can change the value of the namespace element of the current context
  • kubectl also provides the --cluster, --user, --namespace, and --context options that allow you to overwrite individual elements and the current context itself, regardless of what is set in the kubeconfig file.
  • for switching between clusters and namespaces is kubectx.
  • kubectl config get-contexts
  • just have to download the shell scripts named kubectl-ctx and kubectl-ns to any directory in your PATH and make them executable (for example, with chmod +x)
  • kubectl proxy
  • kubectl get roles
  • kubectl get pod
  • Kubectl plugins are distributed as simple executable files with a name of the form kubectl-x. The prefix kubectl- is mandatory,
  • To install a plugin, you just have to copy the kubectl-x file to any directory in your PATH and make it executable (for example, with chmod +x)
  • krew itself is a kubectl plugin
  • check out the kubectl-plugins GitHub topic
  • The executable can be of any type, a Bash script, a compiled Go program, a Python script, it really doesn't matter. The only requirement is that it can be directly executed by the operating system.
  • kubectl plugins can be written in any programming or scripting language.
  • you can write more sophisticated plugins with real programming languages, for example, using a Kubernetes client library. If you use Go, you can also use the cli-runtime library, which exists specifically for writing kubectl plugins.
  • a kubeconfig file consists of a set of contexts
  • changing the current context means changing the cluster, if you have only a single context per cluster.
張 旭

Auto DevOps | GitLab - 0 views

docs.gitlab.com/...index.html

gitlab auto devops CI deploy

shared by 張 旭 on 02 Aug 19 - No Cached
  • Auto DevOps provides pre-defined CI/CD configuration which allows you to automatically detect, build, test, deploy, and monitor your applications
  • Just push your code and GitLab takes care of everything else.
  • Auto DevOps will be automatically disabled on the first pipeline failure.
  • ...78 more annotations...
  • Your project will continue to use an alternative CI/CD configuration file if one is found
  • Auto DevOps works with any Kubernetes cluster;
  • using the Docker or Kubernetes executor, with privileged mode enabled.
  • Base domain (needed for Auto Review Apps and Auto Deploy)
  • Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)
  • Prometheus (needed for Auto Monitoring)
  • scrape your Kubernetes cluster.
  • project level as a variable: KUBE_INGRESS_BASE_DOMAIN
  • A wildcard DNS A record matching the base domain(s) is required
  • Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).
  • review/ (every environment starting with review/)
  • staging
  • production
  • need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.
  • Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.
  • Continuous deployment to production using timed incremental rollout
  • Automatic deployment to staging, manual deployment to production
  • Auto Build creates a build of the application using an existing Dockerfile or Heroku buildpacks.
  • If a project’s repository contains a Dockerfile, Auto Build will use docker build to create a Docker image.
  • Each buildpack requires certain files to be in your project’s repository for Auto Build to successfully build your application.
  • Auto Test automatically runs the appropriate tests for your application using Herokuish and Heroku buildpacks by analyzing your project to detect the language and framework.
  • Auto Code Quality uses the Code Quality image to run static analysis and other code checks on the current code.
  • Static Application Security Testing (SAST) uses the SAST Docker image to run static analysis on the current code and checks for potential security issues.
  • Dependency Scanning uses the Dependency Scanning Docker image to run analysis on the project dependencies and checks for potential security issues.
  • License Management uses the License Management Docker image to search the project dependencies for their license.
  • Vulnerability Static Analysis for containers uses Clair to run static analysis on a Docker image and checks for potential security issues.
  • Review Apps are temporary application environments based on the branch’s code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur.
  • The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain.
  • Review apps are deployed using the auto-deploy-app chart with Helm, which can be customized.
  • Your apps should not be manipulated outside of Helm (using Kubernetes directly).
  • Dynamic Application Security Testing (DAST) uses the popular open source tool OWASP ZAProxy to perform an analysis on the current code and checks for potential security issues.
  • Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.
  • add the paths to a file named .gitlab-urls.txt in the root directory, one per line.
  • After a branch or merge request is merged into the project’s default branch (usually master), Auto Deploy deploys the application to a production environment in the Kubernetes cluster, with a namespace based on the project name and unique project ID
  • Auto Deploy doesn’t include deployments to staging or canary by default, but the Auto DevOps template contains job definitions for these tasks if you want to enable them.
  • Apps are deployed using the auto-deploy-app chart with Helm.
  • For internal and private projects a GitLab Deploy Token will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.
  • If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.
  • If present, DB_INITIALIZE will be run as a shell command within an application pod as a helm post-install hook.
  • a post-install hook means that if any deploy succeeds, DB_INITIALIZE will not be processed thereafter.
  • DB_MIGRATE will be run as a shell command within an application pod as a helm pre-upgrade hook.
    • 張 旭
      張 旭 on 02 Aug 19
       
      如果專案類型不同，就要去查 buildpacks 裡面如何叫用該指令，例如 laravel 的 migration
    • 張 旭
      張 旭 on 02 Aug 19
       
      如果是自己的 Dockerfile 建立起來的，看來就不用鳥 buildpacks 的作法
  • Once your application is deployed, Auto Monitoring makes it possible to monitor your application’s server and response metrics right out of the box.
  • annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254"
  • If you are also using Auto Review Apps and Auto Deploy and choose to provide your own Dockerfile, make sure you expose your application to port 5000 as this is the port assumed by the default Helm chart.
  • While Auto DevOps provides great defaults to get you started, you can customize almost everything to fit your needs; from custom buildpacks, to Dockerfiles, Helm charts, or even copying the complete CI/CD configuration into your project to enable staging and canary deployments, and more.
  • If your project has a Dockerfile in the root of the project repo, Auto DevOps will build a Docker image based on the Dockerfile rather than using buildpacks.
  • Auto DevOps uses Helm to deploy your application to Kubernetes.
  • Bundled chart - If your project has a ./chart directory with a Chart.yaml file in it, Auto DevOps will detect the chart and use it instead of the default one.
  • Create a project variable AUTO_DEVOPS_CHART with the URL of a custom chart to use or create two project variables AUTO_DEVOPS_CHART_REPOSITORY with the URL of a custom chart repository and AUTO_DEVOPS_CHART with the path to the chart.
  • make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.
  • specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment.
    • 張 旭
      張 旭 on 02 Aug 19
       
      Auto DevOps 就是一套人家寫好好的傳便便的 .gitlab-ci.yml
  • Your additions will be merged with the Auto DevOps template using the behaviour described for include
  • copy and paste the contents of the Auto DevOps template into your project and edit this as needed.
  • In order to support applications that require a database, PostgreSQL is provisioned by default.
  • Set up the replica variables using a project variable and scale your application by just redeploying it!
  • You should not scale your application using Kubernetes directly.
  • Some applications need to define secret variables that are accessible by the deployed application.
  • Auto DevOps detects variables where the key starts with K8S_SECRET_ and make these prefixed variables available to the deployed application, as environment variables.
  • Auto DevOps pipelines will take your application secret variables to populate a Kubernetes secret.
  • Environment variables are generally considered immutable in a Kubernetes pod.
  • if you update an application secret without changing any code then manually create a new pipeline, you will find that any running application pods will not have the updated secrets.
  • Variables with multiline values are not currently supported
  • The normal behavior of Auto DevOps is to use Continuous Deployment, pushing automatically to the production environment every time a new pipeline is run on the default branch.
  • If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to 1 as a CI/CD variable), then the application will be automatically deployed to a staging environment, and a production_manual job will be created for you when you’re ready to manually deploy to production.
  • If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to 1 as a CI/CD variable) then two manual jobs will be created: canary which will deploy the application to the canary environment production_manual which is to be used by you when you’re ready to manually deploy to production.
  • If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead of the standard production job, 4 different manual jobs will be created: rollout 10% rollout 25% rollout 50% rollout 100%
  • The percentage is based on the REPLICAS variable and defines the number of pods you want to have for your deployment.
  • To start a job, click on the play icon next to the job’s name.
  • Once you get to 100%, you cannot scale down, and you’d have to roll back by redeploying the old version using the rollback button in the environment page.
  • With INCREMENTAL_ROLLOUT_MODE set to manual and with STAGING_ENABLED
  • not all buildpacks support Auto Test yet
  • When a project has been marked as private, GitLab’s Container Registry requires authentication when downloading containers.
  • Authentication credentials will be valid while the pipeline is running, allowing for a successful initial deployment.
  • After the pipeline completes, Kubernetes will no longer be able to access the Container Registry.
  • We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.
張 旭

Docker can now run within Docker - Docker Blog - 0 views

www.docker.com/...cker-can-now-run-within-docker

devops docker

shared by 張 旭 on 11 Mar 22 - No Cached
  • Docker 0.6 is the new “privileged” mode for containers. It allows you to run some containers with (almost) all the capabilities of their host machine, regarding kernel features and device access.
  • Among the (many!) possibilities of the “privileged” mode, you can now run Docker within Docker itself.
  • in the new privileged mode.
  • ...8 more annotations...
  • that /var/lib/docker should be a volume. This is important, because the filesystem of a container is an AUFS mountpoint, composed of multiple branches; and those branches have to be “normal” filesystems (i.e. not AUFS mountpoints).
  • /var/lib/docker, the place where Docker stores its containers, cannot be an AUFS filesystem.
  • we use them as a pass-through to the “normal” filesystem of the host machine.
  • The /var/lib/docker directory of the nested Docker will live somewhere in /var/lib/docker/volumes on the host system.
  • since the private Docker instances run in privileged mode, they can easily escalate to the host, and you probably don’t want this! If you really want to run something like this and expose it to the public, you will have to fine-tune the LXC template file, to restrict the capabilities and devices available to the Docker instances.
  • When you are inside a privileged container, you can always nest one more level
  • the LXC tools cannot start nested containers if the devices control group is not in its own hierarchy.
  • if you use AppArmor, you need a special policy to support nested containers.
張 旭

Docker ARG, ENV and .env - a Complete Guide · vsupalov.com - 1 views

vsupalov.com/docker-arg-env-variable-guide

docker development system server environment

shared by 張 旭 on 10 Dec 18 - No Cached
  • understand and use Docker build-time variables, environment variables and docker-compose templating the right way.
  • ARG is only available during the build of a Docker image (RUN etc), not after the image is created and containers are started from it (ENTRYPOINT, CMD).
  • ENV values are available to containers, but also RUN-style commands during the Docker build starting with the line where they are introduced.
  • ...20 more annotations...
  • set an environment variable in an intermediate container using bash (RUN export VARI=5 && …) it will not persist in the next command.
  • An env_file, is a convenient way to pass many environment variables to a single command in one batch.
  • not be confused with a .env file
  • the dot in front of env - .env, not an “env_file”.
  • If you have a file named .env in your project, it’s only used to put values into the docker-compose.yml file which is in the same folder. Those are used with Docker Compose and Docker Stack.
  • Just type docker-compose config. This way you’ll see how the docker-compose.yml file content looks after the substitution step has been performed without running anything else.
  • ARG are also known as build-time variables. They are only available from the moment they are ‘announced’ in the Dockerfile with an ARG instruction up to the moment when the image is built.
  • Running containers can’t access values of ARG variables.
  • ENV variables are also available during the build, as soon as you introduce them with an ENV instruction. However, unlike ARG, they are also accessible by containers started from the final image.
  • ENV values can be overridden when starting a container,
  • If you don’t provide a value to expected ARG variables which don’t have a default, you’ll get an error message.
  • args block
  • You can use ARG to set the default values of ENV vars.
  • dynamic on-build env values
  • 2. Pass environment variable values from your host
  • 1. Provide values one by one
  • 3. Take values from a file (env_file)
  • for each RUN statement, a new container is launched from an intermediate image.
  • An image is saved by the end of the command, but environment variables do not persist that way.
  • The precedence is, from stronger to less-strong: stuff the containerized application sets, values from single environment entries, values from the env_file(s) and finally Dockerfile defaults.
張 旭

Kubernetes Volumes Guide - Examples for NFS and Persistent Volume - 0 views

matthewpalmer.net/...ple-nfs-persistent-volume.html

kubernetes volume storage

shared by 張 旭 on 07 Jul 19 - No Cached
  • Persistent volumes exist beyond containers, pods, and nodes.
  • Volumes also let you share data between containers in the same pod.
  • data in that volume will be destroyed when the pod is restarted.
  • ...9 more annotations...
  • Persistent volumes are long-term storage in your Kubernetes cluster.
  • A pod uses a persistent volume claim to to get read and write access to the persistent volume.
  • NFS stands for Network File System – it's a shared filesystem that can be accessed over the network.
  • The NFS must already exist – Kubernetes doesn't run the NFS, pods in just access it.
  • what's already stored in the NFS is not deleted when a pod is destroyed. Data is persistent.
  • an NFS can be accessed from multiple pods at the same time. An NFS can be used to share data between pods!
  • volumes: - name: nfs-volume nfs: # URL for the NFS server server: 10.108.211.244 # Change this! path: /
  • volumeMounts: - name: nfs-volume mountPath: /var/nfs
  • Just add the volume to each pod, and add a volume mount to use the NFS volume from each container.
  • 張 旭 on 07 Jul 19
     
    "Persistent volumes exist beyond containers, pods, and nodes. "
張 旭

Automated Nginx Reverse Proxy for Docker - 0 views

jasonwilder.com/...nginx-reverse-proxy-for-docker

docker nginx proxy haproxy system web server

shared by 張 旭 on 04 Dec 15 - No Cached
  • Docker containers are assigned random IPs and ports which makes addressing them much more complicated from a client perspsective
  • Binding the container to the hosts port can prevent multiple containers from running on the same host. For example, only one container can bind to port 80 at a time.
  • Docker provides a remote API to inspect containers and access their IP, Ports and other configuration meta-data.
  • ...1 more annotation...
  • nginx template can be used to generate a reverse proxy configuration for docker containers using virtual hosts for routing.
張 旭

Docker image building on GitLab CI | $AYMDEV() - 0 views

aymdev.io/...er-image-building-on-gitlab-ci

devops cicd

shared by 張 旭 on 11 Mar 22 - No Cached
  • Continuous Integration (or CI) is a practice where you continously test an application to detect errors as soon as possible.
  • Docker is a container technology, many CI tools execute jobs (the tasks of a pipeline) in container to have an isolated environment.
  • Docker in Docker (« DinD » in short) means executing Docker in a Docker container.
  • ...11 more annotations...
  • images are saved in the host registry, we can benefit from Docker layer caching
  • All jobs will share the same environment, if many of them run simultaneously they might get into conflicts.
  • storage management (accumulating images)
  • The Docker socket binding technique means making a volume of /var/run/docker.sock between host and containers.
  • all containers would share the same Docker daemon.
  • Add privileged = true in the [runners.docker] section, the privileged mode is mandatory to use DinD.
  • To avoid that the runner only run one job at a time, change the concurrent value on the first line.
  • To avoid building a Docker image at each job, it can be built in a first job, pushed to the image registry provided by GitLab, and pulled in the next jobs.
  • functional tests depending on a database.
  • Docker Compose allows you to easily start multiple containers, but it has no more feature than Docker itself
  • Docker in Docker works well, but has its drawbacks, like Docker layer caching which needs some more commands to be used.
張 旭

Containers Vs. Config Management - 0 views

blog.containership.io/config-management-e64cbb744a94

docker devops deploy config ansible puppet system

shared by 張 旭 on 16 Feb 17 - No Cached
  • With configuration management systems, you write code that describes how you want some component of your systems to be installed and configured, and when you execute the code on your server, it should end up in the desired state.
  • building a hosting platform that is capable of a lot of things that system administrators used to do manually
  • build modules on deployment via bundler or npm or similar, it can be incredibly slow to run, taking minutes or longer in some cases
  • ...10 more annotations...
  • pulling from git is slow.
  • deploying with configuration management tools is a pain in the ass and error prone.
  • Support for containers has existed in the Linux kernel since version 2.6.24 when cgroup support was added
  • All of the logic that used to live in your cookbooks/playbooks/manifests/etc now lives in a Dockerfile that resides directly in the repository for the application it is designed to build
  • All of the dependencies of the application are bundled with the container which means no need to build on the fly on every server during deployment.
  • Containers bring standardization which allows for systems like centralized logging, monitoring, and metrics to easily snap into place no matter what is running in the container.
  • Dockerfiles do not give you the same level of control over configuration as your application transitions between environments, like dev, staging, and production.
  • You may even need to have different Dockerfile’s for each environment in certain cases.
  • configuration management systems now have hooks for docker integration.
  • Config management will only be used to install Docker, an orchestration system, configure PAM/SSH auth, and tune OS sysctl values.
  • 張 旭 on 16 Feb 17
     
    "With configuration management systems, you write code that describes how you want some component of your systems to be installed and configured, and when you execute the code on your server, it should end up in the desired state."
張 旭

Getting Started with Docker - Servers for Hackers - 0 views

serversforhackers.com/getting-started-with-docker

docker system virtualization

shared by 張 旭 on 30 Apr 15 - No Cached
  • Docker is an isolated portion of the host computer, sharing the host kernel (OS) and even its bin/libraries if appropriate.
  • the Docker Container contains the parts that make Ubuntu different from CoreOS.
  • A Docker container only stays alive as long as there is an active process being run in it.
  • ...10 more annotations...
  • Allocate a (pseudo) tty
  • Keep stdin open (so we can interact with it)
  • Docker allows us make changes to an image, commit those changes, and then push those changes out somehwere.
  • Docker tracks any changes we make to a container
  • The Dockerfile provides a set of instructions for Docker to run on a container.
  • what image (and tag in this case) to base this off of
  • run the given command (as user "root")
  • copy a file from the host machine into the container
  • expose a port to the host machine. You can expose multiple ports
  • run a command
張 旭

Configuration - docker-sync 0.5.10 documentation - 0 views

docker-sync.readthedocs.io/...configuration.html

docker development rsync volume performance

shared by 張 旭 on 13 Feb 19 - No Cached
  • Be sure to use a sync-name which is unique, since it will be a container name.
    • 張 旭
      張 旭 on 13 Feb 19
       
      慣例是 docker-sync 的 container name 後綴都是 -sync
  • split your docker-compose configuration for production and development (as usual)
  • ...9 more annotations...
  • production stack (docker-compose.yml) does not need any changes and would look like this (and is portable, no docker-sync adjustments).
  • docker-compose-dev.yml ( it needs to be called that way, look like this ) will override
    • 張 旭
      張 旭 on 13 Feb 19
       
      開發版的 docker-compose-dev.yml 僅會覆寫 production docker-compose.yml 的 volumes 設定，也就接上 docker-sync.yml 的 volumes，其它都維持不變
  • nocopy # nocopy is important
  • nocopy # nocopy is important
  • docker-compose -f docker-compose.yml -f docker-compose-dev.yml up
  • add the external volume and the mount here
  • In case the folder we mount to has been declared as a VOLUME during image build, its content will be merged with the name volume we mount from the host
    • 張 旭
      張 旭 on 13 Feb 19
       
      如果在 Dockerfile 裡面有宣告一個 volume，那麼在 docker build 的時候這個 volume mount point 會被記錄起來，在 container 跑起來的時候，會將 host (server) 上的同名的 volume 內容合併進來 (取代)。也就是說 container 跑起來的時候，會去接上已經存在的既有的 host (server) 上的 volume。
  • enforce the content from our host on the initial wiring
  • set your environment variables by creating a .env file at the root of your project
  • 張 旭 on 13 Feb 19
     
    "Be sure to use a sync-name which is unique, since it will be a container name."
張 旭

Baseimage-docker: A minimal Ubuntu base image modified for Docker-friendliness - 0 views

phusion.github.io/baseimage-docker

docker base system linux

shared by 張 旭 on 29 Oct 14 - No Cached
  • We encourage you to use multiple processes.
  • Baseimage-docker is a special Docker image that is configured for correct use within Docker containers.
  • When your Docker container starts, only the CMD command is run.
  • ...16 more annotations...
  • You're not running them, you're only running your app.
  • You have Ubuntu installed in Docker. The files are there. But that doesn't mean Ubuntu's running as it should.
  • The only processes that will be running inside the container is the CMD command, and all processes that it spawns.
  • A proper Unix system should run all kinds of important system services.
  • Ubuntu is not designed to be run inside Docker
  • When a system is started, the first process in the system is called the init process, with PID 1. The system halts when this processs halts.
  • Runit (written in C) is much lighter weight than supervisord (written in Python).
  • Docker runs fine with multiple processes in a container.
  • Baseimage-docker encourages you to run multiple processes through the use of runit.
  • If your init process is your app, then it'll probably only shut down itself, not all the other processes in the container.
  • a Docker container, which is a locked down environment with e.g. no direct access to many kernel resources.
  • Used for service supervision and management.
  • A custom tool for running a command as another user.
  • add additional daemons (e.g. your own app) to the image by creating runit entries.
  • write a small shell script which runs your daemon, and runit will keep it up and running for you, restarting it when it crashes, etc.
  • the shell script must run the daemon without letting it daemonize/fork it.
張 旭

你到底知不知道什麼是 Kubernetes? | Hwchiu Learning Note - 0 views

www.hwchiu.com/kubernetes-concept.html

kubernetes storage networking programming system

shared by 張 旭 on 03 Feb 19 - No Cached
  • Storage(儲存) 實際上一直都不是一個簡單處理的問題，從軟體面來看實際上牽扯到非常多的層級，譬如 Linux Kernel, FileSystem, Block/File-Level, Cache, Snapshot, Object Storage 等各式各樣的議題可以討論。
  • DRBD
  • 異地備援，容錯機制，快照，重複資料刪除等超多相關的議題基本上從來沒有一個完美的解法能夠滿足所有使用情境。
  • ...20 more annotations...
  • 管理者可能會直接在 NFS Server 上進行 MDADM 來設定相關的 Block Device 並且基於上面提供 Export 供 NFS 使用，甚至底層套用不同的檔案系統 (EXT4/BTF4) 來獲取不同的功能與效能。
  • Kubernetes 就只是 NFS Client 的角色
  • CSI(Container Storage Interface)。CSI 本身作為 Kubernetes 與 Storage Solution 的中介層。
  • 基本上 Pod 裡面每個 Container 會使用 Volume 這個物件來代表容器內的掛載點，而在外部實際上會透過 PVC 以及 PV 的方式來描述這個 Volume 背後的儲存方案伺服器的資訊。
  • 整體會透過 CSI 的元件們與最外面實際上的儲存設備連接，所有儲存相關的功能是否有實現，有支援全部都要仰賴最後面的實際提供者， kubernetes 只透過 CSI 的標準去執行。
  • 在網路部分也有與之對應的 CNI(Container Network Interface). kubernetes 透過 CNI 這個介面來與後方的 網路解決方案 溝通
  • CNI 最基本的要求就是在在對應的階段為對應的容器提供網路能力
  • 目前最常見也是 IPv4 + TCP/UDP 的傳輸方式，因此才會看到大部分的 CNI 都在講這些。
  • 希望所有容器彼此之間可以透過 IPv4 來互相存取彼此，不論是同節點或是跨節點的容器們都要可以滿足這個需求。
  • 容器間到底怎麼傳輸的，需不需要封裝，透過什麼網卡，要不要透過 NAT 處理? 這一切都是 CNI 介面背後的實現
  • 外部網路存取容器服務 (Service/Ingress)
  • kubernetes 在 Service/Ingress 中間自行實現了一個模組，大抵上稱為 kube-proxy, 其底層可以使用 iptables, IPVS, user-space software 等不同的實現方法，這部分是跟 CNI 完全無關。
  • CNI 跟 Service/Ingress 是會衝突的，也有可能彼此沒有配合，這中間沒有絕對的穩定整合。
  • CNI 一般會處理的部份，包含了容器內的 網卡數量,網卡名稱,網卡IP, 以及容器與外部節點的連接能力等
  • CRI (Container Runtime Interface) 或是 Device Plugin
  • 對於 kubernetes 來說，其實本身並不在意到底底下的容器化技術實際上是怎麼實現的，你要用 Docker, rkt, CRI-O 都無所謂，甚至背後是一個偽裝成 Container 的 Virtaul Machine virtlet 都可以。
  • 去思考到底為什麼自己本身的服務需要容器化，容器化可以帶來什麼優點
  • 太多太多的人都認為只要寫一個 Dockerfile 將原先的應用程式們全部包裝起來放在一起就是一個很好的容器 來使用了。
  • 最後就會發現根本把 Container 當作 Virtual Machine 來使用，然後再補一句 Contaienr 根本不好用啊
  • 容器化 不是把直接 Virtual Machine 的使用習慣換個環境使用就叫做 容器化，而是要從概念上去暸解與使用
張 旭

Swarm mode key concepts | Docker Documentation - 0 views

docs.docker.com/...key-concepts

docker cluster system server HA

shared by 張 旭 on 16 Jun 17 - No Cached
  • The cluster management and orchestration features embedded in the Docker Engine are built using SwarmKit.
  • Docker engines participating in a cluster are running in swarm mode
  • A swarm is a cluster of Docker engines, or nodes, where you deploy services
  • ...19 more annotations...
  • When you run Docker without using swarm mode, you execute container commands.
  • When you run the Docker in swarm mode, you orchestrate services.
  • You can run swarm services and standalone containers on the same Docker instances.
  • A node is an instance of the Docker engine participating in the swarm
  • You can run one or more nodes on a single physical computer or cloud server
  • To deploy your application to a swarm, you submit a service definition to a manager node.
  • Manager nodes also perform the orchestration and cluster management functions required to maintain the desired state of the swarm.
  • Manager nodes elect a single leader to conduct orchestration tasks.
  • Worker nodes receive and execute tasks dispatched from manager nodes.
  • service is the definition of the tasks to execute on the worker nodes
  • When you create a service, you specify which container image to use and which commands to execute inside running containers.
  • replicated services model, the swarm manager distributes a specific number of replica tasks among the nodes based upon the scale you set in the desired state.
  • global services, the swarm runs one task for the service on every available node in the cluster.
  • A task carries a Docker container and the commands to run inside the container
  • Manager nodes assign tasks to worker nodes according to the number of replicas set in the service scale.
  • Once a task is assigned to a node, it cannot move to another node
  • If you do not specify a port, the swarm manager assigns the service a port in the 30000-32767 range.
  • External components, such as cloud load balancers, can access the service on the PublishedPort of any node in the cluster whether or not the node is currently running the task for the service.
  • Swarm mode has an internal DNS component that automatically assigns each service in the swarm a DNS entry.
張 旭

Gracefully Shutdown Docker Container - Kakashi's Blog - 1 views

kkc.github.io/...ully-shutdown-docker-container

docker process nodejs

shared by 張 旭 on 17 Nov 20 - No Cached
  • The initial idea is to make application invokes deconstructor of each component as soon as the application receives specific signals such as SIGTERM and SIGINT
  • When you run a docker container, by default it has a PID namespace, which means the docker process is isolated from other processes on your host.
  • The PID namespace has an important task to reap zombie processes.
  • ...11 more annotations...
  • This uses /bin/bash as PID1 and runs your program as the subprocess.
  • When a signal is sent to a shell, the signal actually won’t be forwarded to subprocesses.
  • By using the exec form, we can run our program as PID1
  • if you use exec form to run a shell script to spawn your application, remember to use exec syscall to overwrite /usr/bin/bash otherwise it will act as senario1
  • /bin/bash can handle repeating zombie process
  • with Tini, SIGTERM properly terminates your process even if you didn’t explicitly install a signal handler for it.
  • run tini as PID1 and it will forward the signal for subprocesses.
  • tini is a signal proxy and it also can deal with zombie process issue automatically.
  • run your program with tini by passing --init flag to docker run
  • use docker stop, docker will wait for 10s for stopping container before killing a process (by default). The main process inside the container will receive SIGTERM, then docker daemon will wait for 10s and send SIGKILL to terminate process.
  • kill running containers immediately. it’s more like kill -9 and kill --SIGKILL
張 旭

chaifeng/ufw-docker: To fix the Docker and UFW security flaw without disabling iptables - 0 views

github.com/ufw-docker

iptables docker

shared by 張 旭 on 13 Jul 22 - No Cached
  • It requires to disable docker's iptables function first, but this also means that we give up docker's network management function.
  • This causes containers will not be able to access the external network.
  • such as -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE. But this only allows containers that belong to network 172.17.0.0/16 can access outside.
  • ...13 more annotations...
  • Don't need to disable Docker's iptables and let Docker to manage it's network.
  • The public network cannot access ports that published by Docker.
  • In a very convenient way to allow/deny public networks to access container ports without additional software and extra configurations
  • Enable Docker's iptables feature. Remove all changes like --iptables=false , including configuration file /etc/docker/daemon.json
  • Modify the UFW configuration file /etc/ufw/after.rules
  • There may be some unknown reasons cause the UFW rules will not take effect after restart UFW, please reboot servers.
  • If we publish a port by using option -p 8080:80, we should use the container port 80, not the host port 8080
  • allow the private networks to be able to visit each other.
  • The following rules block connection requests initiated by all public networks, but allow internal networks to access external networks.
  • Since the UDP protocol is stateless, it is not possible to block the handshake signal that initiates the connection request as TCP does.
  • For GNU/Linux we can find the local port range in the file /proc/sys/net/ipv4/ip_local_port_range. The default range is 32768 60999
  • It not only exposes ports of containers but also exposes ports of the host.
  • Cannot expose services running on hosts and containers at the same time by the same command.
  • 張 旭 on 13 Jul 22
     
    "It requires to disable docker's iptables function first, but this also means that we give up docker's network management function."
張 旭

Docker Explained: Using Dockerfiles to Automate Building of Images | DigitalOcean - 0 views

www.digitalocean.com/...to-automate-building-of-images

docker tutorial

shared by 張 旭 on 01 Jun 15 - No Cached
  • CMD would be running an application upon creation of a container which is already installed using RUN (e.g. RUN apt-get install …) inside the image
  • ENTRYPOINT argument sets the concrete default application that is used every time a container is created using the image.
  • ENV command is used to set the environment variables (one or more).
  • ...6 more annotations...
  • EXPOSE command is used to associate a specified port to enable networking between the running process inside the container and the outside world
  • defines the base image to use to start the build process
  • Unlike CMD, it actually is used to build the image (forming another layer on top of the previous one which is committed).
  • VOLUME command is used to enable access from your container to a directory on the host machine
  • set where the command defined with CMD is to be executed
  • To detach yourself from the container, use the escape sequence CTRL+P followed by CTRL+Q
‹ Previous 21 - 40 of 178 Next › Last »
Showing 20▼ items per page