Group items tagged

Executor of the underlying technology

Image is a Docker image

Steps starting with a required checkout Step and followed by run: keys that execute commands sequentially on the primary container.

Docker images are typically configured using environment variables,

Continuous integration is a practice that encourages developers to integrate their code into a master branch of a shared repository early and often.

The machine option runs your jobs in a dedicated, ephemeral VM

Using the machine executor gives your application full access to OS resources and provides you with full control over the job environment.

Using the macos executor allows you to run your job in a macOS environment on a VM.

All containers run in a common network and every exposed port will be available on localhost from a primary container.

If you want to work with private images/registries, please refer to Using Private Images.

Docker also has built-in image caching and enables you to build, run, and publish Docker images via Remote Docker.

if you require low-level access to the network or need to mount external volumes consider using machine

Workspaces persist data between jobs in a single Workflow.

Caching persists data between the same job in different Workflow builds.

run using the machine executor which enables reuse of recently used machine executor runs,

docker executor which can compose Docker containers to run your tests and any services they require

macos executor

Steps are a collection of executable commands which are run during a job

checkout: key is required to checkout your code

run: enables addition of arbitrary, multi-line shell command scripting

orchestrating job runs with parallel, sequential, and manual approval workflows.

but building/pushing images and running containers happens in the remote Docker Engine

use a primary image that already has Docker (recommended)

installs Docker and has Git, use 17.05.0-ce-git

It is not possible to start a service in remote docker and ping it directly from a primary container or to start a primary container that can ping a service in remote docker.

use https://github.com/outstand/docker-dockup or a similar image for backup and restore to spin up a container

Commands are reusable sets of steps that you can invoke with specific parameters within an existing job.

you can pass my-executor as the value of a name key under executor. This method is primarily employed when passing parameters to executor invocations.

Development orbs are mutable and expire after 90 days.

Production Orbs are immutable and durable.

CircleCI allows development orbs that have versions that start with dev:

Production orbs are immutable

Each installation of CircleCI, including circleci.com, has only one registry where orbs can be kept.

Organization Admins publish production orbs.

Organization members publish development orbs

You must invoke jobs in the workflow stanza of config.yml file, making sure to pass any necessary parameters as subkeys to the job.

Executors define the environment in which the steps of a job will be run.

Executor declarations in config outside of jobs can be used by all jobs in the scope of that declaration, allowing you to reuse a single executor definition across multiple jobs.

It is also possible to allow an orb to define the executor used by all of its commands.

When invoking an executor in a job any keys in the job itself will override those of the executor invoked.

Steps are used when you have a job or command that needs to mix predefined and user-defined steps.

Use the enum parameter type when you want to enforce that the value must be one from a specific set of string values.

Use an executor parameter type to allow the invoker of a job to decide what executor it will run on

invoke the same job more than once in the workflows stanza of config.yml, passing any necessary parameters as subkeys to the job.

If a job is declared inside an orb it can use commands in that orb or the global commands.

To use parameters in executors, define the parameters under the given executor.

Parameters are in-scope only within the job or command that defined them.

A single configuration may invoke a job multiple times.

Pre and post steps allow you to execute steps in a given job without modifying the job.

conditions are checked before a workflow is actually run

Conditional steps may be located anywhere a regular step could and may only use parameter values as inputs.

A conditional step consists of a step with the key when or unless. Under this conditional key are the subkeys steps and condition

A condition is a single value that evaluates to true or false at the time the config is processed, so you cannot use environment variables as conditions

language images

service images

A language image should be listed first under the docker key in your configuration, making it the primary container during execution.

For example, if you want to add browsers to the circleci/golang:1.9 image, use the circleci/golang:1.9-browsers image.

Service images are convenience images for services like databases

should be listed after language images so they become secondary service containers.

To speed up builds using RAM volume, add the -ram suffix to the end of a service image tag

All convenience images have been extended with additional tools.

all images include the following packages, installed via apt-get

Most CircleCI convenience images are Debian Jessie- or Stretch-based images, however some extend Ubuntu-based images.

The following packages are installed via curl

most organizations practice continuous delivery, which means that your default branch can be deployed.

Merging everything into the master branch and frequently deploying means you minimize the amount of unreleased code, which is in line with lean and continuous delivery best practices.

you can deploy to production every time you merge a feature branch.

deploy a new version by merging master into the production branch.

you can have your deployment script create a tag on each deployment.

to have an environment that is automatically updated to the master branch

commits only flow downstream, ensures that everything is tested in all environments.

first merge these bug fixes into master, and then cherry-pick them into the release branch.

“merge request” since the final action is to merge the feature branch.

“pull request” since the first manual action is to pull the feature branch

it is common to protect the long-lived branches

After you merge a feature branch, you should remove it from the source control software

When you are ready to code, create a branch for the issue from the master branch. This branch is the place for any work related to this change.

A merge request is an online place to discuss the change and review the code.

To automatically close linked issues, mention them with the words “fixes” or “closes,” for example, “fixes #14” or “closes #67.” GitLab closes these issues when the code is merged into the default branch.

If you have an issue that spans across multiple repositories, create an issue for each repository and link all issues to a parent issue.

With Git, you can use an interactive rebase (rebase -i) to squash multiple commits into one or reorder them.

Rebasing creates new commits for all your changes, which can cause confusion because the same change would have multiple identifiers.

if someone has already reviewed your code, rebasing makes it hard to tell what changed since the last review.

always use the “no fast-forward” (--no-ff) strategy when you merge manually.

you should try to avoid merge commits in feature branches

you should never rebase commits you have pushed to a remote server

Sometimes you can reuse recorded resolutions (rerere), but merging is better since you only have to resolve conflicts once.

not frequently merge master into the feature branch.

utilizing new code,

resolving merge conflicts

updating long-running branches.

just cherry-picking a commit.

If your feature branch has a merge conflict, creating a merge commit is a standard way of solving this.

split your features into smaller units of work

you should try to prevent merge commits, but not eliminate them.

Your codebase should be clean, but your history should represent what actually happened.

Splitting up work into individual commits provides context for developers looking at your code later.

push your feature branch frequently, even when it is not yet ready for review.

When using GitLab flow, developers create their branches from this master branch, so it is essential that it never breaks. Therefore, each merge request must be tested before it is accepted.

Auto DevOps works with any Kubernetes cluster;

using the Docker or Kubernetes executor, with privileged mode enabled.

Base domain (needed for Auto Review Apps and Auto Deploy)

Kubernetes (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring)

Prometheus (needed for Auto Monitoring)

scrape your Kubernetes cluster.

project level as a variable: KUBE_INGRESS_BASE_DOMAIN

A wildcard DNS A record matching the base domain(s) is required

Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s).

review/ (every environment starting with review/)

staging

production

need to define a separate KUBE_INGRESS_BASE_DOMAIN variable for all the above based on the environment.

Continuous deployment to production: Enables Auto Deploy with master branch directly deployed to production.

Continuous deployment to production using timed incremental rollout

Automatic deployment to staging, manual deployment to production

Auto Build creates a build of the application using an existing Dockerfile or Heroku buildpacks.

If a project’s repository contains a Dockerfile, Auto Build will use docker build to create a Docker image.

Each buildpack requires certain files to be in your project’s repository for Auto Build to successfully build your application.

Auto Test automatically runs the appropriate tests for your application using Herokuish and Heroku buildpacks by analyzing your project to detect the language and framework.

Auto Code Quality uses the Code Quality image to run static analysis and other code checks on the current code.

Static Application Security Testing (SAST) uses the SAST Docker image to run static analysis on the current code and checks for potential security issues.

Dependency Scanning uses the Dependency Scanning Docker image to run analysis on the project dependencies and checks for potential security issues.

License Management uses the License Management Docker image to search the project dependencies for their license.

Vulnerability Static Analysis for containers uses Clair to run static analysis on a Docker image and checks for potential security issues.

Review Apps are temporary application environments based on the branch’s code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur.

The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain.

Your apps should not be manipulated outside of Helm (using Kubernetes directly).

Dynamic Application Security Testing (DAST) uses the popular open source tool OWASP ZAProxy to perform an analysis on the current code and checks for potential security issues.

Auto Browser Performance Testing utilizes the Sitespeed.io container to measure the performance of a web page.

add the paths to a file named .gitlab-urls.txt in the root directory, one per line.

After a branch or merge request is merged into the project’s default branch (usually master), Auto Deploy deploys the application to a production environment in the Kubernetes cluster, with a namespace based on the project name and unique project ID

Auto Deploy doesn’t include deployments to staging or canary by default, but the Auto DevOps template contains job definitions for these tasks if you want to enable them.

For internal and private projects a GitLab Deploy Token will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved.

If the GitLab Deploy Token cannot be found, CI_REGISTRY_PASSWORD is used. Note that CI_REGISTRY_PASSWORD is only valid during deployment.

If present, DB_INITIALIZE will be run as a shell command within an application pod as a helm post-install hook.

a post-install hook means that if any deploy succeeds, DB_INITIALIZE will not be processed thereafter.

DB_MIGRATE will be run as a shell command within an application pod as a helm pre-upgrade hook.

Once your application is deployed, Auto Monitoring makes it possible to monitor your application’s server and response metrics right out of the box.

annotate the NGINX Ingress deployment to be scraped by Prometheus using prometheus.io/scrape: "true" and prometheus.io/port: "10254"

While Auto DevOps provides great defaults to get you started, you can customize almost everything to fit your needs; from custom buildpacks, to Dockerfiles, Helm charts, or even copying the complete CI/CD configuration into your project to enable staging and canary deployments, and more.

Auto DevOps uses Helm to deploy your application to Kubernetes.

make use of the HELM_UPGRADE_EXTRA_ARGS environment variable to override the default values in the values.yaml file in the default Helm chart.

specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment.

Your additions will be merged with the Auto DevOps template using the behaviour described for include

copy and paste the contents of the Auto DevOps template into your project and edit this as needed.

In order to support applications that require a database, PostgreSQL is provisioned by default.

Set up the replica variables using a project variable and scale your application by just redeploying it!

You should not scale your application using Kubernetes directly.

Some applications need to define secret variables that are accessible by the deployed application.

Auto DevOps pipelines will take your application secret variables to populate a Kubernetes secret.

Environment variables are generally considered immutable in a Kubernetes pod.

If STAGING_ENABLED is defined in your project (e.g., set STAGING_ENABLED to 1 as a CI/CD variable), then the application will be automatically deployed to a staging environment, and a production_manual job will be created for you when you’re ready to manually deploy to production.

If CANARY_ENABLED is defined in your project (e.g., set CANARY_ENABLED to 1 as a CI/CD variable) then two manual jobs will be created: canary which will deploy the application to the canary environment production_manual which is to be used by you when you’re ready to manually deploy to production.

If INCREMENTAL_ROLLOUT_MODE is set to manual in your project, then instead of the standard production job, 4 different manual jobs will be created: rollout 10% rollout 25% rollout 50% rollout 100%

To start a job, click on the play icon next to the job’s name.

not all buildpacks support Auto Test yet

When a project has been marked as private, GitLab’s Container Registry requires authentication when downloading containers.

Authentication credentials will be valid while the pipeline is running, allowing for a successful initial deployment.

We strongly advise using GitLab Container Registry with Auto DevOps in order to simplify configuration and prevent any unforeseen issues.