Skip to main content

Home/ Larvata/ Group items tagged build

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
張 旭

Speeding up Docker image build process of a Rails application | BigBinary Blog - 1 views

blog.bigbinary.com/...ss-of-a-rails-application.html

docker kubernetes rails devops system

shared by 張 旭 on 23 Oct 19 - No Cached
  • we do not want to execute bundle install and rake assets:precompile tasks while starting a container in each pod which would prevent that pod from accepting any requests until these tasks are finished.
  • run bundle install and rake assets:precompile tasks while or before containerizing the Rails application.
  • Kubernetes pulls the image, starts a Docker container using that image inside the pod and runs puma server immediately.
  • ...7 more annotations...
  • Since source code changes often, the previously cached layer for the ADD instruction is invalidated due to the mismatching checksums.
  • The ARG instruction in the Dockerfile defines RAILS_ENV variable and is implicitly used as an environment variable by the rest of the instructions defined just after that ARG instruction.
  • RUN instructions are used to install gems and precompile static assets using sprockets
  • Instead, Docker automatically re-uses the previously built layer for the RUN bundle install instruction if the Gemfile.lock file remains unchanged.
  • everyday we need to build a lot of Docker images containing source code from varying Git branches as well as with varying environments.
  • it is hard for Docker to cache layers for bundle install and rake assets:precompile tasks and re-use those layers during every docker build command run with different application source code and a different environment.
  • By default, Bundler installs gems at the location which is set by Rubygems.
  • 張 旭 on 23 Oct 19
     
    "we do not want to execute bundle install and rake assets:precompile tasks while starting a container in each pod which would prevent that pod from accepting any requests until these tasks are finished."
crazylion lee

mpociot/botman: A framework agnostic PHP library to build chat bots - 0 views

github.com/botman

php chat bot

shared by crazylion lee on 15 Jan 17 - No Cached
  • crazylion lee on 15 Jan 17
     
    "A framework agnostic PHP library to build chat bots http://botman.io"
crazylion lee

jamis/castaway: System for building screencasts and video presentations - 0 views

github.com/castaway

screencapture screencast

shared by crazylion lee on 30 Jan 17 - No Cached
  • crazylion lee on 30 Jan 17
     
    "System for building screencasts and video presentations"
crazylion lee

ianstormtaylor/slate: A completely customizable framework for building rich text editor... - 0 views

github.com/...slate

Javascript development wysiwyg

shared by crazylion lee on 15 Jul 16 - No Cached
  • crazylion lee on 15 Jul 16
     
    "A completely customizable framework for building rich text editors in the browser."
張 旭

MonolithFirst - 0 views

martinfowler.com/...MonolithFirst.html

system architecture deploy

shared by 張 旭 on 05 Jun 15 - No Cached
  • Microservices are a useful architecture, but even their advocates say that using them incurs a significant MicroservicePremium, which means they are only useful with more complex systems.
  • you should build a new application as a monolith initially, even if you think it's likely that it will benefit from a microservices architecture later on.
  • Any refactoring of functionality between services is much harder than it is in a monolith.
  • ...4 more annotations...
  • By building a monolith first, you can figure out what the right boundaries are, before a microservices design brushes a layer of treacle over them.
  • The logical way is to design a monolith carefully, paying attention to modularity within the software, both at the API boundaries and how the data is stored.
  • start with a monolith and gradually peel off microservices at the edges
  • Don't be afraid of building a monolith that you will discard, particularly if a monolith can get you to market quickly
張 旭

Scalable architecture without magic (and how to build it if you're not Google) - DEV Co... - 0 views

dev.to/...ld-it-if-youre-not-google-336a

development architecture system google

shared by 張 旭 on 24 Feb 19 - No Cached
  • Don’t mess up write-first and read-first databases.
  • keep them stateless.
  • you should know how to make a scalable setup on bare metal.
  • ...29 more annotations...
  • Different programming languages are for different tasks.
  • Go or C which are compiled to run on bare metal.
  • To run NodeJS on multiple cores, you have to use something like PM2, but since this you have to keep your code stateless.
  • Python have very rich and sugary syntax that’s great for working with data while keeping your code small and expressive.
  • SQL is almost always slower than NoSQL
  • databases are often read-first or write-first
  • write-first, just like Cassandra.
  • store all of your data to your databases and leave nothing at backend
  • Functional code is stateless by default
  • It’s better to go for stateless right from the very beginning.
  • deliver exactly the same responses for same requests.
  • Sessions? Store them at Redis and allow all of your servers to access it.
  • Only the first user will trigger a data query, and all others will be receiving exactly the same data straight from the RAM
  • never, never cache user input
  • Only the server output should be cached
  • Varnish is a great cache option that works with HTTP responses, so it may work with any backend.
  • a rate limiter – if there’s not enough time have passed since last request, the ongoing request will be denied.
  • different requests blasting every 10ms can bring your server down
  • Just set up entry relations and allow your database to calculate external keys for you
  • the query planner will always be faster than your backend.
  • Backend should have different responsibilities: hashing, building web pages from data and templates, managing sessions and so on.
  • For anything related to data management or data models, move it to your database as procedures or queries.
  • a distributed database.
  • your code has to be stateless
  • Move anything related to the data to the database.
  • For load-balancing a database, go for cluster.
  • DB is balancing requests, as well as your backend.
  • Users from different continents are separated with DNS.
  • Keep is scalable, keep is stateless.
  • 張 旭 on 24 Feb 19
     
    "Don't mess up write-first and read-first databases."
crazylion lee

flood-io/ruby-jmeter: A Ruby based DSL for building JMeter test plans - 0 views

github.com/ruby-jmeter

jmeter ruby

shared by crazylion lee on 12 Oct 16 - No Cached
  • crazylion lee on 12 Oct 16
     
    "A Ruby based DSL for building JMeter test plans"
張 旭

Best practices for building Kubernetes Operators and stateful apps | Google Cloud Blog - 0 views

cloud.google.com/...es-operators-and-stateful-apps

kubernetes system operator

shared by 張 旭 on 17 May 21 - No Cached
  • use the StatefulSet workload controller to maintain identity for each of the pods, and to use Persistent Volumes to persist data so it can survive a service restart.
  • a way to extend Kubernetes functionality with application specific logic using custom resources and custom controllers.
  • An Operator can automate various features of an application, but it should be specific to a single application
  • ...12 more annotations...
  • Kubebuilder is a comprehensive development kit for building and publishing Kubernetes APIs and Controllers using CRDs
  • Design declarative APIs for operators, not imperative APIs. This aligns well with Kubernetes APIs that are declarative in nature.
  • With declarative APIs, users only need to express their desired cluster state, while letting the operator perform all necessary steps to achieve it.
  • scaling, backup, restore, and monitoring. An operator should be made up of multiple controllers that specifically handle each of the those features.
  • the operator can have a main controller to spawn and manage application instances, a backup controller to handle backup operations, and a restore controller to handle restore operations.
  • each controller should correspond to a specific CRD so that the domain of each controller's responsibility is clear.
  • If you keep a log for every container, you will likely end up with unmanageable amount of logs.
  • integrate application-specific details to the log messages such as adding a prefix for the application name.
  • you may have to use external logging tools such as Google Stackdriver, Elasticsearch, Fluentd, or Kibana to perform the aggregations.
  • adding labels to metrics to facilitate aggregation and analysis by monitoring systems.
  • a more viable option is for application pods to expose a metrics HTTP endpoint for monitoring tools to scrape.
  • A good way to achieve this is to use open-source application-specific exporters for exposing Prometheus-style metrics.
張 旭

Full Cycle Developers at Netflix - Operate What You Build - 1 views

netflixtechblog.com/lopers-at-netflix-a08c31f83249

devops system

shared by 張 旭 on 27 May 20 - No Cached
  • Researching issues felt like bouncing a rubber ball between teams, hard to catch the root cause and harder yet to stop from bouncing between one another.
  • In the past, Edge Engineering had ops-focused teams and SRE specialists who owned the deploy+operate+support parts of the software life cycle
  • hearing about those problems second-hand
  • ...17 more annotations...
  • devs could push code themselves when needed, and also were responsible for off-hours production issues and support requests
  • What were we trying to accomplish and why weren’t we being successful?
  • These specialized roles create efficiencies within each segment while potentially creating inefficiencies across the entire life cycle.
  • Grouping differing specialists together into one team can reduce silos, but having different people do each role adds communication overhead, introduces bottlenecks, and inhibits the effectiveness of feedback loops.
  • devops principles
  • develops a system also be responsible for operating and supporting that system
  • Each development team owns deployment issues, performance bugs, capacity planning, alerting gaps, partner support, and so on.
  • Those centralized teams act as force multipliers by turning their specialized knowledge into reusable building blocks.
  • Communication and alignment are the keys to success.
  • Full cycle developers are expected to be knowledgeable and effective in all areas of the software life cycle.
  • ramping up on areas they haven’t focused on before
  • We run dev bootcamps and other forms of ongoing training to impart this knowledge and build up these skills
  • “how can I automate what is needed to operate this system?”
  • “what self-service tool will enable my partners to answer their questions without needing me to be involved?”
  • A full cycle developer thinks and acts like an SWE, SDET, and SRE. At times they create software that solves business problems, at other times they write test cases for that, and still other times they automate operational aspects of that system.
  • the need for continuous delivery pipelines, monitoring/observability, and so on.
  • Tooling and automation help to scale expertise, but no tool will solve every problem in the developer productivity and operations space
張 旭

Overriding Auto Devops - 0 views

www.ilovedevops.com/...9-07-11-overriding-auto-devops

gitlab cicd

shared by 張 旭 on 10 Oct 21 - No Cached
  • most customers need to modify the devops pipeline to suit there needs
  • include Auto Devops and override it.
  • include all of Auto Devops, just as if the Auto Devops checkbox were checked for the project
  • ...4 more annotations...
  • skips for all the scans, as a way of speeding up the build process while working on the CI configuration
  • The Auto Devops test job, which uses Herokuish for testing, does not rely on the Docker image that’s generated during the Build job
  • moving the Test job to the Build stage to speed things along
  • Literally any part of Auto Devops can be overridden in your own CI configuration.
張 旭

Docker image building on GitLab CI | $AYMDEV() - 0 views

aymdev.io/...er-image-building-on-gitlab-ci

devops cicd

shared by 張 旭 on 11 Mar 22 - No Cached
  • Continuous Integration (or CI) is a practice where you continously test an application to detect errors as soon as possible.
  • Docker is a container technology, many CI tools execute jobs (the tasks of a pipeline) in container to have an isolated environment.
  • Docker in Docker (« DinD » in short) means executing Docker in a Docker container.
  • ...11 more annotations...
  • images are saved in the host registry, we can benefit from Docker layer caching
  • All jobs will share the same environment, if many of them run simultaneously they might get into conflicts.
  • storage management (accumulating images)
  • The Docker socket binding technique means making a volume of /var/run/docker.sock between host and containers.
  • all containers would share the same Docker daemon.
  • Add privileged = true in the [runners.docker] section, the privileged mode is mandatory to use DinD.
  • To avoid that the runner only run one job at a time, change the concurrent value on the first line.
  • To avoid building a Docker image at each job, it can be built in a first job, pushed to the image registry provided by GitLab, and pulled in the next jobs.
  • functional tests depending on a database.
  • Docker Compose allows you to easily start multiple containers, but it has no more feature than Docker itself
  • Docker in Docker works well, but has its drawbacks, like Docker layer caching which needs some more commands to be used.
張 旭

Optimizing Gitlab pipelines - Basics (1) | PrinsFrank.nl - 0 views

prinsfrank.nl/...g-gitlab-pipelines-pt-1-basics

devops gitlab cicd proxy performance

shared by 張 旭 on 23 Feb 22 - No Cached
  • When you use specific docker image, make sure you have the Dependency Proxy enabled so the image doesn’t have to be downloaded again for every job.
  • stages are used to group items that can run at the same time.
  • Instead of waiting for all jobs to finish, you can mark jobs as interruptible which signals a job to cancel when a new pipeline starts for the same branch
  • ...8 more annotations...
  • mark all jobs as interruptible as it doesn’t make sense to wait for builds and tests based on old information.
  • Deployment jobs are the main exception as they should probably finish.
  • only running it when specific files have changed
  • To prevent the ‘vendor’ and ‘node_modules’ folder from being regenerated in every job, we can configure a build job for composer and npm assets.
  • To share assets between multiple stages, Gitlab has caches and artifacts. For dependencies we should use caches.
  • The pull-push policy is the default, but specified here for clarity.
  • All consecutive runs for the build step with the same ‘composer.lock’ file don’t update the cache.
  • composer prevents this by caching packages in a global package cache,
crazylion lee

Eve - 0 views

programming.witheve.com

programming ide visual

shared by crazylion lee on 31 Oct 16 - No Cached
  • crazylion lee on 31 Oct 16
     
    "Eve is a programming language and IDE based on years of research into building a human-first programming platform. From code embedded in documents to a language without order, it presents an alternative take on what programming could be - one that focuses on us instead of the machine. This is Eve:"
張 旭

Building a RESTful API in a Rails application - 0 views

www.airpair.com/...ful-api-in-a-rails-application

rails api REST

shared by 張 旭 on 05 Dec 16 - No Cached
  • designing and implementing a REST API in an intentionally simplistic task management web application, and will cover some best practices to ensure maintainability of the code.
  • each individual request should have no context of the requests that came before it.
  • each request that modifies the database should act on one and only one row of one and only one table
  • ...10 more annotations...
  • The resource endpoints should return representations of the resource as data, usually XML or JSON.
  • POST for create, PUT for update, PATCH for upsert (update and insert).
  • an existing API should never be modified, except for critical bugfixes
  • Rather than changing existing endpoints, expose a new version
  • using unique database ids in the route chain allows users to access short routes, and simplifies resource lookup
  • while exposing internal database ids to the consumer and requiring the consumer to maintain a reference to ids on their end
  • The downfall is longer nested routes
  • require reauthentication on a per-request level
  • Devise.secure_compare helps avoid timing attacks
  • Defensive programming is a software design principle that dictates that a piece of software should be designed to continue functioning in unforeseen circumstances.
crazylion lee

PouchDB, the JavaScript Database that Syncs! - 0 views

pouchdb.com

javascript pouchDB db

shared by crazylion lee on 08 Dec 16 - No Cached
  • crazylion lee on 08 Dec 16
     
    " PouchDB is an open-source JavaScript database inspired by Apache CouchDB that is designed to run well within the browser. PouchDB was created to help web developers build applications that work as well offline as they do online. It enables applications to store data locally while offline, then synchronize it with CouchDB and compatible servers when the application is back online, keeping the user's data in sync no matter where they next login."
crazylion lee

Akka - 0 views

akka.io/?_ga=1.69687891.271246675.1483202477

java distributed jvm concurrency

shared by crazylion lee on 01 Jan 17 - No Cached
  • crazylion lee on 01 Jan 17
     
    "Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM. "
crazylion lee

Nix: The Purely Functional Package Manager - 0 views

nixos.org/nix

nix homebrew package management

shared by crazylion lee on 28 May 16 - No Cached
  • crazylion lee on 28 May 16
     
    "Nix The Purely Functional Package Manager Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments. Read more…"
crazylion lee

Pakyow Web Framework - 0 views

pakyow.org

web framework ruby

shared by crazylion lee on 10 Jun 16 - No Cached
  • crazylion lee on 10 Jun 16
     
    "Pakyow lets you build modern apps that don't break the web."
‹ Previous 21 - 40 of 107 Next › Last »
Showing 20▼ items per page