Group items tagged

"Breaking distributed systems so you don't have to. Jepsen is a Clojure library. A test is a Clojure program which uses the Jepsen library to set up a distributed system, run a bunch of operations against that system, and verify that the history of those operations makes sense. Jepsen has been used to verify everything from eventually-consistent commutative databases to linearizable coordination systems to distributed task schedulers. It can also generate graphs of performance and availability, helping you characterize how a system responds to different faults. See jepsen.io for examples of the sorts of analyses you can carry out with Jepsen."

"Rust is a curly-brace, block-structured expression language. It visually resembles the C language family, but differs significantly in syntactic and semantic details. Its design is oriented toward concerns of "programming in the large", that is, of creating and maintaining boundaries - both abstract and operational - that preserve large-system integrity, availability and concurrency. It supports a mixture of imperative procedural, concurrent actor, object-oriented and pure functional styles. Rust also supports generic programming and metaprogramming, in both static and dynamic styles."

"Bandicoot is an open source programming system with a new set-based programming language, persistency capabilities, and run-time environment. The language is similar to general purpose programming languages where you write functions/methods and access data through variables. Though, in Bandicoot, you always manipulate data in sets using a small set-based algebra (the relational algebra)." "Here are the main features:   - functions are automatically exposed via HTTP using CSV for data, e.g. /List, /Append  - supports persistency via global variables (with transactions and ACID)  - can run on multiple computers to scale up the read throughput  - built in operators from the relational algebra with a simple syntax, e.g. "+" (union), "-" (minus)  - small binary ~100KB"

"Service Component Architecture (SCA) is a set of specifications which describe a model for building applications and systems using a Service-Oriented Architecture (SOA). SCA extends and complements prior approaches to implementing services, and SCA builds on open standards such as Web services. SCA is based on the idea that business function is provided as a series of services, which are assembled together to create solutions that serve a particular business need. These composite applications can contain both new services created specifically for the application and also business function from existing systems and applications, reused as part of the composition. SCA provides a model both for the composition of services and for the creation of service components, including the reuse of existing application function within SCA compositions. SCA aims to encompass a wide range of technologies for service components and for the access methods which are used to connect them. For components, this includes not only different programming languages, but also frameworks and environments commonly used with those languages. For access methods, SCA compositions allow for the use of various communication and service access technologies that are in common use, including, for example, Web services, messaging systems and Remote Procedure Call (RPC)."

"The sysout-over-slf4j module allows a user to redirect all calls to System.out and System.err to an SLF4J defined logger with the name of the fully qualified class in which the System.out.println (or similar) call was made, at configurable levels. "

"Apache Hama is a pure BSP (Bulk Synchronous Parallel) computing framework on top of HDFS (Hadoop Distributed File System) for massive scientific computations such as matrix, graph and network algorithms. Today, many practical data processing applications require a more flexible programming abstraction model that is compatible to run on highly scalable and massive data systems (e.g., HDFS, HBase, etc). A message passing paradigm beyond Map-Reduce framework would increase its flexibility in its communication capability. Bulk Synchronous Parallel (BSP) model fills the bill appropriately. Some of its significant advantages over MapReduce and MPI are: * Supports message passing paradigm style of application development * Provides a flexible, simple, and easy-to-use small APIs * Enables to perform better than MPI for communication-intensive applications * Guarantees impossibility of deadlocks or collisions in the communication mechanisms"

"I have two claims of which I would like to convince you today: The notion of the networked application API is an unsalvageable anachronism that fails to account for the necessary complexities of distributed systems. There exist a set of formalisms that do account for these complexities, but which are effectively absent from modern programming practice."

"Introduction In the modern era, software is commonly delivered as a service: called web apps, or software-as-a-service. The twelve-factor app is a methodology for building software-as-a-service apps that: Use declarative formats for setup automation, to minimize time and cost for new developers joining the project; Have a clean contract with the underlying operating system, offering maximum portability between execution environments; Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration; Minimize divergence between development and production, enabling continuous deployment for maximum agility; And can scale up without significant changes to tooling, architecture, or development practices. The twelve-factor methodology can be applied to apps written in any programming language, and which use any combination of backing services (database, queue, memory cache, etc). Background The contributors to this document have been directly involved in the development and deployment of hundreds of apps, and indirectly witnessed the development, operation, and scaling of hundreds of thousands of apps via our work on the Heroku platform. This document synthesizes all of our experience and observations on a wide variety of software-as-a-service apps in the wild. It is a triangulation on ideal practices for app development, paying particular attention to the dynamics of the organic growth of an app over time, the dynamics of collaboration between developers working on the app's codebase, and avoiding the cost of software erosion. Our motivation is to raise awareness of some systemic problems we've seen in modern application development, to provide a shared vocabulary for discussing those problems, and to offer a set of broad conceptual solutions to those problems with accompanying terminology. The format is inspired by Martin Fowler's books Patterns of Enterprise Application Architecture and Refactoring. Who should

Bueno. Eso. Compartí el que me di cuenta que puso antes Pablo en vez del original por error, pero la idea entre ambos, si la obviedad es tolerable, es idéntica :) Está muy bien estructurado en cuanto que cada factor depende de los demás a la vez que los promueve. Permite un enfoque general que incluye prácticas de arquitectura - y de armado cotidiano de productos - que posibilitan llegar donde yo entiendo - según me voy enterando - que es el lugar a donde llegar. Sin embargo, creo que ni éste departamento en sus sistemas más nuevos cumple todos y cada uno de aquellos factores. Esto, lejos de ser una crítica, es una invitación para que revisemos si es el único método posible - cosa improbabilísima - o el mejor método - también bastante improblable - a seguir. Lo que sí sostengo como un absoluto - quien no lo haría - es que es un método practicable. Mi aporte mínimo es defenderlo como uno bueno.

"Buck is a build system for Android that encourages the creation of small, reusable modules consisting of code and resources. Because Android applications are predominantly written in Java, Buck also functions as a Java build system. Learn more about the strategies Buck uses to build Java code so quickly."

"HDF (Hierarchical Data Format) technologies are relevant when the data challenges being faced push the limits of what can be addressed by traditional database systems, XML documents, or in-house data formats. Leveraging the powerful HDF products and the expertise of The HDF Group, organizations realize substantial cost savings while solving challenges that seemed intractable using other data management technologies. Many HDF adopters have very large datasets, very fast access requirements, or very complex datasets. Others turn to HDF because it allows them to easily share data across a wide variety of computational platforms using applications written in different programming languages. Some use HDF to take advantage of the many open-source and commercial tools that understand HDF. Similar to XML documents, HDF files are self-describing and allow users to specify complex data relationships and dependencies. In contrast to XML documents, HDF files can contain binary data (in many representations) and allow direct access to parts of the file without first parsing the entire contents. HDF, not surprisingly, allows hierarchical data objects to be expressed in a very natural manner, in contrast to the tables of relational database. Whereas relational databases support tables, HDF supports n-dimensional datasets and each element in the dataset may itself be a complex object. Relational databases offer excellent support for queries based on field matching, but are not well-suited for sequentially processing all records in the database or for subsetting the data based on coordinate-style lookup."

"The pipe Go package offers an easy way for Go programs to make use of other applications available in the system via a Unix-like pipeline mechanism. The input and output streams in such pipelines do work as streams, so large content can go across the pipeline without being loaded entirely in memory."

"Apache Giraph is an iterative graph processing system built for high scalability. For example, it is currently used at Facebook to analyze the social graph formed by users and their connections. Giraph originated as the open-source counterpart to Pregel, the graph processing architecture developed at Google and described in a 2010 paper. Both systems are inspired by the Bulk Synchronous Parallel model of distributed computation introduced by Leslie Valiant. Giraph adds several features beyond the basic Pregel model, including master computation, sharded aggregators, edge-oriented input, out-of-core computation, and more. With a steady development cycle and a growing community of users worldwide, Giraph is a natural choice for unleashing the potential of structured datasets at a massive scale."

Nux is an open-source Java toolkit making efficient and powerful XML processing easy. It is geared towards embedded use in high-throughput XML messaging middleware such as large-scale Peer-to-Peer infrastructures, message queues, publish-subscribe and matchmaking systems for Blogs/newsfeeds, text chat, data acquisition and distribution systems, application level routers, firewalls, classifiers, etc.

"Zipkin is a distributed tracing system that we created to help us gather timing data for all the disparate services involved in managing a request to the Twitter API."

"Flynn is two things: A "distribution" of components that out-of-the-box gives companies a reasonable starting point for an internal "platform" for running their applications and services. The banner for a collection of independent projects that together make up a toolkit or loose framework for building distributed systems. Flynn is both a whole and many parts, depending on what is most useful for you. The common goal is to democratize years of experience and best practices in building distributed systems. It is the software layer between operators and developers that makes both their lives easier. Unlike most PaaS's, Flynn can run stateful services as well as 12 factor apps. This includes built-in database appliances (just Postgres to start). Flynn is modular so users can easily modify, upgrade, and replace components. "

"Package mangos is an implementation in pure Go of the SP ("Scalability Protocols") messaging system. This makes heavy use of go channels, internally, but it can operate on systems that lack support for cgo."

CopyFS aims to solve a common problem : given a directory, especially one full of configuration files, or other files that one can modify, and which can affect the functionning of a system, or of programs, that may be important to other users (or to the user himself), how to be sure that a person modifying the files will do a backup of the working version first ? This filesystem solves the problem by making the whole process transparent, automatically keeping versionned copies of all the changes done to file under its control. It also allows a user to select an old version of the files, for example to repair a mistake, and allows him/her to continue edition from this point.

"In incoming Play2.1 version, a huge re-thinking has been done about JSON API provided by Play2.0.x which provides some great features but is clearly just the tip of the iceberg… Here is a first presentation of those evolutions aimed at unleashing your JSON usage in Play2 and revealing new forms of manipulation of web dataflows from/to external data systems. A usecase of this is manipulating DB structures directly using Json without any class models for document oriented structures such as MongoDB"

"Internet Routing: It's a program (well, a daemon, as you are going to discover in a moment) which works as a dynamic router in an Internet type network (that is, in a network running either the IPv4 or the IPv6 protocol). Routers are devices which forward packets between interconnected networks in order to allow hosts not connected directly to the same local area network to communicate with each other. They also communicate with the other routers in the Internet to discover the topology of the network which allows them to find optimal (in terms of some metric) rules for forwarding of packets (which are called routing tables) and to adapt themselves to the changing conditions such as outages of network links, building of new connections and so on. Most of these routers are costly dedicated devices running obscure firmware which is hard to configure and not open to any changes (on the other hand, their special hardware design allows them to keep up with lots of high-speed network interfaces, better than general-purpose computer does). Fortunately, most operating systems of the UNIX family allow an ordinary computer to act as a router and forward packets belonging to the other hosts, but only according to a statically configured table."