Group items tagged

"Kiama is a Scala library for language processing. It enables convenient analysis and transformation of structured data. The programming styles supported by the library are based on well-known formal language processing paradigms, including attribute grammars, tree rewriting, abstract state machines, and pretty printing."

"JSON Processing project is the open source reference implementation of JSR 353 - Java API for JSON Processing. The JSR provides portable APIs to parse, generate, transform, and query JSON using the streaming API or the object model API."

"Cascalog is a fully-featured data processing and querying library for Clojure or Java. The main use cases for Cascalog are processing "Big Data" on top of Hadoop or doing analysis on your local computer. Cascalog is a replacement for tools like Pig, Hive, and Cascading and operates at a significantly higher level of abstraction than those tools."

SubScript, a way to extend common programming languages aimed to ease event handling and concurrency. Typical application areas are GUI controllers, text processing applications and discrete event simulations. SubScript is based on a mathematical concurrency theory named Algebra of Communicating Processes (ACP). ACP is a 30-year-old branch of mathematics, as solid as numeric algebra and as Boolean algebra. In fact, you can regard ACP as an extension to Boolean algebra with 'things that can happen'. These items are glued together with operations such alternative, sequential and parallel compositions. This way ACP combines the essence of grammar specification languages and notions of parallelism.

"Apache Giraph is an iterative graph processing system built for high scalability. For example, it is currently used at Facebook to analyze the social graph formed by users and their connections. Giraph originated as the open-source counterpart to Pregel, the graph processing architecture developed at Google and described in a 2010 paper. Both systems are inspired by the Bulk Synchronous Parallel model of distributed computation introduced by Leslie Valiant. Giraph adds several features beyond the basic Pregel model, including master computation, sharded aggregators, edge-oriented input, out-of-core computation, and more. With a steady development cycle and a growing community of users worldwide, Giraph is a natural choice for unleashing the potential of structured datasets at a massive scale."

"Datomic is a distributed database designed to enable scalable, flexible and intelligent applications, running on next-generation cloud architectures. It does this by: Bringing declarative data manipulation into the application, and the data with it Getting time, process and perception right Process (writes) require coordination Perception (reads) require none The past doesn't change Leveraging immutability, and a sound model of state Datomic has: ACID Transactions Joins A sound data model A logical query language - Datalog Thus, Datomic avoids the compromises and losses of many NoSQL solutions. In addition, it offers flexibility and power over the traditional model in supporting: Hierarchy Multi-valued attributes Minimal schema Reliable operation on unreliable, ephemeral cloud instances Time Datomic avoids manual caching and replication, complex configuration, sharding (automatic or manual), logging, locking, latching and disk management of traditional servers."

"NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day."

"Heluna offers a unique SaaS based anti-spam solution that processes millions of e-mail messages  daily. Heluna turned to the Typesafe Stack - made up of Scala, Akka and Play Framework - in order  to future-proof the platform, and allow it to easily scale out to process well into the hundreds of  millions of e-mails."

"We present a novel streaming algorithm for evaluating XPath expressions that use backward axes (parent and ancestor) and forward axes in a single document-order traversal of an XML document. Other streaming XPath processors, such as YFilter, XTrie, and TurboXPath handle only forward axes. We show through experiments that our algorithm significantly outperforms (by more than a factor of two) a traditional non-streaming XPath engine. Furthermore, since our algorithm only retains relevant portions of the input document in memory, it scales better than traditional XPath engines. It can process large documents; we have successfully tested documents over 1GB in size. On the other hand, the traditional XPath engine degrades considerably in performance for documents over 100 MB in size and fails to complete for documents of size over 200 MB."

"Graph for Scala is intended to provide basic graph functionality that seamlessly fits into the Scala standard collections library. Like the other members of scala.collection, Graph for Scala is an in-memory container that exposes a user-friendly interface without sacrificing functionality or flexibility. Graph for Scala also has ready-to-go implementations of JSON-Import/Export and Dot-Export - more popular graph serialization formats are coming soon. In addition, other powerful tools such as graph databases emulation and distributed graph processing are due to be supported."

Introducción a JSONP (Java API for JSON Processing) nacida de la JSR353.

"Resque is our Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later."

"Apache Flink is an open source platform for distributed stream and batch data processing."

Luna is a data processing and visualization environment built on a principle that people need an immediate connection to what they are building. It provides an ever-growing library of highly tailored, domain specific components and an extensible framework for building new ones.

"HDF (Hierarchical Data Format) technologies are relevant when the data challenges being faced push the limits of what can be addressed by traditional database systems, XML documents, or in-house data formats. Leveraging the powerful HDF products and the expertise of The HDF Group, organizations realize substantial cost savings while solving challenges that seemed intractable using other data management technologies. Many HDF adopters have very large datasets, very fast access requirements, or very complex datasets. Others turn to HDF because it allows them to easily share data across a wide variety of computational platforms using applications written in different programming languages. Some use HDF to take advantage of the many open-source and commercial tools that understand HDF. Similar to XML documents, HDF files are self-describing and allow users to specify complex data relationships and dependencies. In contrast to XML documents, HDF files can contain binary data (in many representations) and allow direct access to parts of the file without first parsing the entire contents. HDF, not surprisingly, allows hierarchical data objects to be expressed in a very natural manner, in contrast to the tables of relational database. Whereas relational databases support tables, HDF supports n-dimensional datasets and each element in the dataset may itself be a complex object. Relational databases offer excellent support for queries based on field matching, but are not well-suited for sequentially processing all records in the database or for subsetting the data based on coordinate-style lookup."

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes.

CopyFS aims to solve a common problem : given a directory, especially one full of configuration files, or other files that one can modify, and which can affect the functionning of a system, or of programs, that may be important to other users (or to the user himself), how to be sure that a person modifying the files will do a backup of the working version first ? This filesystem solves the problem by making the whole process transparent, automatically keeping versionned copies of all the changes done to file under its control. It also allows a user to select an old version of the files, for example to repair a mistake, and allows him/her to continue edition from this point.

Nux is an open-source Java toolkit making efficient and powerful XML processing easy. It is geared towards embedded use in high-throughput XML messaging middleware such as large-scale Peer-to-Peer infrastructures, message queues, publish-subscribe and matchmaking systems for Blogs/newsfeeds, text chat, data acquisition and distribution systems, application level routers, firewalls, classifiers, etc.