Group items tagged

"Amazon EC2 Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances. Amazon ECS eliminates the need for you to install, operate, and scale your own cluster management infrastructure. With simple API calls, you can launch and stop container-enabled applications, query the complete state of your cluster, and access many familiar features like security groups, Elastic Load Balancing, EBS volumes, and IAM roles. You can use Amazon ECS to schedule the placement of containers across your cluster based on your resource needs and availability requirements. You can also integrate your own scheduler or third-party schedulers to meet business or application specific requirements."

"The motivation for adding security to Apache Hadoop actually had little to do with traditional notions of security in defending against hackers since all large Hadoop clusters are behind corporate firewalls that only allow employees access. Instead, the motivation was simply that security would allow us to use Hadoop more effectively to pool resources between disjointed groups. Larger clusters are much cheaper to operate and require fewer copies of duplicated data."

"Vagrant project to spin up a cluster of 4 32-bit CentOS6.5 Linux virtual machines with Hadoop v2.6.0 and Spark v1.1.1"

"A typical Java EE application consists of an application server, such as WildFly, and a database, such as MySQL. In addition, you might have a separate front-end tier, say Apache, for load balancing a number of application server. A caching layer, such as Infinispan, may be used to improve overall application performance. Messaging system, such as ActiveMQ, may be used for processing queues. Both the caching and messaging components could be setup as a cluster for further scalability. This Tech Tip will show some simple Docker recipes to configure your containers that use application server and database. Subsequent blog will cover more advanced recipes that will include front-end, caching, messaging, and clustering. "

"CoreOS is a Linux distribution that has been rearchitected to provide features needed to run modern clustered infrastructure stacks. The strategies and architectures that influence CoreOS allow companies like Google, Facebook and Twitter to run their services at scale with high resilience. We've implemented them correctly so you don't have to endure the slow, learn-as-you-go infrastructure building process. CoreOS can run on your existing hardware or on most cloud providers. Clustering works across platforms, making it easy to migrate parts of your gear over to CoreOS, or to switch cloud providers while running CoreOS."

"Kubernetes is an open source implementation of container cluster management. Kubernetes is in pre-production beta! While the concepts and architecture in Kubernetes represent years of experience designing and building large scale cluster manager at Google, the Kubernetes project is still under heavy development. "

"What is Aurora? Platforms like Twitter operate across tens of thousands of machines, with hundreds of engineers deploying software daily. In this type of environment, automation is critical. Aurora is software that keeps services running in the face of many types of failure, and provides engineers a convenient, automated way to create and update these services. To accomplish this, Aurora leverages the Apache Mesos cluster manager, which provides information about the state of the cluster. Aurora uses that knowledge to make scheduling decisions. For example, when a machine experiences failure Aurora automatically reschedules those previously-running services onto a healthy machine in order to keep them running."

If you've been tasked with the job of maintaining large and complex Hadoop clusters, or are about to be, this book is a must. You'll learn the particulars of Hadoop operations, from planning, installing, and configuring the system to providing ongoing maintenance.

Importantísimo de ver... sobre todo para diferenciar bien entre un simple publicador de recursos, estilo openshift, de un verdadero administrador de recursos distribuidos (léase scheduler), como mesos.

"OK Log is a distributed and coördination-free log management system for big ol' clusters. I built it from first principles, to teach myself the gory details of shuffling logs around. This is the story of the prototype."

 Ø  The socket library that acts as a concurrency framework.  Ø  Faster than TCP, for clustered products and supercomputing.  Ø  Carries messages across inproc, IPC, TCP, and multicast.  Ø  Connect N-to-N via fanout, pubsub, pipeline, request-reply.  Ø  Asynch I/O for scalable multicore message-passing apps.  Ø  Large and active open source community.  Ø  30+ languages including C, C++, Java, .NET, Python.  Ø  Most OSes including Linux, Windows, OS X.  Ø  LGPL free software with full commercial support from iMatix.

"Titan is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster. Titan is a transactional database that can support thousands of concurrent users executing complex graph traversals."

"Nomad is a tool for managing a cluster of machines and running applications on them. Nomad abstracts away machines and the location of applications, and instead enables users to declare what they want to run and Nomad handles where they should run and how to run them."

"Ferry helps you create big data clusters on your local machine. Define your big data stack using YAML and share your application with Dockerfiles. Ferry supports Hadoop, Cassandra, Spark, GlusterFS, and Open MPI."

SequenceIQ is a new Hortonworks Technology Partner and recently achieved HDP and YARN Ready certification for Cloudbreak, the SequenceIQs Hadoop as a Service API. In this guest blog, SequenceIQ Co-founder and CTO Janos Matyas (@sequenceiq), describes provisioning and autoscaling HDP cluster with Cloudbreak. During our daily work at SequenceIQ, we are provisioning HDP clusters on different environments.