Group items tagged

Short answer: Apache Mesos doesn't provide distributed FS. So, apps have to work with local FS on slaves or you may run any distributed FS alongside Mesos. Mesos is typically deployed together with HDFS, and most of the frameworks that run on top of Mesos can work with HDFS (Hadoop, Spark, Storm, etc.) And in case your app doesn't support any distributed fs, it has to work with local FS on each slave. I run ElasticSearch on top of Mesos: I specified local directories for ES data in the config file that each mesos slave take when I start framework. So if I restart ES framework, each ES slave will use the specified directories, and if there's some data, that data will be used. I run multiple instances of ES, and they replicate data between each other, so I don't need to worry about losing data. However, there may be a problem at some point: let's say, I have 4 mesos slaves, I run ES on 2 of them. Then I stop ES framework and start some other framework on 2 slaves, then I start ES framework again on 2 slaves, but mesos doesn't guaranty that 2 new slaves are the same 2 slaves that I was running ES on previously. So there'll be no previous copy of data. I run ES on majority of Mesos slaves and I never stop the framework, so I've never encountered this problem.

at a certain scale an abstraction becomes necessary to let developers and operators focus on generic “computation jobs” instead of managing and configuring hosts and their processes.

Systems with a scheduler at their core use the scheduler to run everything. It is the interface to the resources of the cluster.

A scheduling system