Group items tagged

I made some code public that does what you describe: it is a simple   cache interface that has implementations for in-memory, memcache and   the datastore.  You get about 100MB of heap space to use which can   significantly speed up your caching. There is also a CompositeCache class that allows you to layer the   caches so that it first checks in-memory, then memcache , then the   datastore.  Puts go to all levels and cache hits refresh the higher   levels.  e.g. if an item is not in-memory and has been flushed from   memcache but is still present in the datastore then the other two will   be updated.

GaeVFS is an Apache Commons VFS plug-in that implements a distributed, writeable virtual file system for Google App Engine (GAE) for Java. GaeVFS is implemented using the GAE datastore and memcache APIs. The primary goal of GaeVFS is to provide a portability layer that allows you to write application code to access the file system--both reads and writes--that runs unmodified in either GAE or non-GAE servlet environments.

The next problem that arises is that if the database has grown too big, then the csv generation will be slow(as it has to jump a lot of times). So the next idea is , instead of generating all cursors in the main cgi , the main cgi generates only first few set of page links, and then triggers a task queue entry to continue from there. By the time the user would have read the first page and clicks on "next page", the task would have generated few more entries. As the set of cursors for pagination is in memcache, you can use it even in your first page, if you are using AJAX (guessing, not that sure.)

Here is the strategy I use for caching responses. This could apply to other, non-appengine environments too.

About loading request/performance, there're lots of discussions that you can find in the groups, please just try google it. Here are some notes based on reading those. Latency causes by 1- time to start new JVM 2- time to load your application To reduce load time by 1) others star request to a) pay to reserve JVM b) request Google to load your app before start dispatch request to that instance c) accept the situation For 2) we try to a- try to use/replace frameworks with light-weight ones: datastore access framework, MVC framework,... b- try to limit calculation in your index page to alleviate the impact of loading request c- design your object model based on your need, so that you do calculation at insert time, not at query time. For example, in my app, if I want to report on year and quarter, then I have 5 summary "record" for those, instead of querying and computing those d- caching result. For example, If I know 1 one 5 piece of data above would be read frequently, then I will read those 5 all, and put into memcache for later use ... So it's application-specific, I don't know if each of above can help you. But only you who can know if which one of your code can be cached and how...