Group items tagged

I figure it is going to take me at least four iterations to get this right. The first one will be building a GWT application with a simple UI that has no server logic behind it (just to learn how layout in GWT works). Step two will be adding a fake servlet backend (not app engine, just in memory). While not exactly App Engine yet, I should have a completely specified client-server API by the end of this process that I can subsequently implement on App Engine (iteration 3). Iteration four will handle deployment, CSS and whatever I may screw up in iterations one and two. I will log my notes of things I run into while I code.

Task Queue is defined as a mechanism to synchronously distribute a sequence of tasks among parallel threads of execution. The global problem is broken down into tasks and the tasks are enqueued onto the queue. Parallel threads of execution pull tasks from the queue and perform computations on the tasks. The runtime system is responsible for managing thread accesses to the tasks in the queue as well as ensuring proper queue usage (i.e. dequeuing from an empty queue should not be allowed).

GAE Task Queue provides a push model. Instead of having an arbitrary number of worker processes constantly polling for available tasks, GAE Task Queue instead pushes work to workers when tasks are available. This work is then processed by the existing auto-scaling GAE infrastructure, allowing you to not have to worry about scaling up and down workers. You simply define the maximum rates of processing and GAE takes care of farming out the tasks to workers appropriately.

What is also compelling about GAE Task Queue is its use of web hooks as the description of a task unit. When you break it down, an individual task consists of the code to execute for that task as well as the individual data input for that specific task.

You can study the JavaMail API in more detail to understand how to do things like sending attachments in your email too. You can do that using the Google App Engine Email Service.

This project is a framework for storing entities larger than 1MB using the Google App Engine low-level datastore API.

 Twig is an alternative to the standard   persistence interfaces JDO and JPA, specifically designed to make the   most of the underlying datastore's unique features.

Twig is the only interface to support direct unowned relationships so   your code is not dependent on low-level datastore classes.  It is the   only interface to support OR queries by merging together multiple   queries, sorting them and filtering out duplicates at the lowest level   for performance.

Async datastore calls are not yet part of the low-level API.  Twig   uses the underlying Protocol Buffer layer classes to call   ApiProxy.makeAsyncCall() instead of makeSyncCall.  All the code is   open source so you can check out how its done.

> I'm still curious where does "method" name come from? That is just "Get" of "Put" or "RunQuery" etc.    I have also checked   in an implementation of Nick Johnsons ApiProxyHook which logs all this   info: LoggingApiProxyDelegate.  Its handy for seeing how what rpc   calls are being made.  If you want to discuss Twig specifics probably   best to do that here http://groups.google.com/group/twig-persist

Thanks to the Task Queue API released in SDK 1.2.3, it's easier than ever to do work 'offline', separate from user serving requests. In some cases, however, setting up a handler for each distinct task you want to run can be cumbersome, as can serializing and deserializing complex arguments for the task - particularly if you have many diverse but small tasks that you want to run on the queue. Fortunately, a new library in release 1.2.5 of the SDK makes these ad-hoc tasks much easier to write and execute. This library is found in google.appengine.ext.deferred, and from here on in we'll refer to it as the 'deferred' library. The deferred library lets you bypass all the work of setting up dedicated task handlers and serializing and deserializing your parameters by exposing a simple function, deferred.defer().

To demonstrate how powerful the deferred library can be, we're going to reprise an example from the remote_api article - the Mapper class. Like the example in the remote_api article, this class will make it easy to iterate over a large set of entities, making changes or calculating totals. Unlike the remote_api version, though, our version won't require an external computer to run it on, and it'll be more efficient to boot!

Task Queue items are limited to 10kb of associated data. This means that when the deferred library serializes the details of your call, it must amount to less than 10 kilobytes in order to fit on the Task Queue directly. No need to panic, though: If you try to enqueue a task that is too big to fit on the queue by itself, the deferred library will automatically create a new Entity in the datastore to hold information about the task, and will delete the entity once the task has been run. This means that in practice, your function call can be up to 1MB once serialized.

All you can do is fetch the entities of the type you want from the datastore and delete them one by one. (call datastore.delete(Iterable<Entity>))This way you can delete about 1000 entities at each request... There is no way to delete all entities of a Kind by a single api call.

> > If there is a way to list current Kinds or select all entities of any > > kind I could manage this.

A Java app sets up queues using a configuration file named queue.xml, in the WEB-INF/ directory inside the WAR. See Java Task Queue Configuration. If an app does not have a queue.xml file, it has a queue named default with some default settings. To enqueue a task, you get a Queue using the QueueFactory, then call its add() method. You can get a named queue specified in the queue.xml file using the getQueue() method of the factory, or you can get the default queue using getDefaultQueue(). You can call the Queue's add() method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.

Although a queue defines a general FIFO ordering, tasks are not executed entirely serially. Multiple tasks from a single queue may be executed simultaneously by the scheduler, so the usual locking and transaction semantics need to be observed for any work performed by a task.

 * Implements background tasks for * <a href="http://code.google.com/appengine/docs/java/overview.html">Google App * Engine for Java</a>, based on the * <a href="http://code.google.com/appengine/articles/deferred.html">Python 'deferred' * library</a>; simplifies use of the <a href="http://code.google.com/appengine/docs/java/taskqueue/overview.html"> * Task Queue Java API</a> by automatically handling the serialization and * deserializtion of complex task arguments.

Personally, I would recommend using the blobstore API for this since it is officially supported by google and means it won't break in the future. It also has nice bindings into the imageservice. http://code.google.com/appengine/docs/java/blobstore/overview.html