Group items tagged

A Java app sets up queues using a configuration file named queue.xml, in the WEB-INF/ directory inside the WAR. See Java Task Queue Configuration. If an app does not have a queue.xml file, it has a queue named default with some default settings. To enqueue a task, you get a Queue using the QueueFactory, then call its add() method. You can get a named queue specified in the queue.xml file using the getQueue() method of the factory, or you can get the default queue using getDefaultQueue(). You can call the Queue's add() method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.

Although a queue defines a general FIFO ordering, tasks are not executed entirely serially. Multiple tasks from a single queue may be executed simultaneously by the scheduler, so the usual locking and transaction semantics need to be observed for any work performed by a task.

Task Queue is defined as a mechanism to synchronously distribute a sequence of tasks among parallel threads of execution. The global problem is broken down into tasks and the tasks are enqueued onto the queue. Parallel threads of execution pull tasks from the queue and perform computations on the tasks. The runtime system is responsible for managing thread accesses to the tasks in the queue as well as ensuring proper queue usage (i.e. dequeuing from an empty queue should not be allowed).

GAE Task Queue provides a push model. Instead of having an arbitrary number of worker processes constantly polling for available tasks, GAE Task Queue instead pushes work to workers when tasks are available. This work is then processed by the existing auto-scaling GAE infrastructure, allowing you to not have to worry about scaling up and down workers. You simply define the maximum rates of processing and GAE takes care of farming out the tasks to workers appropriately.

What is also compelling about GAE Task Queue is its use of web hooks as the description of a task unit. When you break it down, an individual task consists of the code to execute for that task as well as the individual data input for that specific task.

Thanks to the Task Queue API released in SDK 1.2.3, it's easier than ever to do work 'offline', separate from user serving requests. In some cases, however, setting up a handler for each distinct task you want to run can be cumbersome, as can serializing and deserializing complex arguments for the task - particularly if you have many diverse but small tasks that you want to run on the queue. Fortunately, a new library in release 1.2.5 of the SDK makes these ad-hoc tasks much easier to write and execute. This library is found in google.appengine.ext.deferred, and from here on in we'll refer to it as the 'deferred' library. The deferred library lets you bypass all the work of setting up dedicated task handlers and serializing and deserializing your parameters by exposing a simple function, deferred.defer().

To demonstrate how powerful the deferred library can be, we're going to reprise an example from the remote_api article - the Mapper class. Like the example in the remote_api article, this class will make it easy to iterate over a large set of entities, making changes or calculating totals. Unlike the remote_api version, though, our version won't require an external computer to run it on, and it'll be more efficient to boot!

Task Queue items are limited to 10kb of associated data. This means that when the deferred library serializes the details of your call, it must amount to less than 10 kilobytes in order to fit on the Task Queue directly. No need to panic, though: If you try to enqueue a task that is too big to fit on the queue by itself, the deferred library will automatically create a new Entity in the datastore to hold information about the task, and will delete the entity once the task has been run. This means that in practice, your function call can be up to 1MB once serialized.

One common pattern when using the Task Queue API is known as 'task chaining'. You execute a task on the task queue, and at some point, determine that you're going to need another task, either to complete the work the current task is doing, or to start doing something new.

The solution is straightforward: use task names. Task names ensure that the chained task is only enqueued once, even if the task doing the chaining gets re-executed repeatedly. There are many ways to derive a task name;

If you want to see this in practice, the bulkupdate library makes use of this pattern, constructing task names from the datastore ID of the bulkupdate job (to ensure uniqueness) and the number of the chained task (to ensure consistency).

With the Task Queue API, applications can perform work outside of a user request but initiated by a user request. If an app needs to execute some background work, it may use the Task Queue API to organize that work into small, discrete units, called Tasks. The app then inserts these Tasks into one or more Queues. App Engine automatically detects new Tasks and executes them when system resources permit.

Yes, worker would be a servlet which can handle a request with POST parameters. If you want an asynchronous call from client's point of view then RPC is enough (from server's point of view it is still synchronous). If you want to do "delayed" jobs which don't talk to your client, you can use a task queue.

Instead of trying to create a thread (which is impossible in App Engine), this is a great way to asynchronously run tasks.

Cron jobs / task queues Instead of using a framework like Quartz to schedule jobs, Google App Engine takes care of executing jobs for you. You simply enter a cron-like expression and a URL to call and your job is configured. You also have a task queue at your disposal. Your application code can add tasks to a task queue which will be executed later in the future, asynchronously. An example use case is that you don’t want clients to wait for an email to be sent before he sees the next page. Instead you can put the email task on the task queue and the email will be sent asynchronously.

When an email is received Google App Engine does a post on an URL in your application you configured. The HTTP body of the POST request contains the exact mime message as it was received by Google. To parse this mime message you can use the MimeMessage class provided by the JDK.

queue.add(     DatastoreServiceFactory.getDatastoreService().getCurrentTransaction(),     TaskOptions.Builder.url("/path/to/my/worker"));

How about having task 1 queue task 2 and task 2 queuing task 3?

> The use case is that I have 3 tasks that I want to insert into the > queue. Each task takes about 20 seconds to complete.  The challenge is > that task # 3 depends on task # 2, and task # 2 depends on task # 1. > The question is, how can I insert those 3 tasks and expect task 1 be > finished before task 2 starts, and task 2 be finished before task 3 > starts?

The next problem that arises is that if the database has grown too big, then the csv generation will be slow(as it has to jump a lot of times). So the next idea is , instead of generating all cursors in the main cgi , the main cgi generates only first few set of page links, and then triggers a task queue entry to continue from there. By the time the user would have read the first page and clicks on "next page", the task would have generated few more entries. As the set of cursors for pagination is in memcache, you can use it even in your first page, if you are using AJAX (guessing, not that sure.)

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.