Google AppEngine

A Java app sets up queues using a configuration file named queue.xml, in the WEB-INF/ directory inside the WAR. See Java Task Queue Configuration. If an app does not have a queue.xml file, it has a queue named default with some default settings. To enqueue a task, you get a Queue using the QueueFactory, then call its add() method. You can get a named queue specified in the queue.xml file using the getQueue() method of the factory, or you can get the default queue using getDefaultQueue(). You can call the Queue's add() method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.

Although a queue defines a general FIFO ordering, tasks are not executed entirely serially. Multiple tasks from a single queue may be executed simultaneously by the scheduler, so the usual locking and transaction semantics need to be observed for any work performed by a task.

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.

 * Implements background tasks for * <a href="http://code.google.com/appengine/docs/java/overview.html">Google App * Engine for Java</a>, based on the * <a href="http://code.google.com/appengine/articles/deferred.html">Python 'deferred' * library</a>; simplifies use of the <a href="http://code.google.com/appengine/docs/java/taskqueue/overview.html"> * Task Queue Java API</a> by automatically handling the serialization and * deserializtion of complex task arguments.

You can't in a query, the indexes only index values that exist,  you would have to explicitly set properties to None or some other sentinal value by default to find objects that haven't had a property set.

> How does one test for the presence or absence of a particular property > in an Entity?

protocol

ublic void doGet(HttpServletRequest req, HttpServletResponse resp)     throws IOException {    resp.setContentType("text/plain");    final DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();    final Query query = new Query("Table/Entity Name");    //query.addSort(Entity.KEY_RESERVED_PROPERTY, Query.SortDirection.DESCENDING);    for (final Entity entity : datastore.prepare(query).asIterable()) {        resp.getWriter().println(entity.getKey().toString());        final Map<String, Object> properties = entity.getProperties();        final String[] propertyNames = properties.keySet().toArray(            new String[properties.size()]);        for(final String propertyName : propertyNames) {            resp.getWriter().println("-> " + propertyName + ": " + entity.getProperty(propertyName));        }    }}

we can build the start of an open source App Engine server clone that's suitable for many purposes:  initially just for regression testing & local development (like the "dev_appserver" that comes with the App Engine Python SDK), but perhaps in the future (once Hypertable/Hbase/etc are ready) a full stack to give to ISPs to let them run App Engine apps on their own.

This project is a framework for storing entities larger than 1MB using the Google App Engine low-level datastore API.

How about having task 1 queue task 2 and task 2 queuing task 3?

> The use case is that I have 3 tasks that I want to insert into the > queue. Each task takes about 20 seconds to complete.  The challenge is > that task # 3 depends on task # 2, and task # 2 depends on task # 1. > The question is, how can I insert those 3 tasks and expect task 1 be > finished before task 2 starts, and task 2 be finished before task 3 > starts?

've been able to accomplish what you're doing with Selenium testing. If you're using GWT, then your integration testing and user acceptance probably won't be that far from each other.

> My question is this. What's the best way to use LocalServiceTestHelper > so that I can use my app like it has a persistent store? There is no > 'setUp' and 'tearDown' hooks like a JUnit test and the app runs in a > separate process within the IDE. (And in any case, I'm looking to do > integration testing where I want the state to be consistent across a > number of page requests.)

Another way of solving the problem is to have a Horse object with a "farmKey" field like so: class Horse {  @PrimaryKey  Key mKey  Key farmKey } Then to get horses on a farm you'd do a query on Horse.kind with farmKey equal to your farm key.  (Keys only query would make this faster) What's nice about this approach is that you don't need Horse key at all, just create a Horse object, give it the farm key, and persist it without worrying about it.

> > so a Farm can have some horses. I'd like to create a new horse, then > > put it on a farm, in one transaction.

> This requires Horse to be a descendant of Farm to be in the same   > entity group.  Therefore a horse could not move farms unless you   > delete and recreate it with a new Key.

 Twig is an alternative to the standard   persistence interfaces JDO and JPA, specifically designed to make the   most of the underlying datastore's unique features.

Twig is the only interface to support direct unowned relationships so   your code is not dependent on low-level datastore classes.  It is the   only interface to support OR queries by merging together multiple   queries, sorting them and filtering out duplicates at the lowest level   for performance.

Async datastore calls are not yet part of the low-level API.  Twig   uses the underlying Protocol Buffer layer classes to call   ApiProxy.makeAsyncCall() instead of makeSyncCall.  All the code is   open source so you can check out how its done.

> I'm still curious where does "method" name come from? That is just "Get" of "Put" or "RunQuery" etc.    I have also checked   in an implementation of Nick Johnsons ApiProxyHook which logs all this   info: LoggingApiProxyDelegate.  Its handy for seeing how what rpc   calls are being made.  If you want to discuss Twig specifics probably   best to do that here http://groups.google.com/group/twig-persist

Here is the strategy I use for caching responses. This could apply to other, non-appengine environments too.

You should get prompted for authentication. Make sure remote_api is installed correctly on your server by visiting whatever you used for the --url argument in your browser. It should require you to log in then say "This request did not contain a necessary header"

Authentication with App Engine, regardless of where you're doing it, is a three-stage process: Obtain an authentication token. This can be done with ClientLogin for installed apps, for example, or with AuthSub for a webapp. When logging in directly to an application, this is the part of the login process where your user sees a Google signin screen. Take that authentication token, and use it to obtain an authentication cookie. Use that authentication cookie in all subsequent requests.