Group items tagged

Thanks to the Task Queue API released in SDK 1.2.3, it's easier than ever to do work 'offline', separate from user serving requests. In some cases, however, setting up a handler for each distinct task you want to run can be cumbersome, as can serializing and deserializing complex arguments for the task - particularly if you have many diverse but small tasks that you want to run on the queue. Fortunately, a new library in release 1.2.5 of the SDK makes these ad-hoc tasks much easier to write and execute. This library is found in google.appengine.ext.deferred, and from here on in we'll refer to it as the 'deferred' library. The deferred library lets you bypass all the work of setting up dedicated task handlers and serializing and deserializing your parameters by exposing a simple function, deferred.defer().

To demonstrate how powerful the deferred library can be, we're going to reprise an example from the remote_api article - the Mapper class. Like the example in the remote_api article, this class will make it easy to iterate over a large set of entities, making changes or calculating totals. Unlike the remote_api version, though, our version won't require an external computer to run it on, and it'll be more efficient to boot!

Task Queue items are limited to 10kb of associated data. This means that when the deferred library serializes the details of your call, it must amount to less than 10 kilobytes in order to fit on the Task Queue directly. No need to panic, though: If you try to enqueue a task that is too big to fit on the queue by itself, the deferred library will automatically create a new Entity in the datastore to hold information about the task, and will delete the entity once the task has been run. This means that in practice, your function call can be up to 1MB once serialized.

this tip can cut a few hundred milliseconds or more off of your cold start time. Using JSPs slows you down in two ways: If you have a file with a .jsp extension, anywhere in your website directory, appcfg will detect that you use JSP, and it will add 8 libraries that are used for processing JSPs into your lib directory before uploading to the app engine.  These libraries total two megabytes in size, and simply having them in your lib directory slows your cold starts down by a few hundred milliseconds. If you are interested in what libraries these are, you can see them in the temporary folder appcfg creates while uploading your app. The first JSP file accessed after a cold start, even an empty one, will take a few hundred additional milliseconds to be processed.  I'm not sure what causes this, maybe it has to do with initializing the JSP processor.

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

Cold-start latency is determined by the time it takes to load all the classes needed to handle the request. If you upload a JAR file, but nothing references it, it won't be loaded, and thus won't affect your cold-start latency.

Only those jars under WEB-INF/lib will be uploaded to GAE. You can prevent GWT jars from being uploaded by not placing them under WEB-INF/lib, rather by externally linking to them in your project build path.

Originally my application started out using Spring MVC and JDO. The first use of JDO took my application about 5 seconds to get everything set up.  Similarily, Spring MVC added around 6 seconds to the cold start  time

 If you are not using either JDO or JPA, you can safely delete all related libraries (ones that have JDO, JPA, or Datanucleus in their name) and use the command line tool to upload your app. Deleting these libraries shaves about 400ms off of cold start time.

Have you tried these? for server-side (GAE) *com.google.appengine.repackaged.org.json.JSONObject.JSONObject(String arg0)* for client *com.google.gwt.json.client.JSONParser* I haven't tried japanese or chinese characters though, rather stuff like this: "Příliš žluťoučký kuň úpěl ďábělské ódy"

import_transform A single-argument function that returns the correct value and type data based on the external_name or import_template strings. Examples include the built-in Python conversion operators (such as float), any of several helper functions provided in transform, such as get_date_time or generate_foreign_key, a function provided in your own library, or an in-line lambda function. Or, a two-argument function with the keyword argument bulkload_state, which on return contains useful information about the entity: bulkload_state.current_entity, which is the current entity being processed; bulkload_state.current_dictionary, the current export dictionary, and bulkload_state.filename, the --filename argument that was passed to appcfg.py.

import_template Specifies multiple dictionary items for a single property, using Python string interpolation.

One common pattern when using the Task Queue API is known as 'task chaining'. You execute a task on the task queue, and at some point, determine that you're going to need another task, either to complete the work the current task is doing, or to start doing something new.

The solution is straightforward: use task names. Task names ensure that the chained task is only enqueued once, even if the task doing the chaining gets re-executed repeatedly. There are many ways to derive a task name;

If you want to see this in practice, the bulkupdate library makes use of this pattern, constructing task names from the datastore ID of the bulkupdate job (to ensure uniqueness) and the number of the chained task (to ensure consistency).

GaeVFS is an Apache Commons VFS plug-in that implements a distributed, writeable virtual file system for Google App Engine (GAE) for Java. GaeVFS is implemented using the GAE datastore and memcache APIs. The primary goal of GaeVFS is to provide a portability layer that allows you to write application code to access the file system--both reads and writes--that runs unmodified in either GAE or non-GAE servlet environments.

 Twig is an alternative to the standard   persistence interfaces JDO and JPA, specifically designed to make the   most of the underlying datastore's unique features.

Twig is the only interface to support direct unowned relationships so   your code is not dependent on low-level datastore classes.  It is the   only interface to support OR queries by merging together multiple   queries, sorting them and filtering out duplicates at the lowest level   for performance.

Async datastore calls are not yet part of the low-level API.  Twig   uses the underlying Protocol Buffer layer classes to call   ApiProxy.makeAsyncCall() instead of makeSyncCall.  All the code is   open source so you can check out how its done.

3. Adds a new gae:start goal. This is identical to gae:run, except that it does not automatically execute the package phase before starting the server. The intent is to use this in a project POM, but it can also be used to start the server quickly when you don't want to rebuild the project. Right now I just copied the RunGoal completely, except for the '@execute phase="package"' declaration, because I didn't want to introduce conflicts in case someone else is editing RunGoal.java. A better thing to do in the future would be to have RunGoal extend StartGoal with an empty class that just adds the '@execute phase="package"' annotation (it can't be the other way around, with StartGoal extending Run goal, because there's no way to override a mojo with an @execute annotation with one that doesn't have it---it gets inherited automatically).

The idea behind all of these changes is to make it so that you can use the maven-gae-plugin in a project's POM for automated integration tests. The gae:start goal automatically binds to the pre-integration-test phase by default, and the gae:stop goal binds to post-integration-test. This is most useful when combined with something like the maven-failsafe-plugin and a functional testing library like HtmlUnit that you can use to simulate web requests. It would also be useful for Selenium testing.