Group items tagged

One common pattern when using the Task Queue API is known as 'task chaining'. You execute a task on the task queue, and at some point, determine that you're going to need another task, either to complete the work the current task is doing, or to start doing something new.

The solution is straightforward: use task names. Task names ensure that the chained task is only enqueued once, even if the task doing the chaining gets re-executed repeatedly. There are many ways to derive a task name;

If you want to see this in practice, the bulkupdate library makes use of this pattern, constructing task names from the datastore ID of the bulkupdate job (to ensure uniqueness) and the number of the chained task (to ensure consistency).

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

A join model is a data model that models the relationship between two other models. For example, you might have Person entities and Group entities, and you want Persons to be in Groups and Groups to "have" Persons. The relationship between Persons and Groups is one of Memberships. A Person is "in" a Group "through" a Membership. They may belong to multiple Groups, i.e. have multiple Memberships.

class Membership(db.Model): person = db.ReferenceProperty(Person) group = db.ReferenceProperty(Group)

If certain properties of your joined entities (Person and Group in this example) don't change much, but get queried often via the join model, you may find it worth caching those properties on the join model itself.

class DeleteFull():     def execute(self):         deleting = model_class_name.all().order('__key__').fetch(100)         while deleting:             a = []             key = deleting[-1].key()             for item in deleting:                 a.append(item)             db.delete(a)             deleting = model_class_name.all().filter('__key__ >', key).order('__key__').fetch(100) This purged everything, but it took a hell of a long time.

we can build the start of an open source App Engine server clone that's suitable for many purposes:  initially just for regression testing & local development (like the "dev_appserver" that comes with the App Engine Python SDK), but perhaps in the future (once Hypertable/Hbase/etc are ready) a full stack to give to ISPs to let them run App Engine apps on their own.

Don't bother with GQL for key-based retrieval -- make a key object from the string: k = db.Key('aght52oobW1hZHIOCxIHTWVzc2FnZRiyAQw') and just db.get(k). If you insist on GQL, btw, that k -- a suitably constructed instance of db.Key, NOT a string object!-) -- is also what you need to substitute into the GQL query (by :1 or whatrever).

keys are SERIOUSLY unique -- from a key you can get the entity's .app() (so uniqueness across apps is a given), .kind() (so uniqueness WITHIN the app's no problem either), etc.

The remote_api module consists of two parts: A 'handler', which you install on the server to handle remote datastore requests, and a 'stub', which you set up on the client to translate datastore requests into calls to the remote handler. remote_api works at the lowest level of the datastore, so once you've set up the stub, you don't have to worry about the fact that you're operating on a remote datastore: With a few caveats, it works exactly the same as if you were accessing the datastore directly.

Note that the handler specifies "login: admin". This is extremely important, since we don't want to give just anyone unfettered access to our datastore!

Since you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, since a batch operation is only considered to be a single Datastore operation

 * Implements background tasks for * <a href="http://code.google.com/appengine/docs/java/overview.html">Google App * Engine for Java</a>, based on the * <a href="http://code.google.com/appengine/articles/deferred.html">Python 'deferred' * library</a>; simplifies use of the <a href="http://code.google.com/appengine/docs/java/taskqueue/overview.html"> * Task Queue Java API</a> by automatically handling the serialization and * deserializtion of complex task arguments.

You should never store a password in plain text. Use a ir-reversable data hashing algorithm, like sha or md5 Here is how you can create a hash in python: from hashlib import sha256from random import randomrandom_key = random()sha256('%s%s%s'%('YOUR SECRET KEY',random_key,password)) You should also store the random key and hash the user supplied password similarly.