Group items tagged

Here is the approach I would probably take unless you have many hundreds of child tasks. I would keep a list of keys of all children of each project/task. (You can then do a db.get and get all immediate children with single call) and each child has a reference to it's parent. In addition in each item keep a list of all parents keys back to the root. That way you can search for all children in part of the tree/subtree (This approach is especially useful if you have multiple child types) as back ref sets are kind specific. Keep a  running count of complete children at each level, rather than trying to calculate the current state in a single big function,  (keep these subsidiary values up to date through tasks)

One common pattern when using the Task Queue API is known as 'task chaining'. You execute a task on the task queue, and at some point, determine that you're going to need another task, either to complete the work the current task is doing, or to start doing something new.

The solution is straightforward: use task names. Task names ensure that the chained task is only enqueued once, even if the task doing the chaining gets re-executed repeatedly. There are many ways to derive a task name;

If you want to see this in practice, the bulkupdate library makes use of this pattern, constructing task names from the datastore ID of the bulkupdate job (to ensure uniqueness) and the number of the chained task (to ensure consistency).

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

The property map consists of a set of 'property' entries, each of which specifies how to handle a particular property of the model on import and on export. for our Permission kind, the bulkloader has identified 4 properties, plus the __key__ pseudo-property. Each has an 'external_name', and optional import and export transforms, which specify how to translate between the App Engine datastore representation and an external representation.

All we had to do here was to remove some of the boilerplate and the extraneous invite_nonce entry, and fill in the kind names for the two reference properties, and we're sorted.

we didn't have to write a single line of Python code, or set up an app.yaml, or anything else Python-specific in order to achieve it! Further, the bulkloader took care of generating a mostly-finished configuration file for us, in a format that ensures the data we download can be re-uploaded again without loss of fidelity.

As I understand it, the process of performing a single fetch (call to get())  from the dastastore using a key basically involves finding the host housing the entity, opening a socket, fetching the data, and then cleaning up the connection.  So to fetch something like 30 entities from the datastore, you're repeating the process 30 times over in serial, each time incurring whatever overhead is involved.  I also read that if you perform bulk fetches, (ie passing multiple keys at once) you can eliminate a great deal of that overhead.  In one of the videos I watched from Google I/0 2009, the presenter (whose name I forget - d'oh) said that performing a bulk fetch actually performs the fetches in parallel from the data store and you shoudl see requests noticeably faster.

Queries involving keys use indexes just like queries involving properties. Queries on keys require custom indexes in the same cases as with properties, with a couple of exceptions: inequality filters or an ascending sort order on __key__ do not require a custom index, but a descending sort order on __key__ does. As with all queries, the development web server creates appropriate configuration entries in this file when a query that needs a custom index is tested.

> I'm still curious where does "method" name come from? That is just "Get" of "Put" or "RunQuery" etc.    I have also checked   in an implementation of Nick Johnsons ApiProxyHook which logs all this   info: LoggingApiProxyDelegate.  Its handy for seeing how what rpc   calls are being made.  If you want to discuss Twig specifics probably   best to do that here http://groups.google.com/group/twig-persist

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.

All you have to do is set the keys when you allocate the entities: Entity entity = new Entity(key); Or if you had previously pulled the entities from the datastore, the keys should already be set.

It sounds like the list property can't be appended to without a transaction?  If so, it seems it'll be better to create two tables, one for Users and one for Items.

Have you considered doing both? Then you could quickly get a list of computers a student owns by key OR use a query which returns results in some sorted order. I don't think maintaining a list of keys on the student model is as intimidating as you think.

That was referring to the bulkloader, which lets you do those things. App Engine's datastore isn't a relational database. We can't do a dump of all your data without iterating through all of your indexes, then retrieving your Entities.

You have three options at the moment, when it comes to a 'multi-tenant' app such as you describe: You can have a single app that your customers add to their domains. Your app will have a single datastore, but you can use the Host header to determine which customer is accessing the app, and segregate the datastore entries based on that. Easy to deploy and upgrade Easy for customers to install Users have to have Google accounts, not Apps accounts, to log in. You can deploy a fresh app instance for each customer. Harder to deploy and upgrade More customer involvement required to install Provides firm separation of data Users can log in with their Apps credentials You can work with Google to create a new Apps Marketplace app All the benefits of point 1 and 2, above Requires Google involvement No certain release date yet