Group items tagged

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

A join model is a data model that models the relationship between two other models. For example, you might have Person entities and Group entities, and you want Persons to be in Groups and Groups to "have" Persons. The relationship between Persons and Groups is one of Memberships. A Person is "in" a Group "through" a Membership. They may belong to multiple Groups, i.e. have multiple Memberships.

class Membership(db.Model): person = db.ReferenceProperty(Person) group = db.ReferenceProperty(Group)

If certain properties of your joined entities (Person and Group in this example) don't change much, but get queried often via the join model, you may find it worth caching those properties on the join model itself.

You could create an entity group for this. Post:    data    category    sort UserPost:    user

if you can form the key_name for post and userpost like this Key('Post', '<data>') Key('Post', '<data>', 'UserPost', '<user>') Then you can perform a key_only query for a userpost and using the keys parents perform a get to retrieve the relevant userposts Adding new users incrementally to a Post is very simple and light weight. Deleting a Post would also require you to delete the UserPost children. These being in an entity group would provide for performing transactions... generally you wouldn't need to use them anyway... maybe when performing a full delete.

Don't bother with GQL for key-based retrieval -- make a key object from the string: k = db.Key('aght52oobW1hZHIOCxIHTWVzc2FnZRiyAQw') and just db.get(k). If you insist on GQL, btw, that k -- a suitably constructed instance of db.Key, NOT a string object!-) -- is also what you need to substitute into the GQL query (by :1 or whatrever).

keys are SERIOUSLY unique -- from a key you can get the entity's .app() (so uniqueness across apps is a given), .kind() (so uniqueness WITHIN the app's no problem either), etc.

The remote_api module consists of two parts: A 'handler', which you install on the server to handle remote datastore requests, and a 'stub', which you set up on the client to translate datastore requests into calls to the remote handler. remote_api works at the lowest level of the datastore, so once you've set up the stub, you don't have to worry about the fact that you're operating on a remote datastore: With a few caveats, it works exactly the same as if you were accessing the datastore directly.

Note that the handler specifies "login: admin". This is extremely important, since we don't want to give just anyone unfettered access to our datastore!

Since you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, since a batch operation is only considered to be a single Datastore operation

to write large batch processing jobs without having to think about the plumbing details.

it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel

What would you have to build for yourself if Mapper weren’t available? Begin querying over every Entity in chained Task Queues Store beginning and end cursors (introduced in 1.3.5) Create tasks to work with chunks of your datastore Write the code to manipulate your data Build an interface to control your batch jobs Build a callback system for your multitudes of parallelized workers to call when the entire task has completed It’s certainly not a trivial amount of work

Well, in AppEngine your primary key is the key for the object(Entity) and it can either be a generated ID or a unique string your application provides.  "Google App Engine" by Dan Sanderson has a lot of examples on this.  If you don't use a key_name property, then when you save with a put(), an ID is generated, but if you pass in key_name='thisIsMyKey' into the constructor, then you manually set the key for the object.  You can use the method, id_or_name() to return either the object's key name or its ID, which ever one it has and has_id_or_name() would return a boolean about it, and if it's not saved and your not using key_name, then the ID would not exists yet.  Also you can get the Entity(Object) from the datastore by using the get(k) method where k is a key object and the key object has two parts: kind and ID or key_name.  Additionally you can fetch an object from the datastore with get_by_id() and get_by_key_name()

I propose two possiblities: Duplicate Line_Item. Line_Item appointment_key name price finished ... A Line_Item should have a finished state, when the item was finished or not by the employee. If an employee hadn't finished all line items, mark them as unfinished, create a new appointment and copy all items that were unfinished. You can index on the appointment_key field on all Line_Items, which is a Good Thing. However, the duplicated data may be a problem. Dynamic fields for Line_Item: Line_Item duplicate_key appointment_key name price finished ... Create a new field, duplicate_key, for Line_Item which points to another Line_Item or to null (reserve this key!). Null means that the Line_Item is original, any other value means that this Line_Item is a duplicate of the Line_Item the field points to. All fields of Line_Item marked as a duplicate inherit the fields of the original Line_Item, except the appointment_key: so it will take less storage. Also this solution should have appointment_key indexed, to speed up lookup times. This requires one additional query per duplicated Line_Item, which may be a problem. Now, it's a clear choice: either better speed or better storage. I would go for the first, as it reduces complexity of your model, and storage is never a problem with modern systems. Less complexity generally means less bugs and less development/testing costs, which justifies the cost of the storage requirement.

Have you considered doing both? Then you could quickly get a list of computers a student owns by key OR use a query which returns results in some sorted order. I don't think maintaining a list of keys on the student model is as intimidating as you think.

The best option depends on your requirements. Here's a few solutions (I'm assuming you're using Python, since you didn't specify): If you need to do transactional updates on an entire tree, and you're not going to have more than about 1QPS of sustained updates to any one tree, you can use the built in support for heirarchial storage. When creating an entity, you can pass the "parent" attribute to specify a parent entity or key, and when querying, you can use the .ancestor() method (or 'ANCESTOR IS' in GQL to retrieve all descendants of a given entity. If you don't need transactional updates, you can replicate the functionality of entity groups without the contention issues (and transaction safety): Add a db.ListProperty(db.Key) to your model called 'ancestors', and populate it with the list of ancestors of the object you're inserting. Then you can easily retrieve everything that's descended from a given ancestor with MyModel.all().filter('ancestors =', parent_key). If you don't need transactions, and you only care about retrieving the direct children of an entity (not all descendants), use the approach outlined above, but instead of a ListProperty just use a ReferenceProperty to the parent entity. This is known as an Adjacency List. There are other approaches available, but those three should cover the most common cases.

you don't get sessionDestroyed. I believe there's a couple of issues with notification of a destroyed session and the most significant one would be that there's no guarantee that an instance of your application will even be running (1.4.0 will allow reserved instances but that isn't out yet.) Other issues would be that since this is a distributed environment which instance should receive sessionDestroyed. GAE would have to implement this one their backend. I believe sessions currently are just created by the Servlet Context of an instance when necessary and that instance's sessionCreated is the one that is executed. You however can query the _ah_SESSION table to see if a given session is still active or has expired.