Group items tagged

Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet.

A query like this will use the merge join strategy by default. If it's too slow for the merge join strategy, you could have to build a custom index for it, which would indeed be 'exploding'  - each User entity would have len(friends)*len(likes) index entries.

Merge join queries - queries with multiple equality filters but no inequalities or sort orders - are a special case. They can be satisfied using the built in indexes and a merge join strategy, but they can also use a custom index. The production environment will use the index if it's present, and otherwise will do a merge join. Generally, a merge join works well, but there are situations in which it doesn't - in which case you may need to add an explicit index.

A query like this will use the merge join strategy by default. If it's too slow for the merge join strategy, you could have to build a custom index for it, which would indeed be 'exploding'  - each User entity would have len(friends)*len(likes) index entries.

The next problem that arises is that if the database has grown too big, then the csv generation will be slow(as it has to jump a lot of times). So the next idea is , instead of generating all cursors in the main cgi , the main cgi generates only first few set of page links, and then triggers a task queue entry to continue from there. By the time the user would have read the first page and clicks on "next page", the task would have generated few more entries. As the set of cursors for pagination is in memcache, you can use it even in your first page, if you are using AJAX (guessing, not that sure.)

In a relational world duplication is removed in order to prevent update anomalies. Error prevention is the driving force in relational modeling. Normalization is a kind of ethical system for data.

BigTable data ethics are more Mardi Gras than dinner with the in-laws. Data just wants to have fun. BigTable won’t stop you from hurting yourself. And to get the best results you may have to engage in some conventionally risky behaviors.

So the summary is: do not store nulls into Lists, otherwise you will get into a problem in future.

A join model is a data model that models the relationship between two other models. For example, you might have Person entities and Group entities, and you want Persons to be in Groups and Groups to "have" Persons. The relationship between Persons and Groups is one of Memberships. A Person is "in" a Group "through" a Membership. They may belong to multiple Groups, i.e. have multiple Memberships.

class Membership(db.Model): person = db.ReferenceProperty(Person) group = db.ReferenceProperty(Group)

If certain properties of your joined entities (Person and Group in this example) don't change much, but get queried often via the join model, you may find it worth caching those properties on the join model itself.

Just my 2 cents: I was "forced" to use an "unowned relationship" for my application because I had very large objects: My parent object "Journey" contained thousands of "JourneyPoints" in a List element. A call to "makePersistent" constantly hit the 30s deadline. By modeling it as an unowned relationship and distributing the parent key by hand to the JourneyPoints everything was fine again and the performance was very good.