Group items tagged

The remote_api module consists of two parts: A 'handler', which you install on the server to handle remote datastore requests, and a 'stub', which you set up on the client to translate datastore requests into calls to the remote handler. remote_api works at the lowest level of the datastore, so once you've set up the stub, you don't have to worry about the fact that you're operating on a remote datastore: With a few caveats, it works exactly the same as if you were accessing the datastore directly.

Note that the handler specifies "login: admin". This is extremely important, since we don't want to give just anyone unfettered access to our datastore!

Since you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, since a batch operation is only considered to be a single Datastore operation

to write large batch processing jobs without having to think about the plumbing details.

it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel

What would you have to build for yourself if Mapper weren’t available? Begin querying over every Entity in chained Task Queues Store beginning and end cursors (introduced in 1.3.5) Create tasks to work with chunks of your datastore Write the code to manipulate your data Build an interface to control your batch jobs Build a callback system for your multitudes of parallelized workers to call when the entire task has completed It’s certainly not a trivial amount of work

All you have to do is set the keys when you allocate the entities: Entity entity = new Entity(key); Or if you had previously pulled the entities from the datastore, the keys should already be set.

'm guessing your tests were run locally because the counter in the local datastore does indeed have datastore scope.  The scope of the counter in the prod datastore, however, is parent key + kind.  This is described here: http://code.google.com/appengine/docs/java/datastore/creatinggettinga... If it's important to you that the scope of the generated ids match between dev and and prod please file an issue.

                StringBuffer sb = new StringBuffer();                 KeyRange range = ds.allocateIds("a", 2);                 for (Key key : range) {                         sb.append("\n a " + key.toString());                 }                 range = ds.allocateIds("b", 2);                 for (Key key : range) {                         sb.append("\n b " + key.toString());                 }                 Key parentKey = KeyFactory.createKey("c", 1);                 sb.append("\n c " + parentKey.toString());                 range = ds.allocateIds(parentKey, "d", 2);                 for (Key key : range) {                         sb.append("\n d " + key.toString());                 }                 System.out.println(sb.toString());

The URL I posted earlier in the thread explains it, but here's a little bit more detail: A parent entity plus a kind defines an id-space, so entities with the same parent and the same kind are guaranteed to have unique ids.  For example, if you have an Entity with Parent:A Kind: Person Id: 10 you are guaranteed that no other entity with Parent A and Kind Person will be assigned an Id of 10.  However, an entity with a different Parent and Kind Person or an entity with Parent A and a different Kind _can_ be assigned an Id of 10.  The datastore pre-allocates batches of ids across multiple servers under-the-hood, so you can't make any assumptions about the Id that will get assigned in terms of contiguousness.  The only safe assumption is that the id will be unique for that Parent/Kind combination.

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

It would need to be something like Transaction txn = dataStore.beginTransaction(); Key key = dataStore.put(txn, question); Entity a1 = new Entity("Answer", key); Entity a2 = new Entity("Answer", key); dataStore.put(txn, a1); dataStore.put(txn, a2); txn.commit();