Group items tagged

'm guessing your tests were run locally because the counter in the local datastore does indeed have datastore scope.  The scope of the counter in the prod datastore, however, is parent key + kind.  This is described here: http://code.google.com/appengine/docs/java/datastore/creatinggettinga... If it's important to you that the scope of the generated ids match between dev and and prod please file an issue.

                StringBuffer sb = new StringBuffer();                 KeyRange range = ds.allocateIds("a", 2);                 for (Key key : range) {                         sb.append("\n a " + key.toString());                 }                 range = ds.allocateIds("b", 2);                 for (Key key : range) {                         sb.append("\n b " + key.toString());                 }                 Key parentKey = KeyFactory.createKey("c", 1);                 sb.append("\n c " + parentKey.toString());                 range = ds.allocateIds(parentKey, "d", 2);                 for (Key key : range) {                         sb.append("\n d " + key.toString());                 }                 System.out.println(sb.toString());

The URL I posted earlier in the thread explains it, but here's a little bit more detail: A parent entity plus a kind defines an id-space, so entities with the same parent and the same kind are guaranteed to have unique ids.  For example, if you have an Entity with Parent:A Kind: Person Id: 10 you are guaranteed that no other entity with Parent A and Kind Person will be assigned an Id of 10.  However, an entity with a different Parent and Kind Person or an entity with Parent A and a different Kind _can_ be assigned an Id of 10.  The datastore pre-allocates batches of ids across multiple servers under-the-hood, so you can't make any assumptions about the Id that will get assigned in terms of contiguousness.  The only safe assumption is that the id will be unique for that Parent/Kind combination.

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

An Entity is an individual record that gets stored and retrieved from the datastore. The Kind is the unique string identifier of the type of entity. For example, "Joe" is an Entity with age=42, dob=10-12-2000, and Kind "Person".

Entity groups are a simple concept. Entities in App Engine have ancestors. For e.g. you could model Books and Authors. Author: name->X, and Author: name->Y are two entities in the Author KIND. Book is another KIND (KIND is just a logical grouping of entities). The relationship between Books and Authors can be modeled as a ancestor-child relationship. E.g. Book: name->B1, B2 could have been written by Author: name->X. So you modeled them as: Author: name->X is a parent of both Book: name->B1, B2. Similarly, Book: name->B3 is written by Author: name->Y and so could model that relationship as Author:name->Y is a parent of Book:name->B3. When you try to transact on Books kind, you cannot transact on B1,B3 and B3 together. As they participate in different ancestor-child relationships. Every ancestor child relationship is an Entity group. You can only "lock" on one entity group at a time. Hope this makes things a little clear.

to write large batch processing jobs without having to think about the plumbing details.

it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel

What would you have to build for yourself if Mapper weren’t available? Begin querying over every Entity in chained Task Queues Store beginning and end cursors (introduced in 1.3.5) Create tasks to work with chunks of your datastore Write the code to manipulate your data Build an interface to control your batch jobs Build a callback system for your multitudes of parallelized workers to call when the entire task has completed It’s certainly not a trivial amount of work

Every entity has a key that is unique over all entities in App Engine. A complete key includes several pieces of information, including the application ID, the kind, and an entity ID. (Keys also contain information about entity groups; see Transactions for more information.) An object's key is stored in a field on the instance. You identify the primary key field using the @PrimaryKey annotation. The app can provide the ID portion of the key as a string when the object is created, or it can allow the datastore to generate a numeric ID automatically. The complete key must be unique across all entities in the datastore. In other words, an object must have an ID that is unique across all objects of the same kind and with the same entity group parent (if any). You select the desired behavior of the key using the type of the field and annotations. If the class is used as a "child" class in a relationship, the key field must be of a type capable of representing an entity group parent: either a Key instance, or a Key value encoded as a string. See Transactions for more information on entity groups, and Relationships for more information on relationships. Tip: If the app creates a new object and gives it the same string ID as another object of the same kind (and the same entity group parent), saving the new object overwrites the other object in the datastore. To detect whether a string ID is already in use prior to creating a new object, you can use a transaction to attempt to get an entity with a given ID, then create one if it doesn't exist. See Transactions. There are 4 types of primary key fields:

All you can do is fetch the entities of the type you want from the datastore and delete them one by one. (call datastore.delete(Iterable<Entity>))This way you can delete about 1000 entities at each request... There is no way to delete all entities of a Kind by a single api call.

> > If there is a way to list current Kinds or select all entities of any > > kind I could manage this.

public final class Keyextends java.lang.Objectimplements java.io.Serializable, java.lang.Comparable<Key> The primary key for a datastore entity. A datastore GUID. A Key instance uniquely identifies an entity across all apps, and includes all information necessary to fetch the entity from the datastore with DatastoreService.get(Key). You can create Key objects directly by using KeyFactory.createKey(java.lang.String, long) or getChild(java.lang.String, long). You can also retrieve the Key automatically created when you create a new Entity, or serialize Key objects, or use KeyFactory to convert them to and from websafe String values.

equals public boolean equals(java.lang.Object object) Compares two Key objects by comparing ids, kinds, parent and appIdNamespace. If both keys are assigned names rather than ids, compares names instead of ids. If neither key has an id or a name, the keys are only equal if they reference the same object.

compareTo public int compareTo(Key other) Compares two Key objects. The algorithm proceeds as follows: Turn each Key into an iterator where the first element returned is the top-most ancestor, the next element is the child of the previous element, and so on. The last element will be the Key we started with. Once we have assembled these two iterators (one for 'this' and one for the Key we're comparing to), consume them in parallel, comparing the next element from each iterator. If at any point the comparison of these two elements yields a non-zero result, return that as the result of the overall comparison. If we exhaust the iterator built from 'this' before we exhaust the iterator built from the other Key, we return less than. An example: app1.type1.4.app1.type2.9 < app1.type1.4.app1.type2.9.app1.type3.2 If we exhaust the iterator built from the other Key before we exhaust the iterator built from 'this', we return greater than. An example: app1.type1.4.app1.type2.9.app1.type3.2 > app1.type1.4.app1.type2.9 The relationship between individual Key Keys is performed by comparing app followed by kind followed by id. If both keys are assigned names rather than ids, compares names instead of ids. If neither key has an id or a name we return an arbitrary but consistent result. Assuming all other components are equal, all ids are less than all names.

The property map consists of a set of 'property' entries, each of which specifies how to handle a particular property of the model on import and on export. for our Permission kind, the bulkloader has identified 4 properties, plus the __key__ pseudo-property. Each has an 'external_name', and optional import and export transforms, which specify how to translate between the App Engine datastore representation and an external representation.

All we had to do here was to remove some of the boilerplate and the extraneous invite_nonce entry, and fill in the kind names for the two reference properties, and we're sorted.

we didn't have to write a single line of Python code, or set up an app.yaml, or anything else Python-specific in order to achieve it! Further, the bulkloader took care of generating a mostly-finished configuration file for us, in a format that ensures the data we download can be re-uploaded again without loss of fidelity.

IDs aren't even unique within a kind; they're unique within a kind and entity group. The reason they're not sequential is that an instance is allocated a large block of IDs, and unused ones aren't assigned to later instances

Basically, all you have to do is create a subclass of bulkloader.Loader and implement (at a minimum) the generate_records method, which should yield lists of strings. This same strategy would work for loading data from XML files or ROT13-encrypted files or whatever. Note that the list of strings yielded by the generate_records method must match up (in length and order) with the "properties" list you provide when you initialize the loader (ie, the second argument to the AlbumLoader.__init__ method in this example). This approach actually provides a lot of flexibility: We're overriding the __init__ method on our JSONLoader implementation and automatically determining the kind of model we're loading and its list of properties to provide to the bulkloader.Loader parent class.

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

Entity is the fundamental unit of data storage. It has an immutable identifier (contained in the Key) object, a reference to an optional parent Entity, a kind (represented as an arbitrary string), and a set of zero or more typed properties.

setProperty public void setProperty(java.lang.String propertyName, java.lang.Object value) Sets the property named, propertyName, to value. As the value is stored in the datastore, it is converted to the datastore's native type. This may include widening, such as converting a Short to a Long. All Collections are prone to losing their sort order and their original types as they are stored in the datastore. For example, a TreeSet may be returned as a List from getProperty(java.lang.String), with an arbitrary re-ordering of elements. Overrides any existing value for this property, whether indexed or unindexed. Note that Blob and Text property values are never indexed by the built-in single property indexes. To store other types without being indexed, use #setUnindexedProperty. Parameters:value - may be one of the supported datatypes, a heterogenous Collection of one of the supported datatypes, or an UnindexedValue wrapping one of the supported datatypes. Throws: java.lang.IllegalArgumentException - If the value is not of a type that the data store supports.

Here is the approach I would probably take unless you have many hundreds of child tasks. I would keep a list of keys of all children of each project/task. (You can then do a db.get and get all immediate children with single call) and each child has a reference to it's parent. In addition in each item keep a list of all parents keys back to the root. That way you can search for all children in part of the tree/subtree (This approach is especially useful if you have multiple child types) as back ref sets are kind specific. Keep a  running count of complete children at each level, rather than trying to calculate the current state in a single big function,  (keep these subsidiary values up to date through tasks)

Tip: If the app creates a new object and gives it the same string ID as another object of the same kind (and the same entity group parent), saving the new object overwrites the other object in the datastore. To detect whether a string ID is already in use prior to creating a new object, you can use a transaction to attempt to get an entity with a given ID, then create one if it doesn't exist. See Transactions.

There are 4 types of primary key fields:

In a relational world duplication is removed in order to prevent update anomalies. Error prevention is the driving force in relational modeling. Normalization is a kind of ethical system for data.

BigTable data ethics are more Mardi Gras than dinner with the in-laws. Data just wants to have fun. BigTable won’t stop you from hurting yourself. And to get the best results you may have to engage in some conventionally risky behaviors.

Task Queue is defined as a mechanism to synchronously distribute a sequence of tasks among parallel threads of execution. The global problem is broken down into tasks and the tasks are enqueued onto the queue. Parallel threads of execution pull tasks from the queue and perform computations on the tasks. The runtime system is responsible for managing thread accesses to the tasks in the queue as well as ensuring proper queue usage (i.e. dequeuing from an empty queue should not be allowed).

GAE Task Queue provides a push model. Instead of having an arbitrary number of worker processes constantly polling for available tasks, GAE Task Queue instead pushes work to workers when tasks are available. This work is then processed by the existing auto-scaling GAE infrastructure, allowing you to not have to worry about scaling up and down workers. You simply define the maximum rates of processing and GAE takes care of farming out the tasks to workers appropriately.

What is also compelling about GAE Task Queue is its use of web hooks as the description of a task unit. When you break it down, an individual task consists of the code to execute for that task as well as the individual data input for that specific task.

Another way of solving the problem is to have a Horse object with a "farmKey" field like so: class Horse {  @PrimaryKey  Key mKey  Key farmKey } Then to get horses on a farm you'd do a query on Horse.kind with farmKey equal to your farm key.  (Keys only query would make this faster) What's nice about this approach is that you don't need Horse key at all, just create a Horse object, give it the farm key, and persist it without worrying about it.

> > so a Farm can have some horses. I'd like to create a new horse, then > > put it on a farm, in one transaction.

> This requires Horse to be a descendant of Farm to be in the same   > entity group.  Therefore a horse could not move farms unless you   > delete and recreate it with a new Key.

Don't bother with GQL for key-based retrieval -- make a key object from the string: k = db.Key('aght52oobW1hZHIOCxIHTWVzc2FnZRiyAQw') and just db.get(k). If you insist on GQL, btw, that k -- a suitably constructed instance of db.Key, NOT a string object!-) -- is also what you need to substitute into the GQL query (by :1 or whatrever).

keys are SERIOUSLY unique -- from a key you can get the entity's .app() (so uniqueness across apps is a given), .kind() (so uniqueness WITHIN the app's no problem either), etc.