Group items tagged

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

this won't be available until 1.3.6. You should be able to do something like this:  - property: __key__    external_name: CityId    export_transform: datastore.Key.id    import_transform: lambda value: datastore.Key.from_path('City', int(value))

From what I understand, you are "manually" copying the properties from the Entity to the DTO. But this process is automated in GWT 2.1 So, I am not sending my entities directly, but the proxies as per the documentation : http://code.google.com/webtoolkit/doc/latest/DevGuideRequestFactory.html "An entity proxy is a client-side representation of an entity, otherwise known as a DTO (Data Transfer Object). With RequestFactory, entity proxies are interfaces that extend the EntityProxy interface, which is the hook used to indicate that an object can be managed by RequestFactory. RequestFactory automatically populates bean-style properties between entities on the server and the corresponding EntityProxy on the client, which simplifies using the DTO pattern. Furthermore, the EntityProxy interface enables RequestFactory to compute and send only changes ("deltas") to the server." "Entity proxies simply extend the EntityProxy interface and use the @ProxyFor annotation to reference the server-side entity being represented. It is not necessary to represent every property and method from the server-side entity in the EntityProxy, only getters and setters for properties that should be exposed to the client." The entity proxies are merely interfaces that are being populated by the new GWT 2.1 RequestFactory framework. I have no control over this copying process. Per definition, getters/setters of the real entity are injected into the EntityProxy whenever they are present; So my problem still stands : what about complex values like com.google.appengine.api.datastore.Email that are not known by the client side code ? How to transfer these complex values to the client.

When I copy my entities onto my dtos, some entity fields don't even make it into the dto. Others only have getters in the dto because they are read-only to the client. Those that do get into the dto get converted to native Java types. For instance, Text gets converted to String. Key gets encoded to a url friendly string. If I have Set fields on the entities to manager my relations (property lists), I remap those to ArrayList... First, don't serialize interfaces to GWT client, you'll get Javascript bloat. Then, Hashmap is costly to serialize because String.hashCode() is not the same on in Java and in Javascript. Hence, all the items need to be re-inserted into a client side map. Of course, in Web mode, performance is good enough... but in development mode, your data transfers will become really slow for somewhat big transfers.

Every entity has a key that is unique over all entities in App Engine. A complete key includes several pieces of information, including the application ID, the kind, and an entity ID. (Keys also contain information about entity groups; see Transactions for more information.) An object's key is stored in a field on the instance. You identify the primary key field using the @PrimaryKey annotation. The app can provide the ID portion of the key as a string when the object is created, or it can allow the datastore to generate a numeric ID automatically. The complete key must be unique across all entities in the datastore. In other words, an object must have an ID that is unique across all objects of the same kind and with the same entity group parent (if any). You select the desired behavior of the key using the type of the field and annotations. If the class is used as a "child" class in a relationship, the key field must be of a type capable of representing an entity group parent: either a Key instance, or a Key value encoded as a string. See Transactions for more information on entity groups, and Relationships for more information on relationships. Tip: If the app creates a new object and gives it the same string ID as another object of the same kind (and the same entity group parent), saving the new object overwrites the other object in the datastore. To detect whether a string ID is already in use prior to creating a new object, you can use a transaction to attempt to get an entity with a given ID, then create one if it doesn't exist. See Transactions. There are 4 types of primary key fields:

Entity groups are a simple concept. Entities in App Engine have ancestors. For e.g. you could model Books and Authors. Author: name->X, and Author: name->Y are two entities in the Author KIND. Book is another KIND (KIND is just a logical grouping of entities). The relationship between Books and Authors can be modeled as a ancestor-child relationship. E.g. Book: name->B1, B2 could have been written by Author: name->X. So you modeled them as: Author: name->X is a parent of both Book: name->B1, B2. Similarly, Book: name->B3 is written by Author: name->Y and so could model that relationship as Author:name->Y is a parent of Book:name->B3. When you try to transact on Books kind, you cannot transact on B1,B3 and B3 together. As they participate in different ancestor-child relationships. Every ancestor child relationship is an Entity group. You can only "lock" on one entity group at a time. Hope this makes things a little clear.

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

you can only expect to update any single entity or entity group about five times a second. That is an estimate and the actual update rate for an entity is dependent on several attributes of the entity, including how many properties it has, how large it is, and how many indexes need updating. While a single entity or entity group has a limit on how quickly it can be updated, App Engine excels at handling many parallel requests distributed across distinct entities, and we can take advantage of this by using sharding.

'm guessing your tests were run locally because the counter in the local datastore does indeed have datastore scope.  The scope of the counter in the prod datastore, however, is parent key + kind.  This is described here: http://code.google.com/appengine/docs/java/datastore/creatinggettinga... If it's important to you that the scope of the generated ids match between dev and and prod please file an issue.

                StringBuffer sb = new StringBuffer();                 KeyRange range = ds.allocateIds("a", 2);                 for (Key key : range) {                         sb.append("\n a " + key.toString());                 }                 range = ds.allocateIds("b", 2);                 for (Key key : range) {                         sb.append("\n b " + key.toString());                 }                 Key parentKey = KeyFactory.createKey("c", 1);                 sb.append("\n c " + parentKey.toString());                 range = ds.allocateIds(parentKey, "d", 2);                 for (Key key : range) {                         sb.append("\n d " + key.toString());                 }                 System.out.println(sb.toString());

The URL I posted earlier in the thread explains it, but here's a little bit more detail: A parent entity plus a kind defines an id-space, so entities with the same parent and the same kind are guaranteed to have unique ids.  For example, if you have an Entity with Parent:A Kind: Person Id: 10 you are guaranteed that no other entity with Parent A and Kind Person will be assigned an Id of 10.  However, an entity with a different Parent and Kind Person or an entity with Parent A and a different Kind _can_ be assigned an Id of 10.  The datastore pre-allocates batches of ids across multiple servers under-the-hood, so you can't make any assumptions about the Id that will get assigned in terms of contiguousness.  The only safe assumption is that the id will be unique for that Parent/Kind combination.

protocol

ublic void doGet(HttpServletRequest req, HttpServletResponse resp)     throws IOException {    resp.setContentType("text/plain");    final DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();    final Query query = new Query("Table/Entity Name");    //query.addSort(Entity.KEY_RESERVED_PROPERTY, Query.SortDirection.DESCENDING);    for (final Entity entity : datastore.prepare(query).asIterable()) {        resp.getWriter().println(entity.getKey().toString());        final Map<String, Object> properties = entity.getProperties();        final String[] propertyNames = properties.keySet().toArray(            new String[properties.size()]);        for(final String propertyName : propertyNames) {            resp.getWriter().println("-> " + propertyName + ": " + entity.getProperty(propertyName));        }    }}

What do I mean by life cycle events? Events like entity creation, entity update and entity deletion. Mainstream ORM systems popularised callbacks like oncreate, onupdate, ondelete. Introducing such callbacks in the Java and Python APIs may be easy, but things get messy when you consider the ecosystem of alternative language implementations based on the Java API: developers using alternative languages would be forced to use Java to write the callbacks. There is a more robust solution though. Google App Engine already leverages the power of webhooks in such APIs as taskqueue, email, xmpp and more. Webhooks can elegantly solve the life cycle management problem as well: when an entity is created, updated or deleted through the Datastore viewer a corresponding webhook is triggered. Let's say the user is playing with Article entities, the webhooks uris could be: http://myapp.com/_ah/admin/datastore/le/Article/create/{key} http://myapp.com/_ah/admin/datastore/le/Article/update/{key} http://myapp.com/_ah/admin/datastore/le/Article/delete/{key} Slightly more work than callbacks, but still simple and effective. If there is an even better solution, I would love to hear about it in the comments section.

Entity is the fundamental unit of data storage. It has an immutable identifier (contained in the Key) object, a reference to an optional parent Entity, a kind (represented as an arbitrary string), and a set of zero or more typed properties.

setProperty public void setProperty(java.lang.String propertyName, java.lang.Object value) Sets the property named, propertyName, to value. As the value is stored in the datastore, it is converted to the datastore's native type. This may include widening, such as converting a Short to a Long. All Collections are prone to losing their sort order and their original types as they are stored in the datastore. For example, a TreeSet may be returned as a List from getProperty(java.lang.String), with an arbitrary re-ordering of elements. Overrides any existing value for this property, whether indexed or unindexed. Note that Blob and Text property values are never indexed by the built-in single property indexes. To store other types without being indexed, use #setUnindexedProperty. Parameters:value - may be one of the supported datatypes, a heterogenous Collection of one of the supported datatypes, or an UnindexedValue wrapping one of the supported datatypes. Throws: java.lang.IllegalArgumentException - If the value is not of a type that the data store supports.

If using the datastore API directly, a common pattern of usage is: // Get a handle on the datastore itself DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(); // Lookup data by known key name Entity userEntity = datastore.get(KeyFactory.createKey("UserInfo", email)); // Or perform a query Query query = new Query("Task", userEntity); query.addFilter("dueDate", Query.FilterOperator.LESS_THAN, today); for (Entity taskEntity : datastore.prepare(query).asIterable()) { if ("done".equals(taskEntity.getProperty("status"))) { datastore.delete(taskEntity); } else { taskEntity.setProperty("status", "overdue"); datastore.put(taskEntity); } }

An Entity is an individual record that gets stored and retrieved from the datastore. The Kind is the unique string identifier of the type of entity. For example, "Joe" is an Entity with age=42, dob=10-12-2000, and Kind "Person".

The best option depends on your requirements. Here's a few solutions (I'm assuming you're using Python, since you didn't specify): If you need to do transactional updates on an entire tree, and you're not going to have more than about 1QPS of sustained updates to any one tree, you can use the built in support for heirarchial storage. When creating an entity, you can pass the "parent" attribute to specify a parent entity or key, and when querying, you can use the .ancestor() method (or 'ANCESTOR IS' in GQL to retrieve all descendants of a given entity. If you don't need transactional updates, you can replicate the functionality of entity groups without the contention issues (and transaction safety): Add a db.ListProperty(db.Key) to your model called 'ancestors', and populate it with the list of ancestors of the object you're inserting. Then you can easily retrieve everything that's descended from a given ancestor with MyModel.all().filter('ancestors =', parent_key). If you don't need transactions, and you only care about retrieving the direct children of an entity (not all descendants), use the approach outlined above, but instead of a ListProperty just use a ReferenceProperty to the parent entity. This is known as an Adjacency List. There are other approaches available, but those three should cover the most common cases.

public final class Keyextends java.lang.Objectimplements java.io.Serializable, java.lang.Comparable<Key> The primary key for a datastore entity. A datastore GUID. A Key instance uniquely identifies an entity across all apps, and includes all information necessary to fetch the entity from the datastore with DatastoreService.get(Key). You can create Key objects directly by using KeyFactory.createKey(java.lang.String, long) or getChild(java.lang.String, long). You can also retrieve the Key automatically created when you create a new Entity, or serialize Key objects, or use KeyFactory to convert them to and from websafe String values.

equals public boolean equals(java.lang.Object object) Compares two Key objects by comparing ids, kinds, parent and appIdNamespace. If both keys are assigned names rather than ids, compares names instead of ids. If neither key has an id or a name, the keys are only equal if they reference the same object.

compareTo public int compareTo(Key other) Compares two Key objects. The algorithm proceeds as follows: Turn each Key into an iterator where the first element returned is the top-most ancestor, the next element is the child of the previous element, and so on. The last element will be the Key we started with. Once we have assembled these two iterators (one for 'this' and one for the Key we're comparing to), consume them in parallel, comparing the next element from each iterator. If at any point the comparison of these two elements yields a non-zero result, return that as the result of the overall comparison. If we exhaust the iterator built from 'this' before we exhaust the iterator built from the other Key, we return less than. An example: app1.type1.4.app1.type2.9 < app1.type1.4.app1.type2.9.app1.type3.2 If we exhaust the iterator built from the other Key before we exhaust the iterator built from 'this', we return greater than. An example: app1.type1.4.app1.type2.9.app1.type3.2 > app1.type1.4.app1.type2.9 The relationship between individual Key Keys is performed by comparing app followed by kind followed by id. If both keys are assigned names rather than ids, compares names instead of ids. If neither key has an id or a name we return an arbitrary but consistent result. Assuming all other components are equal, all ids are less than all names.

All you can do is fetch the entities of the type you want from the datastore and delete them one by one. (call datastore.delete(Iterable<Entity>))This way you can delete about 1000 entities at each request... There is no way to delete all entities of a Kind by a single api call.

> > If there is a way to list current Kinds or select all entities of any > > kind I could manage this.

to write large batch processing jobs without having to think about the plumbing details.

it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel

What would you have to build for yourself if Mapper weren’t available? Begin querying over every Entity in chained Task Queues Store beginning and end cursors (introduced in 1.3.5) Create tasks to work with chunks of your datastore Write the code to manipulate your data Build an interface to control your batch jobs Build a callback system for your multitudes of parallelized workers to call when the entire task has completed It’s certainly not a trivial amount of work

All you have to do is set the keys when you allocate the entities: Entity entity = new Entity(key); Or if you had previously pulled the entities from the datastore, the keys should already be set.

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

public final class KeyRangeextends java.lang.Objectimplements java.lang.Iterable<Key>, java.io.Serializable Represents a range of unique datastore identifiers from getStart().getId() to getEnd().getId() inclusive. The Keys returned by an instance of this class have been consumed in the datastore's id-space and are guaranteed never to be reused. This class can be used to construct Entities with Keys that have specific id values without fear of the datastore creating new records with those same ids at a later date. This can be helpful as part of a data migration or large bulk upload where you may need to preserve existing ids and relationships between entities. This class is threadsafe but the Iterators returned by iterator() are not.