Group items tagged

the necessary changes to appengine-web.xml to get Sitebricks to run in development mode:         <sessions-enabled>true</sessions-enabled>         <system-properties>                 <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>                 <property name="mvel2.disable.jit" value="true"/>         </system-properties>

the necessary changes to appengine-web.xml to get Sitebricks to run in development mode:         <sessions-enabled>true</sessions-enabled>         <system-properties>                 <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>                 <property name="mvel2.disable.jit" value="true"/>         </system-properties>

We had a GaeFlashCache built specifically for appengine. Try doing: bind(FlashCache.class).to(GaeFlashCache.class).in(Singleton.class);

You don't have to add each class individually to the black list. They are all regex patterns, so just do this: <extend-configuration-property name="rpc.blacklist" value="com.google.gwt.user.client.ui.*Collection"/> In my case I wanted even finer grained control over which collections get included, so I did this: <extend-configuration-property name="rpc.blacklist" value="-.*List"/> <extend-configuration-property name="rpc.blacklist" value="-.*Map"/> <extend-configuration-property name="rpc.blacklist" value="-.*Collection"/> <extend-configuration-property name="rpc.blacklist" value="+java.util.HashMap"/> <extend-configuration-property name="rpc.blacklist" value="+java.util.LinkedHashMap"/> <extend-configuration-property name="rpc.blacklist" value="+java.util.ArrayList"/>

The property map consists of a set of 'property' entries, each of which specifies how to handle a particular property of the model on import and on export. for our Permission kind, the bulkloader has identified 4 properties, plus the __key__ pseudo-property. Each has an 'external_name', and optional import and export transforms, which specify how to translate between the App Engine datastore representation and an external representation.

All we had to do here was to remove some of the boilerplate and the extraneous invite_nonce entry, and fill in the kind names for the two reference properties, and we're sorted.

we didn't have to write a single line of Python code, or set up an app.yaml, or anything else Python-specific in order to achieve it! Further, the bulkloader took care of generating a mostly-finished configuration file for us, in a format that ensures the data we download can be re-uploaded again without loss of fidelity.

Entity is the fundamental unit of data storage. It has an immutable identifier (contained in the Key) object, a reference to an optional parent Entity, a kind (represented as an arbitrary string), and a set of zero or more typed properties.

setProperty public void setProperty(java.lang.String propertyName, java.lang.Object value) Sets the property named, propertyName, to value. As the value is stored in the datastore, it is converted to the datastore's native type. This may include widening, such as converting a Short to a Long. All Collections are prone to losing their sort order and their original types as they are stored in the datastore. For example, a TreeSet may be returned as a List from getProperty(java.lang.String), with an arbitrary re-ordering of elements. Overrides any existing value for this property, whether indexed or unindexed. Note that Blob and Text property values are never indexed by the built-in single property indexes. To store other types without being indexed, use #setUnindexedProperty. Parameters:value - may be one of the supported datatypes, a heterogenous Collection of one of the supported datatypes, or an UnindexedValue wrapping one of the supported datatypes. Throws: java.lang.IllegalArgumentException - If the value is not of a type that the data store supports.

the necessary changes to appengine-web.xml to get Sitebricks to run in development mode:         <sessions-enabled>true</sessions-enabled>         <system-properties>                 <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>                 <property name="mvel2.disable.jit" value="true"/>         </system-properties>

From what I understand, you are "manually" copying the properties from the Entity to the DTO. But this process is automated in GWT 2.1 So, I am not sending my entities directly, but the proxies as per the documentation : http://code.google.com/webtoolkit/doc/latest/DevGuideRequestFactory.html "An entity proxy is a client-side representation of an entity, otherwise known as a DTO (Data Transfer Object). With RequestFactory, entity proxies are interfaces that extend the EntityProxy interface, which is the hook used to indicate that an object can be managed by RequestFactory. RequestFactory automatically populates bean-style properties between entities on the server and the corresponding EntityProxy on the client, which simplifies using the DTO pattern. Furthermore, the EntityProxy interface enables RequestFactory to compute and send only changes ("deltas") to the server." "Entity proxies simply extend the EntityProxy interface and use the @ProxyFor annotation to reference the server-side entity being represented. It is not necessary to represent every property and method from the server-side entity in the EntityProxy, only getters and setters for properties that should be exposed to the client." The entity proxies are merely interfaces that are being populated by the new GWT 2.1 RequestFactory framework. I have no control over this copying process. Per definition, getters/setters of the real entity are injected into the EntityProxy whenever they are present; So my problem still stands : what about complex values like com.google.appengine.api.datastore.Email that are not known by the client side code ? How to transfer these complex values to the client.

When I copy my entities onto my dtos, some entity fields don't even make it into the dto. Others only have getters in the dto because they are read-only to the client. Those that do get into the dto get converted to native Java types. For instance, Text gets converted to String. Key gets encoded to a url friendly string. If I have Set fields on the entities to manager my relations (property lists), I remap those to ArrayList... First, don't serialize interfaces to GWT client, you'll get Javascript bloat. Then, Hashmap is costly to serialize because String.hashCode() is not the same on in Java and in Javascript. Hence, all the items need to be re-inserted into a client side map. Of course, in Web mode, performance is good enough... but in development mode, your data transfers will become really slow for somewhat big transfers.

list properties are indexed as 'multiply valued properties', with one index row per list entry. The index you specify should work fine.

public void setProperty(java.lang.String propertyName, java.lang.Object value) Sets the property named, propertyName, to value. As the value is stored in the datastore, it is converted to the datastore's native type. This may include widening, such as converting a Short to a Long. All Collections are prone to losing their sort order and their original types as they are stored in the datastore. For example, a TreeSet may be returned as a List from getProperty(java.lang.String), with an arbitrary re-ordering of elements. Overrides any existing value for this property, whether indexed or unindexed. Note that Blob and Text property values are never indexed by the built-in single property indexes. To store other types without being indexed, use #setUnindexedProperty. Parameters:value - may be one of the supported datatypes, a heterogenous Collection of one of the supported datatypes, or an UnindexedValue wrapping one of the supported datatypes. Throws: java.lang.IllegalArgumentException - If the value is not of a type that the data store supports.

build-time substitution. Here’s our appengine-web.xml: <?xml version="1.0" encoding="utf-8"?> <appengine-web-app xmlns="http://appengine.google.com/ns/1.0"> <application>${gae.app.name}</application> <version>${friendlyversion}</version> <system-properties> <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/> </system-properties> </appengine-web-app>

At build time, I use the AntRun plugin (lines 97-118) to create a small file under the target directory that holds a ‘sanitized’ version of the standard Maven version. (I.e. “1.0-SNAPSHOT” becomes GAE-friendly “1-0-snapshot”.) I then use the Maven filter functionality available in the WAR plugin (lines 80-96) to copy the appengine-web.xml into its proper directory with the version substituted in.

You can't in a query, the indexes only index values that exist,  you would have to explicitly set properties to None or some other sentinal value by default to find objects that haven't had a property set.

> How does one test for the presence or absence of a particular property > in an Entity?

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

this won't be available until 1.3.6. You should be able to do something like this:  - property: __key__    external_name: CityId    export_transform: datastore.Key.id    import_transform: lambda value: datastore.Key.from_path('City', int(value))

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

protocol

ublic void doGet(HttpServletRequest req, HttpServletResponse resp)     throws IOException {    resp.setContentType("text/plain");    final DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();    final Query query = new Query("Table/Entity Name");    //query.addSort(Entity.KEY_RESERVED_PROPERTY, Query.SortDirection.DESCENDING);    for (final Entity entity : datastore.prepare(query).asIterable()) {        resp.getWriter().println(entity.getKey().toString());        final Map<String, Object> properties = entity.getProperties();        final String[] propertyNames = properties.keySet().toArray(            new String[properties.size()]);        for(final String propertyName : propertyNames) {            resp.getWriter().println("-> " + propertyName + ": " + entity.getProperty(propertyName));        }    }}

to write large batch processing jobs without having to think about the plumbing details.

it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel

What would you have to build for yourself if Mapper weren’t available? Begin querying over every Entity in chained Task Queues Store beginning and end cursors (introduced in 1.3.5) Create tasks to work with chunks of your datastore Write the code to manipulate your data Build an interface to control your batch jobs Build a callback system for your multitudes of parallelized workers to call when the entire task has completed It’s certainly not a trivial amount of work

Basically, all you have to do is create a subclass of bulkloader.Loader and implement (at a minimum) the generate_records method, which should yield lists of strings. This same strategy would work for loading data from XML files or ROT13-encrypted files or whatever. Note that the list of strings yielded by the generate_records method must match up (in length and order) with the "properties" list you provide when you initialize the loader (ie, the second argument to the AlbumLoader.__init__ method in this example). This approach actually provides a lot of flexibility: We're overriding the __init__ method on our JSONLoader implementation and automatically determining the kind of model we're loading and its list of properties to provide to the bulkloader.Loader parent class.

A join model is a data model that models the relationship between two other models. For example, you might have Person entities and Group entities, and you want Persons to be in Groups and Groups to "have" Persons. The relationship between Persons and Groups is one of Memberships. A Person is "in" a Group "through" a Membership. They may belong to multiple Groups, i.e. have multiple Memberships.

class Membership(db.Model): person = db.ReferenceProperty(Person) group = db.ReferenceProperty(Group)

If certain properties of your joined entities (Person and Group in this example) don't change much, but get queried often via the join model, you may find it worth caching those properties on the join model itself.