Google AppEngine

Actually, remote_api now supports almost all App Engine's APIs. An exception is the Users API, which is fairly useless over remote_api. :)

Just set up remote_api for your app using the directions here - notably the tip: Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet. Then, use the Python bulkloader with --dump or --restore.

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet.

A query like this will use the merge join strategy by default. If it's too slow for the merge join strategy, you could have to build a custom index for it, which would indeed be 'exploding'  - each User entity would have len(friends)*len(likes) index entries.

Merge join queries - queries with multiple equality filters but no inequalities or sort orders - are a special case. They can be satisfied using the built in indexes and a merge join strategy, but they can also use a custom index. The production environment will use the index if it's present, and otherwise will do a merge join. Generally, a merge join works well, but there are situations in which it doesn't - in which case you may need to add an explicit index.

Another way of solving the problem is to have a Horse object with a "farmKey" field like so: class Horse {  @PrimaryKey  Key mKey  Key farmKey } Then to get horses on a farm you'd do a query on Horse.kind with farmKey equal to your farm key.  (Keys only query would make this faster) What's nice about this approach is that you don't need Horse key at all, just create a Horse object, give it the farm key, and persist it without worrying about it.

> > so a Farm can have some horses. I'd like to create a new horse, then > > put it on a farm, in one transaction.

> This requires Horse to be a descendant of Farm to be in the same   > entity group.  Therefore a horse could not move farms unless you   > delete and recreate it with a new Key.

Have you tried these? for server-side (GAE) *com.google.appengine.repackaged.org.json.JSONObject.JSONObject(String arg0)* for client *com.google.gwt.json.client.JSONParser* I haven't tried japanese or chinese characters though, rather stuff like this: "Příliš žluťoučký kuň úpěl ďábělské ódy"

Entity groups are a simple concept. Entities in App Engine have ancestors. For e.g. you could model Books and Authors. Author: name->X, and Author: name->Y are two entities in the Author KIND. Book is another KIND (KIND is just a logical grouping of entities). The relationship between Books and Authors can be modeled as a ancestor-child relationship. E.g. Book: name->B1, B2 could have been written by Author: name->X. So you modeled them as: Author: name->X is a parent of both Book: name->B1, B2. Similarly, Book: name->B3 is written by Author: name->Y and so could model that relationship as Author:name->Y is a parent of Book:name->B3. When you try to transact on Books kind, you cannot transact on B1,B3 and B3 together. As they participate in different ancestor-child relationships. Every ancestor child relationship is an Entity group. You can only "lock" on one entity group at a time. Hope this makes things a little clear.

The best option depends on your requirements. Here's a few solutions (I'm assuming you're using Python, since you didn't specify): If you need to do transactional updates on an entire tree, and you're not going to have more than about 1QPS of sustained updates to any one tree, you can use the built in support for heirarchial storage. When creating an entity, you can pass the "parent" attribute to specify a parent entity or key, and when querying, you can use the .ancestor() method (or 'ANCESTOR IS' in GQL to retrieve all descendants of a given entity. If you don't need transactional updates, you can replicate the functionality of entity groups without the contention issues (and transaction safety): Add a db.ListProperty(db.Key) to your model called 'ancestors', and populate it with the list of ancestors of the object you're inserting. Then you can easily retrieve everything that's descended from a given ancestor with MyModel.all().filter('ancestors =', parent_key). If you don't need transactions, and you only care about retrieving the direct children of an entity (not all descendants), use the approach outlined above, but instead of a ListProperty just use a ReferenceProperty to the parent entity. This is known as an Adjacency List. There are other approaches available, but those three should cover the most common cases.

The most obvious use case is data import. A developer looking to import large amounts of data would take the following steps: Create a CSV file containing the data you want to import. The assumption here is that each line of data corresponds to a datastore entity you want to create Upload the CSV file to the blobstore. You’ll need billing to be enabled for this to work. Create your Mapper, push it live and run your job importing your data. This isn’t meant to be a replacement for the bulk uploader tool; merely an alternative. This method requires a good amount more programmatic changes for custom data transforms. The advantage of this method is that the work is done on the server side, whereas the bulk uploader makes use of the remote API to get work done. Let’s get started on each of the steps.

to build Mappers that map across some large, contiguous piece of data as opposed to Entities in the datastore

I use slf4j! use slf4j-log4j when development! use slf4j-jdk log when GAE Server!(change slf4j-log4j to slf4j-jdk-log before GAE appcfg update war)

It was one of the other ways to do it, I just wanted to avoid more code in my architecture just to do this (MVP has the disadvantage of added boilerplate for simple tasks like this one). Anyway I found a workaroud, passing the URL in an invisible div in my page and then reading and populating another field in my page using GWT RootPanel.get(id), using different ids for various parts of the page.

Basically, all you have to do is create a subclass of bulkloader.Loader and implement (at a minimum) the generate_records method, which should yield lists of strings. This same strategy would work for loading data from XML files or ROT13-encrypted files or whatever. Note that the list of strings yielded by the generate_records method must match up (in length and order) with the "properties" list you provide when you initialize the loader (ie, the second argument to the AlbumLoader.__init__ method in this example). This approach actually provides a lot of flexibility: We're overriding the __init__ method on our JSONLoader implementation and automatically determining the kind of model we're loading and its list of properties to provide to the bulkloader.Loader parent class.

IDs aren't even unique within a kind; they're unique within a kind and entity group. The reason they're not sequential is that an instance is allocated a large block of IDs, and unused ones aren't assigned to later instances

step forward towards our goal of making Spring the best framework for developing enterprise-class cloud applications. Today we announced a partnership with Google to make Spring even better and to integrate it into the new Google AppEngine public cloud offering.

we realized that we have similar visions of the cloud

Our shared vision is to make it easy to build, run, and manage applications for the cloud, and to do so in a way that makes the applications portable across clouds. The rich applications should be able to run in an enterprise's private cloud, on Google's AppEngine, or on other public clouds committed to similar openness. Thus started an ambitious effort resulting in today's demonstrations at Google I/O and the downloads available here.

You can use the RemoteDatastore class to upload or download from a   normal Java application to your local or a remote datastore.  It takes   care of setting up a dummy Environment and ApiProxy.Delegate for you.   You can then use then read local files unrestricted and use the low- level api to insert your data. http://code.google.com/p/remote-datastore/ You just need to call RemoteDatastore.install() and ignore the other   steps about connecting to a remote datastore.

One option is to create a model User in your own application, one of the things it contains would be the google user object.  In this case, you have your own ID generated by each instance of your Model, for each User. EJ: class User(db.Model):         googleUser = db.UserProperty()

On every project I work on, at the minimum I start a git project so I can track changes and push to either Github or a local server running Git (all Git requires is a server running sshd).

An end cursor allows you to specify when you want a query to stop. A cursor represents a query start position - an end cursor is another position. This lets you take, say, 10k entities, break them up into 100 "chunks", then do task queue or other background operations on them. Unlike using an offset (cursor +100), this method solves the case of new Entities being inserted, since you're not working with a cursor +100 Entities. You're working with all the Entities between startCursor and endCursor.

That was referring to the bulkloader, which lets you do those things. App Engine's datastore isn't a relational database. We can't do a dump of all your data without iterating through all of your indexes, then retrieving your Entities.