Group items matching

in title, tags, annotations or url

Just set up remote_api for your app using the directions here - notably the tip: Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet. Then, use the Python bulkloader with --dump or --restore.

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

'm guessing your tests were run locally because the counter in the local datastore does indeed have datastore scope.  The scope of the counter in the prod datastore, however, is parent key + kind.  This is described here: http://code.google.com/appengine/docs/java/datastore/creatinggettinga... If it's important to you that the scope of the generated ids match between dev and and prod please file an issue.

                StringBuffer sb = new StringBuffer();                 KeyRange range = ds.allocateIds("a", 2);                 for (Key key : range) {                         sb.append("\n a " + key.toString());                 }                 range = ds.allocateIds("b", 2);                 for (Key key : range) {                         sb.append("\n b " + key.toString());                 }                 Key parentKey = KeyFactory.createKey("c", 1);                 sb.append("\n c " + parentKey.toString());                 range = ds.allocateIds(parentKey, "d", 2);                 for (Key key : range) {                         sb.append("\n d " + key.toString());                 }                 System.out.println(sb.toString());

The URL I posted earlier in the thread explains it, but here's a little bit more detail: A parent entity plus a kind defines an id-space, so entities with the same parent and the same kind are guaranteed to have unique ids.  For example, if you have an Entity with Parent:A Kind: Person Id: 10 you are guaranteed that no other entity with Parent A and Kind Person will be assigned an Id of 10.  However, an entity with a different Parent and Kind Person or an entity with Parent A and a different Kind _can_ be assigned an Id of 10.  The datastore pre-allocates batches of ids across multiple servers under-the-hood, so you can't make any assumptions about the Id that will get assigned in terms of contiguousness.  The only safe assumption is that the id will be unique for that Parent/Kind combination.

I figure it is going to take me at least four iterations to get this right. The first one will be building a GWT application with a simple UI that has no server logic behind it (just to learn how layout in GWT works). Step two will be adding a fake servlet backend (not app engine, just in memory). While not exactly App Engine yet, I should have a completely specified client-server API by the end of this process that I can subsequently implement on App Engine (iteration 3). Iteration four will handle deployment, CSS and whatever I may screw up in iterations one and two. I will log my notes of things I run into while I code.

In a relational world duplication is removed in order to prevent update anomalies. Error prevention is the driving force in relational modeling. Normalization is a kind of ethical system for data.

BigTable data ethics are more Mardi Gras than dinner with the in-laws. Data just wants to have fun. BigTable won’t stop you from hurting yourself. And to get the best results you may have to engage in some conventionally risky behaviors.

GaeVFS ships with a servlet that handles file uploads and abstracts how it writes them to its virtual file system. 

GaeVFS is built on top of Apache Commons VFS, and you use their FileObject instead of the standard File class.  There are some examples on the GaeVFS wiki, but I had to play around for a bit before I figured out some of the differences I needed to know. 

If you are trying to connect to a local server you will need to provide -s <server> arg Otherwise it will try to connect to <appid>.appspot.com

've been able to accomplish what you're doing with Selenium testing. If you're using GWT, then your integration testing and user acceptance probably won't be that far from each other.

> My question is this. What's the best way to use LocalServiceTestHelper > so that I can use my app like it has a persistent store? There is no > 'setUp' and 'tearDown' hooks like a JUnit test and the app runs in a > separate process within the IDE. (And in any case, I'm looking to do > integration testing where I want the state to be consistent across a > number of page requests.)

> I'm still curious where does "method" name come from? That is just "Get" of "Put" or "RunQuery" etc.    I have also checked   in an implementation of Nick Johnsons ApiProxyHook which logs all this   info: LoggingApiProxyDelegate.  Its handy for seeing how what rpc   calls are being made.  If you want to discuss Twig specifics probably   best to do that here http://groups.google.com/group/twig-persist

The remote_api module consists of two parts: A 'handler', which you install on the server to handle remote datastore requests, and a 'stub', which you set up on the client to translate datastore requests into calls to the remote handler. remote_api works at the lowest level of the datastore, so once you've set up the stub, you don't have to worry about the fact that you're operating on a remote datastore: With a few caveats, it works exactly the same as if you were accessing the datastore directly.

Note that the handler specifies "login: admin". This is extremely important, since we don't want to give just anyone unfettered access to our datastore!

Since you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, since a batch operation is only considered to be a single Datastore operation

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.