Group items tagged

for Java developers, there is a remote API handler available; adding following to your web xml file should work (disclaimer: I have not yet personally tested this.) <servlet>   <servlet-name>remoteapi</servlet-name>   <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class> </servlet> <servlet-mapping>   <servlet-name>remoteapi</servlet-name>   <url-pattern>/remote_api</url-pattern> </servlet-mapping> <security-constraint>   <web-resource-collection>     <web-resource-name>remoteapi</web-resource-name>     <url-pattern>/remote_api</url-pattern>   </web-resource-collection>   <auth-constraint>     <role-name>admin</role-name>   </auth-constraint> </security-constraint>

Just set up remote_api for your app using the directions here - notably the tip: Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet. Then, use the Python bulkloader with --dump or --restore.

Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet.

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

this won't be available until 1.3.6. You should be able to do something like this:  - property: __key__    external_name: CityId    export_transform: datastore.Key.id    import_transform: lambda value: datastore.Key.from_path('City', int(value))

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

import_transform A single-argument function that returns the correct value and type data based on the external_name or import_template strings. Examples include the built-in Python conversion operators (such as float), any of several helper functions provided in transform, such as get_date_time or generate_foreign_key, a function provided in your own library, or an in-line lambda function. Or, a two-argument function with the keyword argument bulkload_state, which on return contains useful information about the entity: bulkload_state.current_entity, which is the current entity being processed; bulkload_state.current_dictionary, the current export dictionary, and bulkload_state.filename, the --filename argument that was passed to appcfg.py.

import_template Specifies multiple dictionary items for a single property, using Python string interpolation.

The data loader feature communicates with your application running on App Engine using remote_api, a request handler included with the App Engine runtime environment that allows remote applications with the proper credentials to access the datastore remotely. To use remote_api, you must map a URL to it.

Actually, remote_api now supports almost all App Engine's APIs. An exception is the Users API, which is fairly useless over remote_api. :)

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

The property map consists of a set of 'property' entries, each of which specifies how to handle a particular property of the model on import and on export. for our Permission kind, the bulkloader has identified 4 properties, plus the __key__ pseudo-property. Each has an 'external_name', and optional import and export transforms, which specify how to translate between the App Engine datastore representation and an external representation.

All we had to do here was to remove some of the boilerplate and the extraneous invite_nonce entry, and fill in the kind names for the two reference properties, and we're sorted.

we didn't have to write a single line of Python code, or set up an app.yaml, or anything else Python-specific in order to achieve it! Further, the bulkloader took care of generating a mostly-finished configuration file for us, in a format that ensures the data we download can be re-uploaded again without loss of fidelity.