Group items tagged

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

This article studies the proxy technique first and then turns to the Google AJAX Feed API method, giving you a chance to intermix Java™and JavaScript coding.

You need a service that, when given a feed URL, connects to that site, downloads its contents, and sends them back to the caller. (For a shell-line parallel, consider the wget or curl command.) You can do this many ways, and Listing 2 shows a simple way to accomplish the task. Because I decided to call my remote proxy RemoteProxy, the server-side class had to be called RemoteProxyImpl; Impl stands for "Implementation."

The SOP won't let your code get data from another site, but there's an exception: You can download and execute JavaScript code using the <script ... /> tag. If the code you download happens to include data and calls a function of yours that puts the data to good use, then you have managed to bypass the SOP. Here's the idea behind the Google AJAX Feed API: It uses the <script ... /> tag to call a Google site that works as a proxy. The remote site gets the feed data and returns it in the form of JavaScript code. The downloaded JavaScript code calls your function so you can process the incoming XML

step forward towards our goal of making Spring the best framework for developing enterprise-class cloud applications. Today we announced a partnership with Google to make Spring even better and to integrate it into the new Google AppEngine public cloud offering.

we realized that we have similar visions of the cloud

Our shared vision is to make it easy to build, run, and manage applications for the cloud, and to do so in a way that makes the applications portable across clouds. The rich applications should be able to run in an enterprise's private cloud, on Google's AppEngine, or on other public clouds committed to similar openness. Thus started an ambitious effort resulting in today's demonstrations at Google I/O and the downloads available here.

The easiest way to get setup rapidly is to download the template project from the download section. It provides a ready-to-go project with the right configuration files pre-filled and an appropriate directory layout: web.xml preconfigured with the Gaelyk servlets appengine-web.xml with the right settings predefined (static file directive) a sample Groovlet and template the needed JARs (Groovy, Gaelyk and Google App Engine SDK)

Running your application locally Google App Engine provides a local servlet container, powered by Jetty, which lets you run your applications locally. If you're using the Gaelyk template, when you're at the root of your project — and we assume you have installed the App Engine SDK on your machine — you can run your application with the following command-line: dev_appserver.sh war

Deploying your application in the cloud Once you're at the root of your application, simply run the usual deployment command: appcfg.sh update war

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

-dump --k

import_transform A single-argument function that returns the correct value and type data based on the external_name or import_template strings. Examples include the built-in Python conversion operators (such as float), any of several helper functions provided in transform, such as get_date_time or generate_foreign_key, a function provided in your own library, or an in-line lambda function. Or, a two-argument function with the keyword argument bulkload_state, which on return contains useful information about the entity: bulkload_state.current_entity, which is the current entity being processed; bulkload_state.current_dictionary, the current export dictionary, and bulkload_state.filename, the --filename argument that was passed to appcfg.py.

import_template Specifies multiple dictionary items for a single property, using Python string interpolation.

The data loader feature communicates with your application running on App Engine using remote_api, a request handler included with the App Engine runtime environment that allows remote applications with the proper credentials to access the datastore remotely. To use remote_api, you must map a URL to it.

Tip: If you have a Java app, you can use the Python bulkloader.py tool by installing the Java version of the remote_api handler, which is included with the Java runtime environment. The handler servlet class is com.google.apphosting.utils.remoteapi.RemoteApiServlet.

The property map consists of a set of 'property' entries, each of which specifies how to handle a particular property of the model on import and on export. for our Permission kind, the bulkloader has identified 4 properties, plus the __key__ pseudo-property. Each has an 'external_name', and optional import and export transforms, which specify how to translate between the App Engine datastore representation and an external representation.

All we had to do here was to remove some of the boilerplate and the extraneous invite_nonce entry, and fill in the kind names for the two reference properties, and we're sorted.

we didn't have to write a single line of Python code, or set up an app.yaml, or anything else Python-specific in order to achieve it! Further, the bulkloader took care of generating a mostly-finished configuration file for us, in a format that ensures the data we download can be re-uploaded again without loss of fidelity.

To get the development environment working the plugin also requires access to the unzipped SDK as packaged by Google. The plugin tries to help you set this up (“gae:unpack”) but that failed for me. I was able to get stuff working by manually unzipping the SDK artifact downloaded directly from Google to the following directory: ~/.m2/repository/com/google/appengine/appengine-java-sdk/1.3.0/appengine-java-sdk-1.3.0

You’ll also notice in our appengine-web.xml that we substitute in our application name. By default this comes from the properties section of the pom.xml file (line 11).

Google App Engine applications written in Java can write information to the log files using java.util.logging.Logger. Log data for an application can be viewed and analyzed using the Administration Console, or downloaded using appcfg.sh request_logs. More info in the Logging documentation.

On every project I work on, at the minimum I start a git project so I can track changes and push to either Github or a local server running Git (all Git requires is a server running sshd).

You can use the RemoteDatastore class to upload or download from a   normal Java application to your local or a remote datastore.  It takes   care of setting up a dummy Environment and ApiProxy.Delegate for you.   You can then use then read local files unrestricted and use the low- level api to insert your data. http://code.google.com/p/remote-datastore/ You just need to call RemoteDatastore.install() and ignore the other   steps about connecting to a remote datastore.