Group items tagged

Blobstore crash course It’ll be best if we gave a quick refresher course on the blobstore before we begin. Here’s the standard flow for a blobstore upload: Create a new blobstore session and generate an upload URL for a form to POST to. This is done using the createUploadUrl() method of BlobstoreService. Pass a callback URL to this method. This URL is where the user will be forwarded after the upload has completed. Present an upload form to the user. The action is the URL generated in step 1. Each URL must be unique: you cannot use the same URL for multiple sessions, as this will cause an error. After the URL has uploaded the file, the user is forwarded to the callback URL in your App Engine application specified in step 1. The key of the uploaded blob, a String blob key, is passed as an URL parameter. Save this URL and pass the user to their final destination

Once you've enabled OpenID authentication for your app, a few things change: URLs generated by create_login_url without a federated_identity parameter specified will redirect to the OpenID login page for Google Accounts. URLs that are protected by "login: required" in app.yaml or web.xml will result in a redirect to the path "/_ah/login_required", with a "continue" parameter of the page originally fetched. This allows you to provide your own openid login page. URLs generated by create_login_url with a federated_identity provider will redirect to the specified provider.

many users, when asked for their "openid URL" will simply look at you in puzzlement. What we need is a solution that avoids the need for most users to enter their URL themselves, while still allowing savvy users to do just that. Fortunately, there are a number of such solutions. One of them is clickpass. After signing in there and setting up an entry for your site, they'll provide you with code for a button that you can embed in your login page, and which allows users to choose from a number of well-known identity providers, including Hotmail, Yahoo!, Google, and Facebook, as well as entering their own URL.

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

The Blobstore API provides two types of functions:   An ability to upload and save the blob automaticallyThe BlobstoreService which is provided by the com.google.appengine.api.blobstore.BlobstoreService allows us to specify a URL where users can upload their large files. You can think of this url as the action element in the HTML form. The implementation at this URL is internal to the BlobstoreService. But what it does is significant. It will extract out the file contents that you uploaded and store it as a Blob in the database. Each blob that is stored in the database is associated with the a Blob Key. This Blob key is then provided to your url and you can then use the Blob Key to do anything within your application. In our case, we form a url that is tweeted to the users who can then view the picture that we uploaded. An ability to serve or retrieve the blob.The BlobstoreService also provides an ability to serve or retrieve the blob that was saved successfully. It will provide the blob as a response that could then use as a source in an <img> element for example. All you need to do is provide it the Blob Key and the response stream and in return, it will provide the content properly encoded as per its type that you could then use.

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

this won't be available until 1.3.6. You should be able to do something like this:  - property: __key__    external_name: CityId    export_transform: datastore.Key.id    import_transform: lambda value: datastore.Key.from_path('City', int(value))

for Java developers, there is a remote API handler available; adding following to your web xml file should work (disclaimer: I have not yet personally tested this.) <servlet>   <servlet-name>remoteapi</servlet-name>   <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class> </servlet> <servlet-mapping>   <servlet-name>remoteapi</servlet-name>   <url-pattern>/remote_api</url-pattern> </servlet-mapping> <security-constraint>   <web-resource-collection>     <web-resource-name>remoteapi</web-resource-name>     <url-pattern>/remote_api</url-pattern>   </web-resource-collection>   <auth-constraint>     <role-name>admin</role-name>   </auth-constraint> </security-constraint>

This tutorial concerns using Google Accounts to maintain the existence of your users in a Google App Engine application. Google App Engine provides the class UserServiceFactory to facilitate that. UserServiceFactory is then used to generate UserService object, which in turn provides the following features createLoginURL createLogoutURL getCurrentUser isUserAdmin isUserLoggedIn You would use UserService object to generate the login URL for the browser. The browser would be directed/redirected to this URL. On reaching this URL, the Google log-in prompt would be displayed by Google's server.

Yes, you will need to use the federated login (OpenID) stuff. The long-and-short of it is that you pass the federated_identity parameter to users.create_login_url.  You'll need to setup a page for users to tell you what goes in federated_identity somehow, perhaps by clicking a google logo or entering an apps domain. For Google accounts:   users.create_login_url(federated_identity='google.com/accounts/o8/id')   or   users.create_login_url(federated_identity='gmail.com') For an Apps account:   users.create_login_url(federated_identity='google.com/accounts/o8/site-xrds?hd=yourappsdomain.com') There is a little info here:   http://code.google.com/appengine/docs/python/users/overview.html And Wesley has a nice article about it here:   http://code.google.com/appengine/articles/openid.html Some info on Google Apps domains and OpenID:   http://groups.google.com/group/google-federated-login-api/web/openid-...

i had to do half an hour of googling to find the easiest solution; http://localhost:8080/_ah/admin/datastore imo it would be useful to have this information on http://code.google.com/appengine/docs/python/datastore/overview.html and related pages.

If you use the RemoteDatastore you have complete control in Java of   what you key is. http://code.google.com/p/remote-datastore/ e.g. this code puts an entity with the id set in your remote datastore   from your local machine:      RemoteDatastore.install();      RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");      DatastoreService service =   DatastoreServiceFactory.getDatastoreService();      Key key = KeyFactory.createKey("MyKindName, 35);      Entity entity1 = new Entity(key);      entity1.setProperty("property1", "hello");      datastore.put(Arrays.asList(entity1, entity2);

You can do this now using the RemoteDatastore Java utility http://code.google.com/p/remote-datastore/ For example, this code runs on your desktop and creates a single   entity in your live datastore:     // divert datastore operations to live application     RemoteDatastore.install();     RemoteDatastore.divert("http://myVersion.latest.myApp.appspot.com/remote-datastore ", "myApp", "myVersion");     // create an entity with a numeric key     Key key = KeyFactory.createKey("MyKindName, 35);     Entity entity1 = new Entity(key);     entity1.setProperty("property1", "hello");     // put entity to the remote datastore     DatastoreService service =   DatastoreServiceFactory.getDatastoreService();     datastore.put(entity1); This also works for bulk puts

It was one of the other ways to do it, I just wanted to avoid more code in my architecture just to do this (MVP has the disadvantage of added boilerplate for simple tasks like this one). Anyway I found a workaroud, passing the URL in an invisible div in my page and then reading and populating another field in my page using GWT RootPanel.get(id), using different ids for various parts of the page.

 If you're following the how-to article on unit testing ( http://code.google.com/appengine/docs/java/howto/unittesting.html) you'll need to update your TestEnvironment class to look like this one: http://code.google.com/p/datanucleus-appengine/source/browse/branches... (lines 34 - 66) We're working on getting the docs updated right now.

'm guessing your tests were run locally because the counter in the local datastore does indeed have datastore scope.  The scope of the counter in the prod datastore, however, is parent key + kind.  This is described here: http://code.google.com/appengine/docs/java/datastore/creatinggettinga... If it's important to you that the scope of the generated ids match between dev and and prod please file an issue.

                StringBuffer sb = new StringBuffer();                 KeyRange range = ds.allocateIds("a", 2);                 for (Key key : range) {                         sb.append("\n a " + key.toString());                 }                 range = ds.allocateIds("b", 2);                 for (Key key : range) {                         sb.append("\n b " + key.toString());                 }                 Key parentKey = KeyFactory.createKey("c", 1);                 sb.append("\n c " + parentKey.toString());                 range = ds.allocateIds(parentKey, "d", 2);                 for (Key key : range) {                         sb.append("\n d " + key.toString());                 }                 System.out.println(sb.toString());

The URL I posted earlier in the thread explains it, but here's a little bit more detail: A parent entity plus a kind defines an id-space, so entities with the same parent and the same kind are guaranteed to have unique ids.  For example, if you have an Entity with Parent:A Kind: Person Id: 10 you are guaranteed that no other entity with Parent A and Kind Person will be assigned an Id of 10.  However, an entity with a different Parent and Kind Person or an entity with Parent A and a different Kind _can_ be assigned an Id of 10.  The datastore pre-allocates batches of ids across multiple servers under-the-hood, so you can't make any assumptions about the Id that will get assigned in terms of contiguousness.  The only safe assumption is that the id will be unique for that Parent/Kind combination.

Cron jobs / task queues Instead of using a framework like Quartz to schedule jobs, Google App Engine takes care of executing jobs for you. You simply enter a cron-like expression and a URL to call and your job is configured. You also have a task queue at your disposal. Your application code can add tasks to a task queue which will be executed later in the future, asynchronously. An example use case is that you don’t want clients to wait for an email to be sent before he sees the next page. Instead you can put the email task on the task queue and the email will be sent asynchronously.

When an email is received Google App Engine does a post on an URL in your application you configured. The HTTP body of the POST request contains the exact mime message as it was received by Google. To parse this mime message you can use the MimeMessage class provided by the JDK.

Please support the ability to server static files from the runtime based on regex patterns similar to the current Python runtime. Currently the only way to simulate this functionality is with a servlet. This is not ideal peformance, as evidenced by existing special handling of static files. The most compelling use case is for versioning static files with the app's version ID so that browsers can maximally cache static files without experiencing stale caches later when the app is updated.

From what I understand, you are "manually" copying the properties from the Entity to the DTO. But this process is automated in GWT 2.1 So, I am not sending my entities directly, but the proxies as per the documentation : http://code.google.com/webtoolkit/doc/latest/DevGuideRequestFactory.html "An entity proxy is a client-side representation of an entity, otherwise known as a DTO (Data Transfer Object). With RequestFactory, entity proxies are interfaces that extend the EntityProxy interface, which is the hook used to indicate that an object can be managed by RequestFactory. RequestFactory automatically populates bean-style properties between entities on the server and the corresponding EntityProxy on the client, which simplifies using the DTO pattern. Furthermore, the EntityProxy interface enables RequestFactory to compute and send only changes ("deltas") to the server." "Entity proxies simply extend the EntityProxy interface and use the @ProxyFor annotation to reference the server-side entity being represented. It is not necessary to represent every property and method from the server-side entity in the EntityProxy, only getters and setters for properties that should be exposed to the client." The entity proxies are merely interfaces that are being populated by the new GWT 2.1 RequestFactory framework. I have no control over this copying process. Per definition, getters/setters of the real entity are injected into the EntityProxy whenever they are present; So my problem still stands : what about complex values like com.google.appengine.api.datastore.Email that are not known by the client side code ? How to transfer these complex values to the client.

When I copy my entities onto my dtos, some entity fields don't even make it into the dto. Others only have getters in the dto because they are read-only to the client. Those that do get into the dto get converted to native Java types. For instance, Text gets converted to String. Key gets encoded to a url friendly string. If I have Set fields on the entities to manager my relations (property lists), I remap those to ArrayList... First, don't serialize interfaces to GWT client, you'll get Javascript bloat. Then, Hashmap is costly to serialize because String.hashCode() is not the same on in Java and in Javascript. Hence, all the items need to be re-inserted into a client side map. Of course, in Web mode, performance is good enough... but in development mode, your data transfers will become really slow for somewhat big transfers.