Skip to main content

Home/ Google AppEngine/ Group items tagged bulkuploading

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
Esfand S

Using the bulkloader with Java App Engine « Ikai Lan says - 0 views

ikaisays.com/...ulkloader-with-java-app-engine

gaej gae bulkloader java top best appengine bulkloading theone

shared by Esfand S on 28 Sep 10 - Cached
  • I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.
  • create a custom uploader using the file upload example provided on appengine’s java FAQ.
  • App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.
  • ...1 more annotation...
  • I used a combination of uploading entire chunks of my data via FileUpload (see link below), and explicitly creating my Java objects with the keys that I wanted (which were easily implicitly defined by the data format as the first one would be ‘n’ and every object after it was n++). I would then insert the set of objects in bulk. The problem I hit the most was finding the right number of objects per store call. There are specific limits that make this process long and annoying. I ran something locally that would continue trying to upload the chunk of data until it got a good response from the server page. It took me something on the order of 6-8 hours to upload about 1.5M tiny objects. http://code.google.com/appengine/kb/java.html#fileforms
1 - 2 of 2
Showing 20▼ items per page