Group items tagged

Google app engine uses the concept of static files. This is a performance optimization. Using the file appengine-web.xml you can configure the way google handles static files. You can include and exclude certain files using their extension or name. More information can be found here at google. This all works nice in the online version, however there seems to be a problem with the development server. Some solutions try to configure the local version as well, still that did not work for me. I decided to look for a servlet that serves static files.

That is it, now you can test your stuff locally and all your scripts, images, styles are loaded by your application. Of course you have to remove this servlet before uploading your application. Hope it helps people with their local debugging of jquery scripts or other javascript things.

GaeVFS is an Apache Commons VFS plug-in that implements a distributed, writeable virtual file system for Google App Engine (GAE) for Java. GaeVFS is implemented using the GAE datastore and memcache APIs. The primary goal of GaeVFS is to provide a portability layer that allows you to write application code to access the file system--both reads and writes--that runs unmodified in either GAE or non-GAE servlet environments.

1) Show the plugins.groovy and routes.groovy files in the WEB-INF folder in the Gaelyk project diagram. They are currently absent from the diagram. 2 Put a star or some other icon next to the mandatory plugins folder and the 2 mandatory plugin files (plugins.groovy and the plugin descriptor file) in the diagram. This will help show that one just needs to mix in in these files with the rest of their own project's files.

Please support the ability to server static files from the runtime based on regex patterns similar to the current Python runtime. Currently the only way to simulate this functionality is with a servlet. This is not ideal peformance, as evidenced by existing special handling of static files. The most compelling use case is for versioning static files with the app's version ID so that browsers can maximally cache static files without experiencing stale caches later when the app is updated.

GaeVFS ships with a servlet that handles file uploads and abstracts how it writes them to its virtual file system. 

GaeVFS is built on top of Apache Commons VFS, and you use their FileObject instead of the standard File class.  There are some examples on the GaeVFS wiki, but I had to play around for a bit before I figured out some of the differences I needed to know. 

How do I handle multipart form data? or How do I handle file uploads to my app? You can obtain the uploaded file data from a multipart form post using classes from the Apache Commons FileUpload package. Specifically you may want to use FileItemStream, FileItermIterator and ServletFileUpload as illustrated below. If you see a java.lang.NoClassDefFoundError after starting your application, make sure that the Apache Commons FileUpload JAR file has been copied to your war/WEB-INF/lib directory and added to your build path. import org.apache.commons.fileupload.FileItemStream;import org.apache.commons.fileupload.FileItemIterator;import org.apache.commons.fileupload.servlet.ServletFileUpload;import java.io.InputStream;import java.io.IOException;import java.util.logging.Logger;import javax.servlet.ServletException;import javax.servlet.http.HttpServlet;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;public class FileUpload extends HttpServlet {  private static final Logger log =      Logger.getLogger(FileUpload.class.getName());  public void doPost(HttpServletRequest req, HttpServletResponse res)      throws ServletException, IOException {    try {      ServletFileUpload upload = new ServletFileUpload();      res.setContentType("text/plain");      FileItemIterator iterator = upload.getItemIterator(req);      while (iterator.hasNext()) {        FileItemStream item = iterator.next();        InputStream stream = item.openStream();        if (item.isFormField()) {          log.warning("Got a form field: " + item.getFieldName());        } else {          log.warning("Got an uploaded file: " + item.getFieldName() +                      ", name = " + item.getName());          // You now have the filename (item.getName() and the          // contents (which you can read from stream).  Here we just          // print them back out to the servlet output stream, but you          // will probably want to do something more interesting (for          // example, wrap them in a Blob and commit them to the          // datastore).          int len;          byte[] buffer = new

The war/ directory uses the WAR standard layout for bundling web applications. (WAR archive files are not yet supported by the SDK.) The Eclipse plugin uses this directory for running the development server, and for deploying the app to App Engine. When Eclipse builds your project, it creates a directory named classes/ in war/WEB-INF/, and puts compiled class files here. Eclipse also copies non-source files found in src/ to war/WEB-INF/classes/, including META-INF/ and the log4j.properties and logging.properties files. The final contents of the war/ directory make up your application for testing and deployment. For details about the new project that the plugin creates, see the Getting Started Guide.

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

A Java app sets up queues using a configuration file named queue.xml, in the WEB-INF/ directory inside the WAR. See Java Task Queue Configuration. If an app does not have a queue.xml file, it has a queue named default with some default settings. To enqueue a task, you get a Queue using the QueueFactory, then call its add() method. You can get a named queue specified in the queue.xml file using the getQueue() method of the factory, or you can get the default queue using getDefaultQueue(). You can call the Queue's add() method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.

Although a queue defines a general FIFO ordering, tasks are not executed entirely serially. Multiple tasks from a single queue may be executed simultaneously by the scheduler, so the usual locking and transaction semantics need to be observed for any work performed by a task.

Basically, all you have to do is create a subclass of bulkloader.Loader and implement (at a minimum) the generate_records method, which should yield lists of strings. This same strategy would work for loading data from XML files or ROT13-encrypted files or whatever. Note that the list of strings yielded by the generate_records method must match up (in length and order) with the "properties" list you provide when you initialize the loader (ie, the second argument to the AlbumLoader.__init__ method in this example). This approach actually provides a lot of flexibility: We're overriding the __init__ method on our JSONLoader implementation and automatically determining the kind of model we're loading and its list of properties to provide to the bulkloader.Loader parent class.

a plugin is just a bunch of files as is accurately described in the docs and these files go in certain places in your application's folders in order to be used as is also accurately described in the docs.

The most obvious use case is data import. A developer looking to import large amounts of data would take the following steps: Create a CSV file containing the data you want to import. The assumption here is that each line of data corresponds to a datastore entity you want to create Upload the CSV file to the blobstore. You’ll need billing to be enabled for this to work. Create your Mapper, push it live and run your job importing your data. This isn’t meant to be a replacement for the bulk uploader tool; merely an alternative. This method requires a good amount more programmatic changes for custom data transforms. The advantage of this method is that the work is done on the server side, whereas the bulk uploader makes use of the remote API to get work done. Let’s get started on each of the steps.

to build Mappers that map across some large, contiguous piece of data as opposed to Entities in the datastore

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

the necessary changes to appengine-web.xml to get Sitebricks to run in development mode:         <sessions-enabled>true</sessions-enabled>         <system-properties>                 <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>                 <property name="mvel2.disable.jit" value="true"/>         </system-properties>

the necessary changes to appengine-web.xml to get Sitebricks to run in development mode:         <sessions-enabled>true</sessions-enabled>         <system-properties>                 <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>                 <property name="mvel2.disable.jit" value="true"/>         </system-properties>

We had a GaeFlashCache built specifically for appengine. Try doing: bind(FlashCache.class).to(GaeFlashCache.class).in(Singleton.class);

My tag files worked very well in dev server without any problem. When uploading to gae/j, I encountered the problem describe in the above mentioned issue and I could fix it according to the workaround guidelines provided in that issue. My environment: WinXP SP2 Eclipse version 3.4 GAE/J SDK 1.3.1 JDK 5.0

The easiest way to get setup rapidly is to download the template project from the download section. It provides a ready-to-go project with the right configuration files pre-filled and an appropriate directory layout: web.xml preconfigured with the Gaelyk servlets appengine-web.xml with the right settings predefined (static file directive) a sample Groovlet and template the needed JARs (Groovy, Gaelyk and Google App Engine SDK)

Running your application locally Google App Engine provides a local servlet container, powered by Jetty, which lets you run your applications locally. If you're using the Gaelyk template, when you're at the root of your project — and we assume you have installed the App Engine SDK on your machine — you can run your application with the following command-line: dev_appserver.sh war

Deploying your application in the cloud Once you're at the root of your application, simply run the usual deployment command: appcfg.sh update war

build-time substitution. Here’s our appengine-web.xml: <?xml version="1.0" encoding="utf-8"?> <appengine-web-app xmlns="http://appengine.google.com/ns/1.0"> <application>${gae.app.name}</application> <version>${friendlyversion}</version> <system-properties> <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/> </system-properties> </appengine-web-app>

At build time, I use the AntRun plugin (lines 97-118) to create a small file under the target directory that holds a ‘sanitized’ version of the standard Maven version. (I.e. “1.0-SNAPSHOT” becomes GAE-friendly “1-0-snapshot”.) I then use the Maven filter functionality available in the WAR plugin (lines 80-96) to copy the appengine-web.xml into its proper directory with the version substituted in.

Personally, I would recommend using the blobstore API for this since it is officially supported by google and means it won't break in the future. It also has nice bindings into the imageservice. http://code.google.com/appengine/docs/java/blobstore/overview.html