Group items tagged

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

The GAE docs talk about how to setup a test to use the datastore.  But also follow the instructions here until the docs are fully updated.

The data loader feature communicates with your application running on App Engine using remote_api, a request handler included with the App Engine runtime environment that allows remote applications with the proper credentials to access the datastore remotely. To use remote_api, you must map a URL to it.

Yes, worker would be a servlet which can handle a request with POST parameters. If you want an asynchronous call from client's point of view then RPC is enough (from server's point of view it is still synchronous). If you want to do "delayed" jobs which don't talk to your client, you can use a task queue.

Instead of trying to create a thread (which is impossible in App Engine), this is a great way to asynchronously run tasks.

GaeVFS is an Apache Commons VFS plug-in that implements a distributed, writeable virtual file system for Google App Engine (GAE) for Java. GaeVFS is implemented using the GAE datastore and memcache APIs. The primary goal of GaeVFS is to provide a portability layer that allows you to write application code to access the file system--both reads and writes--that runs unmodified in either GAE or non-GAE servlet environments.

Google app engine uses the concept of static files. This is a performance optimization. Using the file appengine-web.xml you can configure the way google handles static files. You can include and exclude certain files using their extension or name. More information can be found here at google. This all works nice in the online version, however there seems to be a problem with the development server. Some solutions try to configure the local version as well, still that did not work for me. I decided to look for a servlet that serves static files.

That is it, now you can test your stuff locally and all your scripts, images, styles are loaded by your application. Of course you have to remove this servlet before uploading your application. Hope it helps people with their local debugging of jquery scripts or other javascript things.

disabling session id in code will not stop the cookie from saving session id. When you enable session in app engine xml file than you are telling the framework that you don't want to save sessions in cookies (client side), instead you want to save them on server i.e. using _ah_session. Cookies JSession id is default behavior of Java session mgmt. And it is nothing to do with APP engine framework.*

A Java app sets up queues using a configuration file named queue.xml, in the WEB-INF/ directory inside the WAR. See Java Task Queue Configuration. If an app does not have a queue.xml file, it has a queue named default with some default settings. To enqueue a task, you get a Queue using the QueueFactory, then call its add() method. You can get a named queue specified in the queue.xml file using the getQueue() method of the factory, or you can get the default queue using getDefaultQueue(). You can call the Queue's add() method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.

Although a queue defines a general FIFO ordering, tasks are not executed entirely serially. Multiple tasks from a single queue may be executed simultaneously by the scheduler, so the usual locking and transaction semantics need to be observed for any work performed by a task.

About loading request/performance, there're lots of discussions that you can find in the groups, please just try google it. Here are some notes based on reading those. Latency causes by 1- time to start new JVM 2- time to load your application To reduce load time by 1) others star request to a) pay to reserve JVM b) request Google to load your app before start dispatch request to that instance c) accept the situation For 2) we try to a- try to use/replace frameworks with light-weight ones: datastore access framework, MVC framework,... b- try to limit calculation in your index page to alleviate the impact of loading request c- design your object model based on your need, so that you do calculation at insert time, not at query time. For example, in my app, if I want to report on year and quarter, then I have 5 summary "record" for those, instead of querying and computing those d- caching result. For example, If I know 1 one 5 piece of data above would be read frequently, then I will read those 5 all, and put into memcache for later use ... So it's application-specific, I don't know if each of above can help you. But only you who can know if which one of your code can be cached and how...

Google App Engine applications written in Java can write information to the log files using java.util.logging.Logger. Log data for an application can be viewed and analyzed using the Administration Console, or downloaded using appcfg.sh request_logs. More info in the Logging documentation.

How do I handle multipart form data? or How do I handle file uploads to my app? You can obtain the uploaded file data from a multipart form post using classes from the Apache Commons FileUpload package. Specifically you may want to use FileItemStream, FileItermIterator and ServletFileUpload as illustrated below. If you see a java.lang.NoClassDefFoundError after starting your application, make sure that the Apache Commons FileUpload JAR file has been copied to your war/WEB-INF/lib directory and added to your build path. import org.apache.commons.fileupload.FileItemStream;import org.apache.commons.fileupload.FileItemIterator;import org.apache.commons.fileupload.servlet.ServletFileUpload;import java.io.InputStream;import java.io.IOException;import java.util.logging.Logger;import javax.servlet.ServletException;import javax.servlet.http.HttpServlet;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;public class FileUpload extends HttpServlet {  private static final Logger log =      Logger.getLogger(FileUpload.class.getName());  public void doPost(HttpServletRequest req, HttpServletResponse res)      throws ServletException, IOException {    try {      ServletFileUpload upload = new ServletFileUpload();      res.setContentType("text/plain");      FileItemIterator iterator = upload.getItemIterator(req);      while (iterator.hasNext()) {        FileItemStream item = iterator.next();        InputStream stream = item.openStream();        if (item.isFormField()) {          log.warning("Got a form field: " + item.getFieldName());        } else {          log.warning("Got an uploaded file: " + item.getFieldName() +                      ", name = " + item.getName());          // You now have the filename (item.getName() and the          // contents (which you can read from stream).  Here we just          // print them back out to the servlet output stream, but you          // will probably want to do something more interesting (for          // example, wrap them in a Blob and commit them to the          // datastore).          int len;          byte[] buffer = new

What do I mean by life cycle events? Events like entity creation, entity update and entity deletion. Mainstream ORM systems popularised callbacks like oncreate, onupdate, ondelete. Introducing such callbacks in the Java and Python APIs may be easy, but things get messy when you consider the ecosystem of alternative language implementations based on the Java API: developers using alternative languages would be forced to use Java to write the callbacks. There is a more robust solution though. Google App Engine already leverages the power of webhooks in such APIs as taskqueue, email, xmpp and more. Webhooks can elegantly solve the life cycle management problem as well: when an entity is created, updated or deleted through the Datastore viewer a corresponding webhook is triggered. Let's say the user is playing with Article entities, the webhooks uris could be: http://myapp.com/_ah/admin/datastore/le/Article/create/{key} http://myapp.com/_ah/admin/datastore/le/Article/update/{key} http://myapp.com/_ah/admin/datastore/le/Article/delete/{key} Slightly more work than callbacks, but still simple and effective. If there is an even better solution, I would love to hear about it in the comments section.

Originally my application started out using Spring MVC and JDO. The first use of JDO took my application about 5 seconds to get everything set up.  Similarily, Spring MVC added around 6 seconds to the cold start  time

 If you are not using either JDO or JPA, you can safely delete all related libraries (ones that have JDO, JPA, or Datanucleus in their name) and use the command line tool to upload your app. Deleting these libraries shaves about 400ms off of cold start time.