Group items tagged

As I understand it, the process of performing a single fetch (call to get())  from the dastastore using a key basically involves finding the host housing the entity, opening a socket, fetching the data, and then cleaning up the connection.  So to fetch something like 30 entities from the datastore, you're repeating the process 30 times over in serial, each time incurring whatever overhead is involved.  I also read that if you perform bulk fetches, (ie passing multiple keys at once) you can eliminate a great deal of that overhead.  In one of the videos I watched from Google I/0 2009, the presenter (whose name I forget - d'oh) said that performing a bulk fetch actually performs the fetches in parallel from the data store and you shoudl see requests noticeably faster.

You could create an entity group for this. Post:    data    category    sort UserPost:    user

if you can form the key_name for post and userpost like this Key('Post', '<data>') Key('Post', '<data>', 'UserPost', '<user>') Then you can perform a key_only query for a userpost and using the keys parents perform a get to retrieve the relevant userposts Adding new users incrementally to a Post is very simple and light weight. Deleting a Post would also require you to delete the UserPost children. These being in an entity group would provide for performing transactions... generally you wouldn't need to use them anyway... maybe when performing a full delete.

If using the datastore API directly, a common pattern of usage is: // Get a handle on the datastore itself DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(); // Lookup data by known key name Entity userEntity = datastore.get(KeyFactory.createKey("UserInfo", email)); // Or perform a query Query query = new Query("Task", userEntity); query.addFilter("dueDate", Query.FilterOperator.LESS_THAN, today); for (Entity taskEntity : datastore.prepare(query).asIterable()) { if ("done".equals(taskEntity.getProperty("status"))) { datastore.delete(taskEntity); } else { taskEntity.setProperty("status", "overdue"); datastore.put(taskEntity); } }

Blobstore crash course It’ll be best if we gave a quick refresher course on the blobstore before we begin. Here’s the standard flow for a blobstore upload: Create a new blobstore session and generate an upload URL for a form to POST to. This is done using the createUploadUrl() method of BlobstoreService. Pass a callback URL to this method. This URL is where the user will be forwarded after the upload has completed. Present an upload form to the user. The action is the URL generated in step 1. Each URL must be unique: you cannot use the same URL for multiple sessions, as this will cause an error. After the URL has uploaded the file, the user is forwarded to the callback URL in your App Engine application specified in step 1. The key of the uploaded blob, a String blob key, is passed as an URL parameter. Save this URL and pass the user to their final destination

Task Queue is defined as a mechanism to synchronously distribute a sequence of tasks among parallel threads of execution. The global problem is broken down into tasks and the tasks are enqueued onto the queue. Parallel threads of execution pull tasks from the queue and perform computations on the tasks. The runtime system is responsible for managing thread accesses to the tasks in the queue as well as ensuring proper queue usage (i.e. dequeuing from an empty queue should not be allowed).

GAE Task Queue provides a push model. Instead of having an arbitrary number of worker processes constantly polling for available tasks, GAE Task Queue instead pushes work to workers when tasks are available. This work is then processed by the existing auto-scaling GAE infrastructure, allowing you to not have to worry about scaling up and down workers. You simply define the maximum rates of processing and GAE takes care of farming out the tasks to workers appropriately.

What is also compelling about GAE Task Queue is its use of web hooks as the description of a task unit. When you break it down, an individual task consists of the code to execute for that task as well as the individual data input for that specific task.

Just my 2 cents: I was "forced" to use an "unowned relationship" for my application because I had very large objects: My parent object "Journey" contained thousands of "JourneyPoints" in a List element. A call to "makePersistent" constantly hit the 30s deadline. By modeling it as an unowned relationship and distributing the parent key by hand to the JourneyPoints everything was fine again and the performance was very good.

About loading request/performance, there're lots of discussions that you can find in the groups, please just try google it. Here are some notes based on reading those. Latency causes by 1- time to start new JVM 2- time to load your application To reduce load time by 1) others star request to a) pay to reserve JVM b) request Google to load your app before start dispatch request to that instance c) accept the situation For 2) we try to a- try to use/replace frameworks with light-weight ones: datastore access framework, MVC framework,... b- try to limit calculation in your index page to alleviate the impact of loading request c- design your object model based on your need, so that you do calculation at insert time, not at query time. For example, in my app, if I want to report on year and quarter, then I have 5 summary "record" for those, instead of querying and computing those d- caching result. For example, If I know 1 one 5 piece of data above would be read frequently, then I will read those 5 all, and put into memcache for later use ... So it's application-specific, I don't know if each of above can help you. But only you who can know if which one of your code can be cached and how...

public final class Keyextends java.lang.Objectimplements java.io.Serializable, java.lang.Comparable<Key> The primary key for a datastore entity. A datastore GUID. A Key instance uniquely identifies an entity across all apps, and includes all information necessary to fetch the entity from the datastore with DatastoreService.get(Key). You can create Key objects directly by using KeyFactory.createKey(java.lang.String, long) or getChild(java.lang.String, long). You can also retrieve the Key automatically created when you create a new Entity, or serialize Key objects, or use KeyFactory to convert them to and from websafe String values.

equals public boolean equals(java.lang.Object object) Compares two Key objects by comparing ids, kinds, parent and appIdNamespace. If both keys are assigned names rather than ids, compares names instead of ids. If neither key has an id or a name, the keys are only equal if they reference the same object.

compareTo public int compareTo(Key other) Compares two Key objects. The algorithm proceeds as follows: Turn each Key into an iterator where the first element returned is the top-most ancestor, the next element is the child of the previous element, and so on. The last element will be the Key we started with. Once we have assembled these two iterators (one for 'this' and one for the Key we're comparing to), consume them in parallel, comparing the next element from each iterator. If at any point the comparison of these two elements yields a non-zero result, return that as the result of the overall comparison. If we exhaust the iterator built from 'this' before we exhaust the iterator built from the other Key, we return less than. An example: app1.type1.4.app1.type2.9 < app1.type1.4.app1.type2.9.app1.type3.2 If we exhaust the iterator built from the other Key before we exhaust the iterator built from 'this', we return greater than. An example: app1.type1.4.app1.type2.9.app1.type3.2 > app1.type1.4.app1.type2.9 The relationship between individual Key Keys is performed by comparing app followed by kind followed by id. If both keys are assigned names rather than ids, compares names instead of ids. If neither key has an id or a name we return an arbitrary but consistent result. Assuming all other components are equal, all ids are less than all names.

You can tell if it's using sqlite by executing 'sqlite3 /path/to/datastore' on the command line, and seeing if sqlite recognizes it as a valid DB. The code is essentially the same as that which I announced on my blog, so there shouldn't be significant performance differences.

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

With the Task Queue API, applications can perform work outside of a user request but initiated by a user request. If an app needs to execute some background work, it may use the Task Queue API to organize that work into small, discrete units, called Tasks. The app then inserts these Tasks into one or more Queues. App Engine automatically detects new Tasks and executes them when system resources permit.

Google app engine uses the concept of static files. This is a performance optimization. Using the file appengine-web.xml you can configure the way google handles static files. You can include and exclude certain files using their extension or name. More information can be found here at google. This all works nice in the online version, however there seems to be a problem with the development server. Some solutions try to configure the local version as well, still that did not work for me. I decided to look for a servlet that serves static files.

That is it, now you can test your stuff locally and all your scripts, images, styles are loaded by your application. Of course you have to remove this servlet before uploading your application. Hope it helps people with their local debugging of jquery scripts or other javascript things.

 Twig is an alternative to the standard   persistence interfaces JDO and JPA, specifically designed to make the   most of the underlying datastore's unique features.

Twig is the only interface to support direct unowned relationships so   your code is not dependent on low-level datastore classes.  It is the   only interface to support OR queries by merging together multiple   queries, sorting them and filtering out duplicates at the lowest level   for performance.

Async datastore calls are not yet part of the low-level API.  Twig   uses the underlying Protocol Buffer layer classes to call   ApiProxy.makeAsyncCall() instead of makeSyncCall.  All the code is   open source so you can check out how its done.

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.

A Java app sets up queues using a configuration file named queue.xml, in the WEB-INF/ directory inside the WAR. See Java Task Queue Configuration. If an app does not have a queue.xml file, it has a queue named default with some default settings. To enqueue a task, you get a Queue using the QueueFactory, then call its add() method. You can get a named queue specified in the queue.xml file using the getQueue() method of the factory, or you can get the default queue using getDefaultQueue(). You can call the Queue's add() method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.

Although a queue defines a general FIFO ordering, tasks are not executed entirely serially. Multiple tasks from a single queue may be executed simultaneously by the scheduler, so the usual locking and transaction semantics need to be observed for any work performed by a task.

One thing you get used to on appengine is that any bulk data work requires the task queue.  You can use a little bit of framework and make all of these transforms (including deleting data) a question of just writing a simple task class and firing it off.  You'll want a copy of the Deferred servlet: http://code.google.com/p/gaevfs/source/browse/trunk/src/com/newatlant... Fair warning:  I found that I needed to change the code to make it perform base64 encoding all the time, not just on the dev instance.