Group items tagged

Task Queue is defined as a mechanism to synchronously distribute a sequence of tasks among parallel threads of execution. The global problem is broken down into tasks and the tasks are enqueued onto the queue. Parallel threads of execution pull tasks from the queue and perform computations on the tasks. The runtime system is responsible for managing thread accesses to the tasks in the queue as well as ensuring proper queue usage (i.e. dequeuing from an empty queue should not be allowed).

GAE Task Queue provides a push model. Instead of having an arbitrary number of worker processes constantly polling for available tasks, GAE Task Queue instead pushes work to workers when tasks are available. This work is then processed by the existing auto-scaling GAE infrastructure, allowing you to not have to worry about scaling up and down workers. You simply define the maximum rates of processing and GAE takes care of farming out the tasks to workers appropriately.

What is also compelling about GAE Task Queue is its use of web hooks as the description of a task unit. When you break it down, an individual task consists of the code to execute for that task as well as the individual data input for that specific task.

As I understand it, the process of performing a single fetch (call to get())  from the dastastore using a key basically involves finding the host housing the entity, opening a socket, fetching the data, and then cleaning up the connection.  So to fetch something like 30 entities from the datastore, you're repeating the process 30 times over in serial, each time incurring whatever overhead is involved.  I also read that if you perform bulk fetches, (ie passing multiple keys at once) you can eliminate a great deal of that overhead.  In one of the videos I watched from Google I/0 2009, the presenter (whose name I forget - d'oh) said that performing a bulk fetch actually performs the fetches in parallel from the data store and you shoudl see requests noticeably faster.

Authentication with App Engine, regardless of where you're doing it, is a three-stage process: Obtain an authentication token. This can be done with ClientLogin for installed apps, for example, or with AuthSub for a webapp. When logging in directly to an application, this is the part of the login process where your user sees a Google signin screen. Take that authentication token, and use it to obtain an authentication cookie. Use that authentication cookie in all subsequent requests.

this tip can cut a few hundred milliseconds or more off of your cold start time. Using JSPs slows you down in two ways: If you have a file with a .jsp extension, anywhere in your website directory, appcfg will detect that you use JSP, and it will add 8 libraries that are used for processing JSPs into your lib directory before uploading to the app engine.  These libraries total two megabytes in size, and simply having them in your lib directory slows your cold starts down by a few hundred milliseconds. If you are interested in what libraries these are, you can see them in the temporary folder appcfg creates while uploading your app. The first JSP file accessed after a cold start, even an empty one, will take a few hundred additional milliseconds to be processed.  I'm not sure what causes this, maybe it has to do with initializing the JSP processor.

From what I understand, you are "manually" copying the properties from the Entity to the DTO. But this process is automated in GWT 2.1 So, I am not sending my entities directly, but the proxies as per the documentation : http://code.google.com/webtoolkit/doc/latest/DevGuideRequestFactory.html "An entity proxy is a client-side representation of an entity, otherwise known as a DTO (Data Transfer Object). With RequestFactory, entity proxies are interfaces that extend the EntityProxy interface, which is the hook used to indicate that an object can be managed by RequestFactory. RequestFactory automatically populates bean-style properties between entities on the server and the corresponding EntityProxy on the client, which simplifies using the DTO pattern. Furthermore, the EntityProxy interface enables RequestFactory to compute and send only changes ("deltas") to the server." "Entity proxies simply extend the EntityProxy interface and use the @ProxyFor annotation to reference the server-side entity being represented. It is not necessary to represent every property and method from the server-side entity in the EntityProxy, only getters and setters for properties that should be exposed to the client." The entity proxies are merely interfaces that are being populated by the new GWT 2.1 RequestFactory framework. I have no control over this copying process. Per definition, getters/setters of the real entity are injected into the EntityProxy whenever they are present; So my problem still stands : what about complex values like com.google.appengine.api.datastore.Email that are not known by the client side code ? How to transfer these complex values to the client.

When I copy my entities onto my dtos, some entity fields don't even make it into the dto. Others only have getters in the dto because they are read-only to the client. Those that do get into the dto get converted to native Java types. For instance, Text gets converted to String. Key gets encoded to a url friendly string. If I have Set fields on the entities to manager my relations (property lists), I remap those to ArrayList... First, don't serialize interfaces to GWT client, you'll get Javascript bloat. Then, Hashmap is costly to serialize because String.hashCode() is not the same on in Java and in Javascript. Hence, all the items need to be re-inserted into a client side map. Of course, in Web mode, performance is good enough... but in development mode, your data transfers will become really slow for somewhat big transfers.

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

I have a class which can contain many small elements in a list. Looks like: public class Farm {    private ArrayList<Horse> mHorses;} just wondering what will happen if the mHorses array grew to something crazy like 15,000 elements. I'm assuming that trying to write and read this from the datastore would be crazy, because I'd get killed on the serialization process. It's important that I can get the entire array in one shot without paging, and each Horse element may only have two string properties in it, so they are pretty lightweight: public class Horse {    private String mId;    private String mName;} I don't need these horses indexed at all. Does it sound reasonable to just store the mHorse array as a raw Text field, and force my clients to do the deserialization? Something like: public class Farm {    private Text mHorsesSerialized;} then whenever the client receives a Farm instance, it has to take the raw string of horses, and split it in order to reinstantiate the list, something like: // GWT client perhapsFarm farm = rpcCall.getMyFarm();String horsesSerialized = farm.getHorses();String[] horseBlocks = horsesSerialized.split(",");for (int i = 0; i < horseBlocks.length; i++) {    // .. continue deserializing the individual objects ...} yeah...

import_transform A single-argument function that returns the correct value and type data based on the external_name or import_template strings. Examples include the built-in Python conversion operators (such as float), any of several helper functions provided in transform, such as get_date_time or generate_foreign_key, a function provided in your own library, or an in-line lambda function. Or, a two-argument function with the keyword argument bulkload_state, which on return contains useful information about the entity: bulkload_state.current_entity, which is the current entity being processed; bulkload_state.current_dictionary, the current export dictionary, and bulkload_state.filename, the --filename argument that was passed to appcfg.py.

import_template Specifies multiple dictionary items for a single property, using Python string interpolation.

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

I figure it is going to take me at least four iterations to get this right. The first one will be building a GWT application with a simple UI that has no server logic behind it (just to learn how layout in GWT works). Step two will be adding a fake servlet backend (not app engine, just in memory). While not exactly App Engine yet, I should have a completely specified client-server API by the end of this process that I can subsequently implement on App Engine (iteration 3). Iteration four will handle deployment, CSS and whatever I may screw up in iterations one and two. I will log my notes of things I run into while I code.

've been able to accomplish what you're doing with Selenium testing. If you're using GWT, then your integration testing and user acceptance probably won't be that far from each other.

> My question is this. What's the best way to use LocalServiceTestHelper > so that I can use my app like it has a persistent store? There is no > 'setUp' and 'tearDown' hooks like a JUnit test and the app runs in a > separate process within the IDE. (And in any case, I'm looking to do > integration testing where I want the state to be consistent across a > number of page requests.)

to write large batch processing jobs without having to think about the plumbing details.

it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel

What would you have to build for yourself if Mapper weren’t available? Begin querying over every Entity in chained Task Queues Store beginning and end cursors (introduced in 1.3.5) Create tasks to work with chunks of your datastore Write the code to manipulate your data Build an interface to control your batch jobs Build a callback system for your multitudes of parallelized workers to call when the entire task has completed It’s certainly not a trivial amount of work

This article studies the proxy technique first and then turns to the Google AJAX Feed API method, giving you a chance to intermix Java™and JavaScript coding.

You need a service that, when given a feed URL, connects to that site, downloads its contents, and sends them back to the caller. (For a shell-line parallel, consider the wget or curl command.) You can do this many ways, and Listing 2 shows a simple way to accomplish the task. Because I decided to call my remote proxy RemoteProxy, the server-side class had to be called RemoteProxyImpl; Impl stands for "Implementation."

The SOP won't let your code get data from another site, but there's an exception: You can download and execute JavaScript code using the <script ... /> tag. If the code you download happens to include data and calls a function of yours that puts the data to good use, then you have managed to bypass the SOP. Here's the idea behind the Google AJAX Feed API: It uses the <script ... /> tag to call a Google site that works as a proxy. The remote site gets the feed data and returns it in the form of JavaScript code. The downloaded JavaScript code calls your function so you can process the incoming XML