Group items tagged

db.get('ag1iYXRjaC1nZW5lcmljchcLEgxCYXRjaGVzTW9kZWwiBUpvYiAyDA')

If using the datastore API directly, a common pattern of usage is: // Get a handle on the datastore itself DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(); // Lookup data by known key name Entity userEntity = datastore.get(KeyFactory.createKey("UserInfo", email)); // Or perform a query Query query = new Query("Task", userEntity); query.addFilter("dueDate", Query.FilterOperator.LESS_THAN, today); for (Entity taskEntity : datastore.prepare(query).asIterable()) { if ("done".equals(taskEntity.getProperty("status"))) { datastore.delete(taskEntity); } else { taskEntity.setProperty("status", "overdue"); datastore.put(taskEntity); } }

Tip: If the app creates a new object and gives it the same string ID as another object of the same kind (and the same entity group parent), saving the new object overwrites the other object in the datastore. To detect whether a string ID is already in use prior to creating a new object, you can use a transaction to attempt to get an entity with a given ID, then create one if it doesn't exist. See Transactions.

There are 4 types of primary key fields:

class DeleteFull():     def execute(self):         deleting = model_class_name.all().order('__key__').fetch(100)         while deleting:             a = []             key = deleting[-1].key()             for item in deleting:                 a.append(item)             db.delete(a)             deleting = model_class_name.all().filter('__key__ >', key).order('__key__').fetch(100) This purged everything, but it took a hell of a long time.

Queries involving keys use indexes just like queries involving properties. Queries on keys require custom indexes in the same cases as with properties, with a couple of exceptions: inequality filters or an ascending sort order on __key__ do not require a custom index, but a descending sort order on __key__ does. As with all queries, the development web server creates appropriate configuration entries in this file when a query that needs a custom index is tested.

What do I mean by life cycle events? Events like entity creation, entity update and entity deletion. Mainstream ORM systems popularised callbacks like oncreate, onupdate, ondelete. Introducing such callbacks in the Java and Python APIs may be easy, but things get messy when you consider the ecosystem of alternative language implementations based on the Java API: developers using alternative languages would be forced to use Java to write the callbacks. There is a more robust solution though. Google App Engine already leverages the power of webhooks in such APIs as taskqueue, email, xmpp and more. Webhooks can elegantly solve the life cycle management problem as well: when an entity is created, updated or deleted through the Datastore viewer a corresponding webhook is triggered. Let's say the user is playing with Article entities, the webhooks uris could be: http://myapp.com/_ah/admin/datastore/le/Article/create/{key} http://myapp.com/_ah/admin/datastore/le/Article/update/{key} http://myapp.com/_ah/admin/datastore/le/Article/delete/{key} Slightly more work than callbacks, but still simple and effective. If there is an even better solution, I would love to hear about it in the comments section.

Blobstore crash course It’ll be best if we gave a quick refresher course on the blobstore before we begin. Here’s the standard flow for a blobstore upload: Create a new blobstore session and generate an upload URL for a form to POST to. This is done using the createUploadUrl() method of BlobstoreService. Pass a callback URL to this method. This URL is where the user will be forwarded after the upload has completed. Present an upload form to the user. The action is the URL generated in step 1. Each URL must be unique: you cannot use the same URL for multiple sessions, as this will cause an error. After the URL has uploaded the file, the user is forwarded to the callback URL in your App Engine application specified in step 1. The key of the uploaded blob, a String blob key, is passed as an URL parameter. Save this URL and pass the user to their final destination

Try this: SELECT * FROM PreparedTransaction WHERE __key__=KEY('agdwYXllbGV4cjkLEhNQcmVwYXJlZFRyYW5zYWN0aW9uIiAwMDAwMGNjMjMwYzg2MTFjZTFhOWZjZDJkZDEzMWMyNww') This is documented here: http://code.google.com/appengine/docs/python/datastore/gqlreference.html

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

Now you can choose to put this in an existing version of your applications config.ru. Or you can choose to put it in a new version (ie. version just for bulk upload/download without your application code). Either way once a version with the RemoteApiServlet is uploaded run this command to download your data: bulkloader.py --dump --app_id="application id" --url="url_to_remote_api_servlet" --filename="file to download data to" Here is an example from one of my apps: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data This command will download every object in your applications Datastore. If you only want to download the objects of one kind then simple add the --kind= option. Another example: bulkloader.py --dump --app_id=jsm277 --url=http://bulkmove.latest.jsm277.appspot.com/remote_api --filename=test_download_data --kind=Posts In order to restore the data that you have just downloaded use this command: bulkloader.py --restore --url="url_to_remote_api_servlet" --filename="file name from dump command" --app_id="application id" Here is an example command: bulkloader.py --restore --url=http://bulkmove.latest.railsturbinetest.appspot.com/remote_api --filename=test_download_data --app_id=railsturbinetest There are several important thing to keep in mind when restoring the Datastore objectes. The restore command simple restores whatever is in the filename that you provide. So it does not matter if you dump your entire Datastore or just one kind you use the same command to restore them both. You can restore Datastore objects to a different application then the one that they were downloaded from. However, is important to keep in mind that the object's key is used to restore each object. So if there already esists an object in the Datastore with the same key as an object that is being restored the object in the Datastore will be overwritten The bulkloader.py is a good tool for moving data between applications on the Google App Engine and for locally backing up your data. However, it is not good for data conversion or manipulation in anyway since the data is stored in a binary form. So happy data moving.

Here is the approach I would probably take unless you have many hundreds of child tasks. I would keep a list of keys of all children of each project/task. (You can then do a db.get and get all immediate children with single call) and each child has a reference to it's parent. In addition in each item keep a list of all parents keys back to the root. That way you can search for all children in part of the tree/subtree (This approach is especially useful if you have multiple child types) as back ref sets are kind specific. Keep a  running count of complete children at each level, rather than trying to calculate the current state in a single big function,  (keep these subsidiary values up to date through tasks)

As I understand it, the process of performing a single fetch (call to get())  from the dastastore using a key basically involves finding the host housing the entity, opening a socket, fetching the data, and then cleaning up the connection.  So to fetch something like 30 entities from the datastore, you're repeating the process 30 times over in serial, each time incurring whatever overhead is involved.  I also read that if you perform bulk fetches, (ie passing multiple keys at once) you can eliminate a great deal of that overhead.  In one of the videos I watched from Google I/0 2009, the presenter (whose name I forget - d'oh) said that performing a bulk fetch actually performs the fetches in parallel from the data store and you shoudl see requests noticeably faster.

I figure it is going to take me at least four iterations to get this right. The first one will be building a GWT application with a simple UI that has no server logic behind it (just to learn how layout in GWT works). Step two will be adding a fake servlet backend (not app engine, just in memory). While not exactly App Engine yet, I should have a completely specified client-server API by the end of this process that I can subsequently implement on App Engine (iteration 3). Iteration four will handle deployment, CSS and whatever I may screw up in iterations one and two. I will log my notes of things I run into while I code.

You should never store a password in plain text. Use a ir-reversable data hashing algorithm, like sha or md5 Here is how you can create a hash in python: from hashlib import sha256from random import randomrandom_key = random()sha256('%s%s%s'%('YOUR SECRET KEY',random_key,password)) You should also store the random key and hash the user supplied password similarly.

A reference property simply stores the unique key of the entity it references. So the mother and father entities could each contain a copy of the key corresponding to their child

Have you considered doing both? Then you could quickly get a list of computers a student owns by key OR use a query which returns results in some sorted order. I don't think maintaining a list of keys on the student model is as intimidating as you think.

The best option depends on your requirements. Here's a few solutions (I'm assuming you're using Python, since you didn't specify): If you need to do transactional updates on an entire tree, and you're not going to have more than about 1QPS of sustained updates to any one tree, you can use the built in support for heirarchial storage. When creating an entity, you can pass the "parent" attribute to specify a parent entity or key, and when querying, you can use the .ancestor() method (or 'ANCESTOR IS' in GQL to retrieve all descendants of a given entity. If you don't need transactional updates, you can replicate the functionality of entity groups without the contention issues (and transaction safety): Add a db.ListProperty(db.Key) to your model called 'ancestors', and populate it with the list of ancestors of the object you're inserting. Then you can easily retrieve everything that's descended from a given ancestor with MyModel.all().filter('ancestors =', parent_key). If you don't need transactions, and you only care about retrieving the direct children of an entity (not all descendants), use the approach outlined above, but instead of a ListProperty just use a ReferenceProperty to the parent entity. This is known as an Adjacency List. There are other approaches available, but those three should cover the most common cases.