Group items tagged

you can only expect to update any single entity or entity group about five times a second. That is an estimate and the actual update rate for an entity is dependent on several attributes of the entity, including how many properties it has, how large it is, and how many indexes need updating. While a single entity or entity group has a limit on how quickly it can be updated, App Engine excels at handling many parallel requests distributed across distinct entities, and we can take advantage of this by using sharding.

What do I mean by life cycle events? Events like entity creation, entity update and entity deletion. Mainstream ORM systems popularised callbacks like oncreate, onupdate, ondelete. Introducing such callbacks in the Java and Python APIs may be easy, but things get messy when you consider the ecosystem of alternative language implementations based on the Java API: developers using alternative languages would be forced to use Java to write the callbacks. There is a more robust solution though. Google App Engine already leverages the power of webhooks in such APIs as taskqueue, email, xmpp and more. Webhooks can elegantly solve the life cycle management problem as well: when an entity is created, updated or deleted through the Datastore viewer a corresponding webhook is triggered. Let's say the user is playing with Article entities, the webhooks uris could be: http://myapp.com/_ah/admin/datastore/le/Article/create/{key} http://myapp.com/_ah/admin/datastore/le/Article/update/{key} http://myapp.com/_ah/admin/datastore/le/Article/delete/{key} Slightly more work than callbacks, but still simple and effective. If there is an even better solution, I would love to hear about it in the comments section.

The best option depends on your requirements. Here's a few solutions (I'm assuming you're using Python, since you didn't specify): If you need to do transactional updates on an entire tree, and you're not going to have more than about 1QPS of sustained updates to any one tree, you can use the built in support for heirarchial storage. When creating an entity, you can pass the "parent" attribute to specify a parent entity or key, and when querying, you can use the .ancestor() method (or 'ANCESTOR IS' in GQL to retrieve all descendants of a given entity. If you don't need transactional updates, you can replicate the functionality of entity groups without the contention issues (and transaction safety): Add a db.ListProperty(db.Key) to your model called 'ancestors', and populate it with the list of ancestors of the object you're inserting. Then you can easily retrieve everything that's descended from a given ancestor with MyModel.all().filter('ancestors =', parent_key). If you don't need transactions, and you only care about retrieving the direct children of an entity (not all descendants), use the approach outlined above, but instead of a ListProperty just use a ReferenceProperty to the parent entity. This is known as an Adjacency List. There are other approaches available, but those three should cover the most common cases.

 If you're following the how-to article on unit testing ( http://code.google.com/appengine/docs/java/howto/unittesting.html) you'll need to update your TestEnvironment class to look like this one: http://code.google.com/p/datanucleus-appengine/source/browse/branches... (lines 34 - 66) We're working on getting the docs updated right now.

The App Engine bulk update library is a library for the App Engine Python runtime, designed to facilitate doing bulk processing of datastore records. It provides a framework for easily performing operations on all the entities matched by any arbitrary datastore query, as well as providing an administrative interface to monitor and control batch update jobs.

I’m trying to use the bulkuploader for a java program but am running into an interesting issue. My PrimaryKey property is a Long, and in java I can explicitly give them id numbers and they show in the data store as “id=xxx”. When I download the data via the appcfg.py I get a reasonably looking data file. If I reupload the same file it actually inserts things into the data store with key “name=xxx” and therefore doubles every one of my entries.

create a custom uploader using the file upload example provided on appengine’s java FAQ.

App Engine’s datastore is schemaless. That is – it is possible to have Entities of the same Kind with completely different sets of properties. Most of the time, this is a good thing. MySQL, for instance, requires a table lock to do a schema update. By being schema free, migrations can happen lazily, and application developers can check at runtime for whether a Property exists on a given Entity, then create or set the value as needed. But there are times when this isn’t sufficient. One use case is if we want to change a default value on Entities and grandfather older Entities to the new default value, but we also want the default value to possibly be null.

I use slf4j! use slf4j-log4j when development! use slf4j-jdk log when GAE Server!(change slf4j-log4j to slf4j-jdk-log before GAE appcfg update war)

The GAE docs talk about how to setup a test to use the datastore.  But also follow the instructions here until the docs are fully updated.

In a relational world duplication is removed in order to prevent update anomalies. Error prevention is the driving force in relational modeling. Normalization is a kind of ethical system for data.

BigTable data ethics are more Mardi Gras than dinner with the in-laws. Data just wants to have fun. BigTable won’t stop you from hurting yourself. And to get the best results you may have to engage in some conventionally risky behaviors.

The easiest way to get setup rapidly is to download the template project from the download section. It provides a ready-to-go project with the right configuration files pre-filled and an appropriate directory layout: web.xml preconfigured with the Gaelyk servlets appengine-web.xml with the right settings predefined (static file directive) a sample Groovlet and template the needed JARs (Groovy, Gaelyk and Google App Engine SDK)

Running your application locally Google App Engine provides a local servlet container, powered by Jetty, which lets you run your applications locally. If you're using the Gaelyk template, when you're at the root of your project — and we assume you have installed the App Engine SDK on your machine — you can run your application with the following command-line: dev_appserver.sh war

Deploying your application in the cloud Once you're at the root of your application, simply run the usual deployment command: appcfg.sh update war

The remote_api module consists of two parts: A 'handler', which you install on the server to handle remote datastore requests, and a 'stub', which you set up on the client to translate datastore requests into calls to the remote handler. remote_api works at the lowest level of the datastore, so once you've set up the stub, you don't have to worry about the fact that you're operating on a remote datastore: With a few caveats, it works exactly the same as if you were accessing the datastore directly.

Note that the handler specifies "login: admin". This is extremely important, since we don't want to give just anyone unfettered access to our datastore!

Since you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, since a batch operation is only considered to be a single Datastore operation

I made some code public that does what you describe: it is a simple   cache interface that has implementations for in-memory, memcache and   the datastore.  You get about 100MB of heap space to use which can   significantly speed up your caching. There is also a CompositeCache class that allows you to layer the   caches so that it first checks in-memory, then memcache , then the   datastore.  Puts go to all levels and cache hits refresh the higher   levels.  e.g. if an item is not in-memory and has been flushed from   memcache but is still present in the datastore then the other two will   be updated.