Category Archives: features

Making Cloud Datastore Easy For PHP on App Engine

Firstly, thank you to @slangley and the team for asking me to guest post here. I really hope this helps you PHP folks on App Engine.

TL;DR – I can has code?

Sure, no problem. https://github.com/tomwalder/php-gds

Introduction

PHP on Google App Engine went Generally Available earlier this year, and that’s pretty awesome.

Certainly for me (and probably some of you) it makes both prototyping and running super-scalable web applications really easy and reliable.

Whilst the PHP part of the App Engine environment is ultra easy to use and scalable, you are likely to need a similarly featured data persistence layer for any useful applications. MySQL is cool and all, but it costs money (via Cloud SQL) and can’t scale automatically in the same way App Engine can.

Enter Cloud Datastore.

It’s pretty sweet – auto-scaling, distributed, supports transactions, mostly schema-less, managed, free to get started (up to 1GB of data and 50k queries per day).

So, you ask yourself, how do I use this amazing thing? Well until not that long ago it was a bit of a pain. you needed loads of glue code to get even the basics running.

So the php-gds library was built to simplify usage of Datastore from PHP. It tries to make it really, really easy to get started, whilst also being feature rich. It’s seeing a fair amount of adoption as far as I can tell, and the guys at Google are pretty keen on it too – hence this post!

Example – Write to Datastore

For any code running on the local development server or on a real App Engine application, it’s this easy…

// Build a new entity
$obj_book = new GDS\Entity();
$obj_book->title = 'Romeo and Juliet';
$obj_book->author = 'William Shakespeare';
$obj_book->isbn = '1840224339';

// Write it to Datastore
$obj_store = new GDS\Store('Book');
$obj_store->upsert($obj_book);

Example – Read from Datastore

Again, I try to make reading data is simple and intuitive as possible…

$obj_store = new GDS\Store('Book');
foreach($obj_store->fetchAll() as $obj_book) {
    echo "Title: {$obj_book->title}, ISBN: {$obj_book->isbn} <br />", PHP_EOL;
}

Native Access or JSON API

The php-gds library uses the native, Protocol Buffer APIs to access the Datastore – this means it is super fast.

It also supports remote access using the JSON API – in fact, you can use the same code against both, just passing the right Gateway object into your Store. More details on GitHub here.

Query Language – GQL

At the time of writing, php-gds uses the GQL, which is very similar to SQL, to query Datastore – and parameter binding, which is recommended.

$obj_book_store->fetchOne("SELECT * FROM Book WHERE isbn = @isbnNumber", [
    'isbnNumber' => '1853260304'
]);

Best Practice – Define a Schema/Model

You will get more (for less) if you define your Schema clearly, including what fields in your data to index. Indexing costs money (above the free quota) so it’s a good move to think about and define your model

More detail on defining a model can be found here.

Further Reading

There is a pretty decent set of guidance in the GitHub README and in the examples.

Support

Create an issue on GitHub or post on SO.

About Me

My name is Tom Walder, I’m CTO at Docnet, an e-commerce platform and solutions provider based in Manchester, UK. You can find me on twitter @tom_walder

I am the author of the php-gds Datastore library for PHP.

PHP App Engine Apps and File System Concepts

If you’re new to App Engine, then the file system model may be a bit different from what you might have experienced using other hosts.

With an App Engine application, you cannot write to the file-system where your application is deployed. Your application can read any files from the deployed directory structure, but it can’t write to that file-system. Instead, the application can use Google Cloud Storage (GCS) for both reading and writing files.
To convert an app to use GCS for writable files, here are the primary things that you will typically need to do:

Another implication of the read-only file system is that if your app has a ‘plugin’ model, like WordPress, you can’t install or update plugins from your deployed app. Instead, you will need to install or update any plugins locally, then redeploy the app.

You can find lots more info on all of the following in the documentation for the PHP runtime.

Continue reading →

Getting started with the Cloud Datastore on PHP App Engine, Part II

Introduction

This is the second in a series of posts on using the Google Cloud Datastore on PHP App Engine.
The code for the examples in this post is here: https://github.com/amygdala/appengine_php_datastore_example.

The previous post walked through the process of setting up your PHP App Engine app to connect with the Datastore, using the Google API client libraries.
It demonstrated how to create a test Datastore entity, and showed how to set up a “Datastore service” instance, using a DatastoreService class, to make it easier to send requests.

However, using the API client libraries, there is a fair bit of overhead in composing requests to the Datastore and processing the responses, and mapping entities to instances of classes in your app. So, in this post, we’ll define some classes to make it easier to create, update, and fetch Datastore entities.
We’ll define a `Model` base class, then show an example of subclassing it to map to a specific Datastore Kind.

Continue reading →

Getting Started with the Cloud Datastore on PHP App Engine

Introduction

In this post, we’ll show how to access the Google Cloud Datastore from a PHP App Engine app.

This is the first in a series of posts on using the Cloud Datastore on PHP App Engine.
The code for the examples in this post is here: https://github.com/amygdala/appengine_php_datastore_example. (Some of the files in this repo won’t be mentioned in this post, but will be referenced in the post that follows this one).

PHP on Google App Engine supports several alternatives as for a persistent store. One is the Cloud Datastore, a distributed, robust, fully managed, schemaless database, which supports transactions and scales automatically.
(If you have built App Engine apps for other runtimes, this is the same Datastore that you’ve used before; it now supports an HTTP interface as well).
This article assumes that you already have a basic familiarity with Datastore concepts.

There is not yet a native Datastore API for PHP on app engine. So, instead we’ll show how to use the Cloud Datastore API to connect to the Datastore from your PHP app.
First, we’ll show how to configure your app and associated Cloud Project so that your app will be able to connect to the Datastore.

Then, we’ll look at how to use the Google API client libraries as a basis for authenticating with the Datastore, sending requests, and receiving responses, and we’ll build a DatastoreService class to make connecting a bit easier.

Continue reading →

App Engine 1.8.8

App Engine 1.8.8 has just been released, and the PHP runtime has some great updates:

Harnessing The Power of Versions On App Engine

One powerful, but often overlooked, feature of running applications on App Engine is the ability to have multiple versions of your application deployed and running simultaneously. This feature is invaluable when it comes to being able to “smoke test” a new version of your site live in production before your users can see it. Or you can use traffic splitting to allow you to test different versions of your site against live traffic in a controlled and deterministic fashion.

The release of WordPress 3.5.2 is a great opportunity for us to demonstrate how to use versions to do a controlled upgrade of our WordPress blog to the latest version.

A Note About App Engine Versions

Before we get started on the steps for running multiple versions of your application, we should briefly describe application versions and the default version. App Engine allows you to deploy up to 10 versions of your application to the production servers. Versions are completely independent of each other, and in fact versions can even be using different runtimes – so you can deploy a PHP application and a Java application as different versions of the same appspot application and run them simultaneously. Versions are identified by the version string that you specify either in the app.yaml file or as a command line paramater when using the appcfg.py tool. If you specify the version as a command line option then it will override the version that is supplied in the app.yaml file.

The first time that you upload your application to the production servers, it will become the default version, and all user requests will be served using that version. If you upload a new version of your application with a new version identifier, then it will not serve any user requests until you go into the admin console and either mark it as the default version or enable traffic splitting (more detail about both are later in this post). If you use a version identifier that matches an already uploaded version, then it will overwrite the old version with the new one. If that version happens to be the default version, then it will start serving immediately.

Now we know a little about versions, we can go about using them to upgrade our version of WordPress we’re using.

Step 1: Uploading The New Version Of Your Site.

The first step to running a new version is actually uploading the updated version of your site to the production servers. The steps outlined below assume that you have setup your WordPress source code following the instructions on the Running WordPress page. We will get the latest version of WordPress and unpack it to the APPLICATION_DIRECTORY directory as mentioned in that article, using the following steps.

$ wget http://wordpress.org/latest.zip
$ unzip -o latest.zip -d APPLICATION_DIRECTORY

You should now run the site in your development environment to ensure that it behaves as you expect it to (once again, following the instructions on the Running WordPress page).

$ APP_ENGINE_SDK_PATH/dev_appserver.py --php_executable_path=PHP_CGI_EXECUTABLE_PATH APPLICATION_DIRECTORY

Once you’re satisfied that the site is running as you would expect in the development environment, you can upload it to the production servers. We’ll use the --version flag of appcfg.py to upload the site with a new version number, that differs from the exiting version. In this example, we’ll use today’s date and the postfix “-352″ as the version identifier.

$ appcfg.py -R --version 20130624-352 update .

The appcfg tool will now upload a new version of your site. One interesting thing to note is that the tool will only upload files to the server that are different to existing files that you’ve previously uploaded in an existing version. This results in very fast uploads for new versions of large site that have only a small number of changes to the source.

You should see output similar to what’s shown below

10:53 AM Host: appengine.google.com
10:53 AM Application: gae-php-tips; version: 20130624-352 (was: wp-0501)
10:53 AM Starting update of app: gae-php-tips, version: 20130624-352
10:53 AM Getting current resource limits.
10:53 AM Scanning files on local disk.
...
10:53 AM Checking if deployment succeeded.
10:53 AM Deployment successful.
10:53 AM Checking if updated app version is serving.
10:53 AM Completed update of app: gae-php-tips, version: 20130624-352
10:53 AM Uploading cron entries.

You can now go to the admin console for you site and view the new version that was just uploaded.

Screen Shot 2013-06-24 at 11.02.41 AM

As you can see from the screen shot, the new version has successfully uploaded, but our site is still serving live traffic from the existing version labelled “17062013”.

Step 2: Manually Testing The New Version

App Engine versions allow you to access any of the deployed versions of your site by prepending the correct version specifier to the site address. In our example here, I can access the new version of http://gae-php-tips.appspot.com/ by entering the url http://20130624-352-dot-gae-php-tips.appspot.com/ in the address bar of my browser.

Note: We use this feature to deploy phpMyAdmin side by side with WordPress in our application. You can see a version named “phpmyadmin” in the list of deployed versions.

Step 3: Traffic Splitting Live Traffic To The New Version.

Now that we’ve deployed the new version of the site and manually tested that it serves as we expect, we can now start sending it live traffic. We have two options for serving traffic from this applcation.

  • Make the new version the default version so that it handles all requests for the site.
  • Use traffic splitting to send a percentage of users to the new version, to assess its stability before making it the default.

In this example, we’ll use traffic splitting to ensure that the new version of the site is stable before making it the default. To get started, click on the traffic splitting link on the versions page for your site. It will present the option to enable traffic splitting which will look like this.

Screen Shot 2013-06-24 at 11.17.03 AM

From here click on the “Add Traffic Split…” button to configure traffic splitting. You will be asked to select the version that you want to split traffic to, and the percentage of traffic to split to the new version. Select the version that you just uploaded and the amount of traffic you want to send it (in my case, 25%).

Screen Shot 2013-06-24 at 11.17.26 AM

Once you click on OK App Engine will enable traffic splitting using the configuration that you’ve supplied. A reminder that traffic splitting is enabled will be shown on your admin console page for the site.

Screen Shot 2013-06-24 at 11.17.42 AM

Your site is now serving a percentage of user requests with the new version of your site.

Step 4: Monitoring The New Version.

You can use the application logs to verify that the new version of your site is serving without errors. At the top of the logs page it allows you to view logs for a specific  version. Using this, you can quickly drill down if the new version of your site is serving correctly.

Screen Shot 2013-06-24 at 11.31.34 AM

You can see that these are the logs for the versions 17062013 which is currently serving 75% of the traffic for the site, while below are the logs for a new version 20130624-352 which is serving 25% of the traffic for our site.

Screen Shot 2013-06-24 at 11.30.58 AM

Step 5: Making The New Version The Default

Once we’re satisfied that the new version of the site is performing correctly, we can switch it over to be the default for all user requests.To do this, we go back to the versions page in the admin console for our site. By selecting the radio button next to the new version and pressing Make Default, the traffic split will be dropped and the new version will handle all live traffic.

Screen Shot 2013-06-24 at 11.38.29 AM

You could also roll back to the older version of the app by removing the traffic split, or change the split percentage if you want to gradually increase the traffic load on the new version over time.

Notes About Running Multiple Versions

One important thing to consider when running multiple versions of your site, is that each different version of your site requires at least one running instance to serve from. These instances are charged independently, so that if you are traffic splitting on a free site you are likely to run out of the free quota during one 24 hour billing period.

Another thing to consider when running multiple versions is that all versions share the same database. If there is a schema change to the database for a new version, the older versions may not function correctly. In our example we upgraded from WordPress 3.5.1 to 3.5.2 which uses the same schema so we could serve both versions from the same database. If you want to run multiple versions that have different schemas then you use separate databases for each version. You could use the backup and restore functionality of CloudSQL create a copy of your live database for the new version.

What’s new for PHP in the App Engine 1.8.1 Release

Today we publicly announced the 1.8.1 release of Google App Engine, and as part of this release we’ve added a bunch of new features for the PHP runtime. The highlights of what’s new includes:

  • Three more runtime extensions:
  • The ability to include and/or require files PHP scripts from Google Cloud Storage.
  • Better support for url_stat of items in Google Cloud Storage.
  • Support for application files with ‘@’ in their name.
  • Honor the default_stream_context() when using the Google Cloud Storage stream wrapper.
  • Added the CloudStorageTools::deleteImageServiceUrl() call for the high speed image serving service.
  • Fixed a bug where $_SERVER[“PHP_SELF”] was including the query string of the URL.
  • Removed the Capabilities API.
  • Simplified the types of exceptions that can be thrown from the Users API.

Including Files From Google Cloud Storage

As mentioned, with this release we’ve made it possible to include and/or require script files from Google Cloud Storage.

To enable this feature, you must list the buckets that you want to include files from in your applications php.ini file. The bucket names can be specified as a comma separated list to the google_app_engine.allow_include_gs_buckets ini setting.

google_app_engine.allow_include_gs_buckets = "my_bucket"

Once you have configured the bucket, you can then include the files directly in your PHP application code.

<?php

require_once "gs://my_bucket/file.php";

The ideal use case for storing script files in Google Cloud Storage is for the intermediate files generated by various templating frameworks. Accessing script files from Google Cloud Storage is slower than accessing files that have been uploaded as part of you application and these files cannot take advantage of the powerful versioning system that App Engine provides.