Category Archives: php

Making Cloud Datastore Easy For PHP on App Engine

Firstly, thank you to @slangley and the team for asking me to guest post here. I really hope this helps you PHP folks on App Engine.

TL;DR – I can has code?

Sure, no problem. https://github.com/tomwalder/php-gds

Introduction

PHP on Google App Engine went Generally Available earlier this year, and that’s pretty awesome.

Certainly for me (and probably some of you) it makes both prototyping and running super-scalable web applications really easy and reliable.

Whilst the PHP part of the App Engine environment is ultra easy to use and scalable, you are likely to need a similarly featured data persistence layer for any useful applications. MySQL is cool and all, but it costs money (via Cloud SQL) and can’t scale automatically in the same way App Engine can.

Enter Cloud Datastore.

It’s pretty sweet – auto-scaling, distributed, supports transactions, mostly schema-less, managed, free to get started (up to 1GB of data and 50k queries per day).

So, you ask yourself, how do I use this amazing thing? Well until not that long ago it was a bit of a pain. you needed loads of glue code to get even the basics running.

So the php-gds library was built to simplify usage of Datastore from PHP. It tries to make it really, really easy to get started, whilst also being feature rich. It’s seeing a fair amount of adoption as far as I can tell, and the guys at Google are pretty keen on it too – hence this post!

Example – Write to Datastore

For any code running on the local development server or on a real App Engine application, it’s this easy…

// Build a new entity
$obj_book = new GDS\Entity();
$obj_book->title = 'Romeo and Juliet';
$obj_book->author = 'William Shakespeare';
$obj_book->isbn = '1840224339';

// Write it to Datastore
$obj_store = new GDS\Store('Book');
$obj_store->upsert($obj_book);

Example – Read from Datastore

Again, I try to make reading data is simple and intuitive as possible…

$obj_store = new GDS\Store('Book');
foreach($obj_store->fetchAll() as $obj_book) {
    echo "Title: {$obj_book->title}, ISBN: {$obj_book->isbn} <br />", PHP_EOL;
}

Native Access or JSON API

The php-gds library uses the native, Protocol Buffer APIs to access the Datastore – this means it is super fast.

It also supports remote access using the JSON API – in fact, you can use the same code against both, just passing the right Gateway object into your Store. More details on GitHub here.

Query Language – GQL

At the time of writing, php-gds uses the GQL, which is very similar to SQL, to query Datastore – and parameter binding, which is recommended.

$obj_book_store->fetchOne("SELECT * FROM Book WHERE isbn = @isbnNumber", [
    'isbnNumber' => '1853260304'
]);

Best Practice – Define a Schema/Model

You will get more (for less) if you define your Schema clearly, including what fields in your data to index. Indexing costs money (above the free quota) so it’s a good move to think about and define your model

More detail on defining a model can be found here.

Further Reading

There is a pretty decent set of guidance in the GitHub README and in the examples.

Support

Create an issue on GitHub or post on SO.

About Me

My name is Tom Walder, I’m CTO at Docnet, an e-commerce platform and solutions provider based in Manchester, UK. You can find me on twitter @tom_walder

I am the author of the php-gds Datastore library for PHP.

Direct File Uploads for PHP 5.5

One of the new features we announced in the 1.9.18 App Engine release is the ability to upload files directly to you application, without the need to upload the files to Google Cloud Storage first.

Direct uploads leverages the same in memory virtual filesystem that is used to provide temporary filesystem support. Direct uploads are only available with the PHP 5.5 runtime and are also limited to a maximum combined size of 32MB, which is the incoming request size limit.

Direct Upload Example

Following is a small sample application that demonstrates direct file uploads. This sample application:

  • Uses HTML5 multiple file upload support to upload image files.
  • Saves the original uploaded file into Google Cloud Storage.
  • Creates a greyscale version of the uploaded image and writes that to Google Cloud Storage.
  • Uses the image serving API to create links to serve the stored images at original size.
  • Also uses the image serving API to create thumbnail links of the images, without the application having to create the thumbnails.
  • Finally, displays a simple page with the thunbnailed links to the saved files.

For this application we’ll create 3 source files.

app.yaml

The app.yaml file has the following contents. Note that the runtime is php55.

application: php-uploads
version: 1
runtime: php55
api_version: 1
threadsafe: true

handlers:
- url: /handle_upload
  script: handle_upload.php

- url: .*
  script: direct_upload.php

direct_upload.php

This file presents a form for the user to upload multiple files to your application. As direct uploads are only in PHP 55 it checks that as well.

<?php
// Direct uploads requires PHP 5.5 on App Engine.
if (strncmp("5.5", phpversion(), strlen("5.5")) != 0) {
  die("Direct uploads require the PHP 5.5 runtime. Your runtime: " . phpversion());
}
?>
<html>
<body>
<form action="handle_upload" method="post" enctype="multipart/form-data">
  Send these files:<p/>
  <input name="userfile[]" type="file" multiple="multiple"/><p/>
  <input type="submit" value="Send files" />
</form>
</body>
</html>

handle_upload.php

The handle_upload script does the heavy lifting of creating the greyscale images, saving them in Cloud Storage and displaying the results. If you were doing this in a production app you’d need a lot more error handling, but I’ve left it out for brevity.

<?php

use google\appengine\api\cloud_storage\CloudStorageTools;

$bucket = CloudStorageTools::getDefaultGoogleStorageBucketName();
$root_path = 'gs://' . $bucket . '/' . $_SERVER["REQUEST_ID_HASH"] . '/';

$public_urls = [];
foreach($_FILES['userfile']['name'] as $idx => $name) {
  if ($_FILES['userfile']['type'][$idx] === 'image/jpeg') {
    $im = imagecreatefromjpeg($_FILES['userfile']['tmp_name'][$idx]);
    imagefilter($im, IMG_FILTER_GRAYSCALE);
    $grayscale = $root_path .  'gray/' . $name;
    imagejpeg($im, $grayscale);

    $original = $root_path . 'original/' . $name;
    move_uploaded_file($_FILES['userfile']['tmp_name'][$idx], $original);

    $public_urls[] = [
        'name' => $name,
        'original' => CloudStorageTools::getImageServingUrl($original),
        'original_thumb' => CloudStorageTools::getImageServingUrl($original, ['size' => 75]),
        'grayscale' => CloudStorageTools::getImageServingUrl($grayscale),
        'grayscale_thumb' => CloudStorageTools::getImageServingUrl($grayscale, ['size' => 75]),
    ];
  } 
}

?>
<html>
<body>
<?php
foreach($public_urls as $urls) {
  echo '<a href="' . $urls['original'] .'"><IMG src="' . $urls['original_thumb'] .'"></a> ';
  echo '<a href="' . $urls['grayscale'] .'"><IMG src="' . $urls['grayscale_thumb'] .'"></a>';
  echo '<p>';
}
?>
<p>
<a href="/">Upload More</a>
</body>
</html>

If you stumble across any problems, please let us know on either stack overflow or by creating an issue in our tracker.

File system changes in App Engine 1.9.18

In the 1.9.18 App Engine release we added a two new filesystem features to help application developers. The first is a change to the local filesystem in the development server that makes it appear to be readonly, to match how the application works in production. The second is we’ve added an in memory virtual filesystem that makes it possible to create temporary files using sys_get_temp_dir and associated functions.

Development Server Read-Only File System

We notices that a lot of new developers were caught out by the fact that the local filesystem in production is read-only, so we added a feature to mimic this behavior in the development environment. Now, by default, if your application tries to write to the local filesystem you will see an error message similar to what is shown below.

>>> file_put_contents('foo.txt', 'ttt');
file_put_contents(foo.txt): failed to open stream: Read-only file system

We know there are scenarios where you want to be able to write to the local filesystem, for example if pre-caching data before you deploy the application to production. To make this possible, we’ve added a flag that you can set in your applications php.ini file to make the filesystem read/write. Note this flag only works in the development server, the local filesystem will always be readonly in production.

To make the filesystem read/write, add the following to your php.ini file.

google_app_engine.disable_readonly_filesystem = 1

Then you’ll be able to write to the local filesystem without restriction – just remember that the production filesystem is always read-only.

Memory backed virtual filesystem

In the new PHP 5.5 runtime You can now use sys_get_temp_dir(), tmpfile() and tempnam() functions to create temporary files.

The temporary files generated are stored in an in memory virtual filesystem. The paths will start with vfs:// and look something like

>>> echo tempnam('foo', 'bar');
vfs://root/temp/foo/bar54f5751c88c520.85116141
>>> echo sys_get_temp_dir();
vfs://root/temp

The in memory filesystem will be flushed at the end of the request, and is not shared in between requests to the same instance. We’ve made it possible to call rename() with a cloud storage path as the destination so that you can write out the temporary file for permanent storage.

$tmp_name = tempnam('foo', 'bar');
$gcs_file = 'gs://my_bucket/foo/bar.txt';

file_put_contents($tmp_name, "Hello World");
rename($tmp_name, $gcs_file);
echo file_get_contents($gcs_file);

If you run into any problems with either of these new features, please let us know by filing an issue.

Update your Google API PHP Client

tl;dr – The canary process for the 1.9.17 PHP Runtime has identified an issue with the Google APIs Client Library for PHP that will cause applications to break. Update your version to at least 1.1.2 ASAP.

Hi All,

During the canary process for the new version of the PHP runtime, an issue has been uncovered related to the Google API PHP client and the cURL extension. The App Engine sockets library does not allow socket connections to www.googleapis.com, which prevents the client library from working if it uses cURL, which is it’s default configuration.

We have made a change to the client library so that it will always use PHP streams on App Engine, which prevents this error from occurring. However this means that your application code will need to be updated to the new version of the client library and the code re-deployed to the production environment.

To reduce the possibility of an outage for your application we encourage you to update this library and re-deploy your application as soon as possible.

In the meantime we continue to assess the number of affected applications to determine what the best course of action is to ensure minimal disruption for existing apps. This may well mean delaying the release of the PHP 5.5 runtime with cURL support to allow sufficient time for applications to be upgraded.

Update 28 Nov: We have decided to roll back the 1.9.17 PHP release with PHP 5.5 and cURL for the time being. You have plenty of time to update your applications to the latest API client.

gae_php

App Engine PHP 1.9.17 pre-release SDK

Well it’s been a while since the PHP team have had any updates to post to this blog. For the last few months the team has been heads down working with the rest of the App Engine team working on improving the reliability, stability and performance of the platform. You can read a little more on this in this post that our very own VP Daniel Sturman made on the App Engine discussion forum.

That being said, we’re ready to get the ball rolling once more on delivering a great PHP experience on App Engine, moving the platform from experimental to GA as soon as possible. In fact, we have the following features ready to launch in the next App Engine release, 1.9.17.

  • Upgrading the interpreter to PHP 5.5.18, which includes
    • support for Zend opcache.
    • APCu extension for in process caching.
    • backend improvements for parallel request serving from the same runtime instance.
  • Adding support for the cURL extension.
  • Adding support for the mailparse extension.

Now usually we do not pre-announce features that are scheduled in an upcoming release, because plans can change at the last minute and features can be pushed back a release or two (which could still happen here). But we regard this is a special case as the change from PHP 5.4.32 to PHP 5.5.18 could break existing applications. There is a small list of PHP 5.5 backwards incompatible changes and we want to give you developers a chance to fix any issues that your apps might have because of the switch.

To assist in identifying any potential problems in your applications, we’ve created 1.9.17 pre-release SDKs that have the PHP 5.5 interpreter bundled. You can download these SDKs from the following links.

For linux users, we suggest building the interpreter from source, using the following shell commands.

  sudo apt-get install gcc libmysqlclient-dev libxml2-dev libcurl4-openssl-dev libpng-dev libjpeg-dev
  wget --trust-server-names http://us2.php.net/get/php-5.5.18.tar.bz2/from/us1.php.net/mirror
  tar -xvf php-5.5.18.tar.bz2
  cd php-5.5.18
  DEST=$HOME/app_engine/5.5
  ./configure --prefix=$DEST/installdir \
    --enable-bcmath \
    --enable-calendar \
    --enable-ftp \
    --enable-mbstring \
    --enable-opcache \
    --enable-soap \
    --enable-sockets \
    --enable-zip \
    --disable-fileinfo \
    --disable-flatfile \
    --disable-posix \
    --with-curl \
    --with-gd \
    --with-openssl \
    --without-sqlite3 \
    --without-pdo-sqlite \
    --without-imap \
    --without-kerberos \
    --without-imap-ssl\
    --without-interbase \
    --without-ldap \
    --without-mssql \
    --without-oci8 \
    --without-pgsql \
    --without-pear \
    --disable-phar \
    --without-snmp \
    --enable-mysqlnd \
    --with-pdo-mysql=mysqlnd \
    --with-mysqli=mysqlnd \
    --with-mysql=mysqlnd
  make install -j
  cd -

This will build an php-cgi binary that is used by the App Engine development server, which can be found located in the folder $DEST/app_engine/5.5/installdir/bin. Follow these instructions to run the development appserver with the php-cgi binary that you’ve built.

As always, you can report any issues in the public issue tracker and we’ll do our best to solve them promptly.

Upcoming changes in App Engine 1.9.0 for SCRIPT_NAME and PHP_SELF

The implementation of $_SERVER[‘SCRIPT_NAME’] and $_SERVER[‘PHP_SELF’] prior to 1.9.0 differs significantly from 1.9.0 and onward. The changes were made to be consistent with the Apache implementation generally expected by PHP applications.

The following examples demonstrate the difference.

<= 1.8.9 >= 1.9.0 
app.yaml:
- url: /.*
  script: index.php
REQUEST_URI: /
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /
PHP_SELF: /		 REQUEST_URI: /
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /index.php
PHP_SELF: /index.php
REQUEST_URI: /bar
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /bar
PHP_SELF: /bar		 REQUEST_URI: /bar
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /index.php
PHP_SELF: /index.php
REQUEST_URI: /index.php/foo/bar
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /index.php/foo/bar
PHP_SELF: /index.php/foo/bar		 REQUEST_URI: /index.php/foo/bar
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /index.php
PHP_SELF: /index.php/foo/bar
REQUEST_URI: /test.php/foo/bar
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /test.php/foo/bar
PHP_SELF: /test.php/foo/bar		 REQUEST_URI: /test.php/foo/bar
SCRIPT_FILENAME: /path/to/index.php
SCRIPT_NAME: /index.php
PHP_SELF: /index.php		 
app.yaml:
- url: /.*
  script: foo/index.php
REQUEST_URI: /bar
SCRIPT_FILENAME: /path/to/foo/index.php
SCRIPT_NAME: /bar
PHP_SELF: /bar		 REQUEST_URI: /bar
SCRIPT_FILENAME: /path/to/foo/index.php
SCRIPT_NAME: /foo/index.php
PHP_SELF: /foo/index.php

Note the odd behavior of PHP_SELF which is consistent with PHP on Apache.

Many frameworks and applications depend on SCRIPT_NAME to determine if the application is running within a subdirectory, typically within an Apache virtual host, and as such will not be affected by this change. On the other hand this may break custom applications or frameworks that have been configured to work with the previous implementation.

PHP App Engine Apps and File System Concepts

If you’re new to App Engine, then the file system model may be a bit different from what you might have experienced using other hosts.

With an App Engine application, you cannot write to the file-system where your application is deployed. Your application can read any files from the deployed directory structure, but it can’t write to that file-system. Instead, the application can use Google Cloud Storage (GCS) for both reading and writing files.
To convert an app to use GCS for writable files, here are the primary things that you will typically need to do:

Another implication of the read-only file system is that if your app has a ‘plugin’ model, like WordPress, you can’t install or update plugins from your deployed app. Instead, you will need to install or update any plugins locally, then redeploy the app.

You can find lots more info on all of the following in the documentation for the PHP runtime.

Continue reading →

App Engine WordPress plugin Version 1.2 released

We’re happy to announce that we’ve just released version 1.2 of the App Engine WordPress plugin. The new version includes the following changes

  • Fix for the bug “Uncaught exception ‘InvalidArgumentException’ with message ‘max_bytes_per_blob must be an integer'” (forum link)
  • Removes the need for PIL to be installed in the local development environment.
  • Removes the need for PyCrypto to be installed in the local development environment.
  • Better checking if the application has write access to the Google Cloud Storage bucket for uploads.

If you find any issues with the latest version of the plugin you can file them in the support forum.

We also gladly accept pull requests – you can find the plugin source on Github.

Generating Dynamic WordPress sitemaps on App Engine

Sitemaps are a valuable tool in helping search engines crawl your website. In this post I’ll show you how you can have a dynamically generated sitemap for a WordPress blog hosted on Google App Engine.

Because of the read-only nature of the file system on App Engine, we’ll use Google Cloud Storage to store the generated sitemap file. We’ll also add some handlers to serve the sitemap, and to update it as part of a scheduled cron job.

Continue reading →

Getting started with the Cloud Datastore on PHP App Engine, Part II

Introduction

This is the second in a series of posts on using the Google Cloud Datastore on PHP App Engine.
The code for the examples in this post is here: https://github.com/amygdala/appengine_php_datastore_example.

The previous post walked through the process of setting up your PHP App Engine app to connect with the Datastore, using the Google API client libraries.
It demonstrated how to create a test Datastore entity, and showed how to set up a “Datastore service” instance, using a DatastoreService class, to make it easier to send requests.

However, using the API client libraries, there is a fair bit of overhead in composing requests to the Datastore and processing the responses, and mapping entities to instances of classes in your app. So, in this post, we’ll define some classes to make it easier to create, update, and fetch Datastore entities.
We’ll define a `Model` base class, then show an example of subclassing it to map to a specific Datastore Kind.

Continue reading →