• Our Code
• In Depth
• Android
• iOS

Queued tasks on App Engine for Firebase

App Engine

App Engine is a fully managed platform which hosts your server side code. It allows you to build scalable web and mobile backends in many supported languages. There are two different environments available. Flexible and standard. The flexible environment automatically scales your app up and down while balancing the load. In addition, it allows you to customize the runtime and operating system of your virtual machine.

The standard environment is based on container instances running on Google's infrastructure. Containers are preconfigured with one of several available runtimes. For our use case, we wanted to write a reasonably straight forward java backend, so standard was all we needed. (Help choosing - https://cloud.google.com/appengine/docs/the-appengine-environments).

Within the standard environment, you have a choice of how you want your container instances to scale (startup and shutdown). Automatic Scaling is based on request rate, response latencies, and other application metrics. Manual Scaling allows your container instances to run continuously. This means you can perform more complex initialization and rely on the state of its memory over time. Basic Scaling will create an instance when the application receives a request. The instance will be turned down when the app becomes idle.

Basic scaling is ideal for work that is intermittent or driven by user activity, making it the perfect choice for us.

You also have the choice over what spec your instances are. There is a performance vs price balance suitable for most. (https://cloud.google.com/appengine/docs/standard/#instance_classes https://cloud.google.com/appengine/pricing)

How to get started

We started by creating an app engine module within an Android project in Android Studio. File -> New -> New Module… Click Google Cloud Module and then Next. Module type: App Engine Java Servlet Module, Module name: backend, Package name: com.example.myapp.backend, Client module: app (your mobile app although not important at this stage). Then click Finish. This will create an app engine module for you. When you explore what this creates, you will find a MyServlet.class. This HttpServlet accepts a POST request, with parameter ‘name’ and responds with ‘Hello [name]’. Starting from this simple servlet should give you an idea of how to create a function for your use case.

Set up

As you have already seen, you can use Android Studio to develop an app engine module, which is helpful for mobile developers as it’s a familiar environment. You need a couple of other things set up though. First follow only the ‘Before you begin’ section on this page: https://cloud.google.com/endpoints/docs/frameworks/java/get-started-frameworks-java#before-you-begin. Now, to ensure your $PATH variable is set up (mac) run the following command:

touch ~/.bash_profile; open ~/.bash_profile

I am using zsh (http://ohmyz.sh/) for which you will need:

touch ~/.zshrc; open ~/.zshrc

And then ensure you have the following in that file:

export PATH=$PATH:/Users/name/...pathToSDK/...

To deploy your app engine code:

./gradlew build
./gradlew appengineDeploy

In it for the long haul

App Engine does a great job at handling long running tasks. It has the concept of a queue and tasks. A queue will work through a list of tasks. A task is a HttpServlet as seen in the example above.

You will need to create a queue configuration file, queue.xml, under the WEB-INF folder which might look something like this:

<queue-entries>
 <queue>
 <name>queue-1</name>
 <rate>500/s</rate>
 <bucket-size>500</bucket-size>
 <retry-parameters>
 <task-age-limit>1d</task-age-limit>
 </retry-parameters>
 </queue>
</queue-entries>

All options can be found here https://cloud.google.com/appengine/docs/standard/java/config/queueref

Queue queue = QueueFactory.getQueue("queue-1");

This line will give you a queue object which you can post tasks to.

queue.addAsync(TaskOptions.Builder.withUrl("/task/task1")
 .param("PARAM1", "info1")
 .param("PARAM2", "info2")
 );

To post a task call addAsync(), using a TaskOptions.Builder. The task url is defined in your web.xml file.

From one HttpServlet you would add tasks into a queue. You then call this “enqueuing” HttpServlet when you wish to kick this off.

To deploy your queue configuration changes:

./gradlew appengineDeployQueue

How to call from Firebase Cloud Function

We used our existing Firebase Cloud Functions to kick off one of these “enqueuing” servlets whenever we detect a change to a file in the Firebase Cloud Storage. This was a specific requirement on a recent project to serve data to an app from regularly updated text files.

const functions = require('firebase-functions'); 
const request = require('request-promise-native'); 
const path = require('path');

exports.startEnqueuing = functions.storage.object().onChange(event => {
 const file = event.data; // The Storage object.
 const fileBucket = file.bucket; // The Storage bucket that contains the file.
 const filePath = file.name; // File path in the bucket.
 const contentType = file.contentType; // File content type.
 const fileName = path.basename(filePath); // Get the file name.

 if (fileName != "FileOfInterest.txt") {
 return;
 }
 console.log("CALLING APP ENGINE");

 let url = "https://project-id.appspot.com/enqueue/start-enqueuing";

 return request(url)
 .then(function(responseStr) {
 console.log(responseStr);
 })
 .catch(function(error) {
 console.error(error);
 });
});

Security Issues

Be warned - these servlets are insecure by default. You can secure these servlets with google accounts via a web user interface, but it isn’t possible to secure them without any ui, in and of themselves. You can solve this problem using Google Cloud Endpoints - click here for more information.