If you’re using Firebase with a server to perform tasks like generating thumbnails of images, filtering message contents and censoring data, or fanning data out to multiple locations in your Firebase database, chances are you’re using (or need to use) Firebase Queue.

Firebase Queue is a fault-tolerant multi-worker job pipeline built on Firebase. It provides you with a way to organize workers or perform background work on your Firebase database or files.

This isn’t a tutorial on how to use firebase-queue, it is simply to show you how you can host & run your queue workers in a scalable environment (Google app engine). If you’re want to learn how to use Firebase Queues, there’s a great guide here.

Note: Firebase-queue is only available for Node.js, so we shall be using the App Engine Node.js flexible runtime.

Why Google Appengine?

Google app engine provides an easy to use platform for developers to build, deploy, manage and automatically scale services on Google’s infrastructure. There are no servers for you to provision or maintain.

Installing

Assuming you’ve already installed Node.js, create a directory to hold your application, and make that your working directory.

$ mkdir myapp
$ cd myapp

Use the npm init command to create a package.json file for your application. For more information on how package.json works, see Specifics of npm’s package.json handling.

$ npm init

This command prompts you for a number of things, such as the name and version of your application. For now, you can simply hit RETURN to accept the defaults for most of them, with the following exception:

entry point: (index.js)

Enter app.js, or whatever you want the name of the main file to be. If you want it to be index.js, hit RETURN to accept the suggested default file name.

Now install Express, Firebase & Firebase-Queue dependencies in the myapp directory and save them in the dependencies list.

$ npm install express --save
$ npm install firebase firebase-queue --save

Next you need to install Google Cloud SDK, in order to be able to deploy to app engine.

Coding the app

In the myapp directory, in the app.js file you created above, start your queue worker as shown below:

var express = require('express');
var app = express();

var firebase = require("firebase");
firebase.initializeApp({
  serviceAccount: "service_account.json",
  databaseURL: "https://databaseName.firebaseio.com"
});

var Queue = require('firebase-queue')


//start your worker here
var ref = firebase.database().ref('queue');
var queue = new Queue(ref, function(data, progress, resolve, reject) {
  // Read and process task data
  console.log(data);

  // Do some work
  progress(50);

  // Finish the task asynchronously
  setTimeout(function() {
    resolve();
  }, 1000);
});


app.get('/', function (req, res) {
  res.send('The Firebase Queue worker is running!');
});

var server = app.listen(process.env.PORT || '8080', function () {
  console.log('App listening on port %s', server.address().port);
  console.log('Press Ctrl+C to quit.');
});

The above code starts a queue worker which simply logs the task data passed to it, and then starts an express server. This worker will continue running for as long as the instance is running.

Next, you need to create an app.yaml file in the myapp directory, which will be used to deploy the app to appengine.

runtime: nodejs
vm: true

Before we can deploy, we need to edit the "scripts" and "engines" properties in the package.json file so that our app can run on app engine.

"engines": {
    "node": "~4.2"
  },
"scripts": {
    "start": "node app.js",
    "monitor": "nodemon app.js",
    "deploy": "gcloud app deploy"
  }

Finally, deploy the app to appengine:

$ gcloud app deploy app.yaml

To test the worker, you can push an object with some data to the tasks subtree of your queue:

# Using curl in shell
$ curl -X POST -d '{"foo": "bar"}' https://databaseName.firebaseio.com/queue/tasks.json

You should see ‘{“foo”: “bar”}’ somewhere in your logs, or which ever data you posted.

Although the app.js code above initialized only one worker, Queue Workers can take options to specify the number of initial workers to run simultaneously for a single node.js thread.

var options = {
  'specId': 'spec_1',
  'numWorkers': 5,
};

var queue = new Queue(ref, options, function(data, progress, resolve, reject) {
  ...
});

The above code initializes 5 workers on a single thread. Depending on your application needs and CPU resources, you may want to initalize multiple workers.

Scaling on App engine

Every app engine application is auto-scaled by default basing on certain default parameters. However, you can control how/when you want your app to be scaled. You can use either auto or manual scaling, depending in your application needs.

Here’s an example of how to configure your app.yaml for auto-scaling:

#auto scaling
automatic_scaling:
  min_num_instances: 1
  max_num_instances: 10
  cool_down_period_sec: 120 # default value
  cpu_utilization:
    target_utilization: 0.75

You only need to specify the minimum and maximum number of instances when using auto-scaling, the rest are optional. App engine will average CPU use across running instances. The value your specify for target_utilization is used to determine whether to reduce or increase the number of instances.

Manual scaling is suitable if you can reliably predict your worker’s load @ all times. Auto-scaling is more suitable if your app (queue workers) experiences spikes in usage. See example below:

#manual scaling
manual_scaling:
  instances: 3

You can also specify the computing power/resources for each instance, depending on your needs:

resources:
  cpu: .5
  memory_gb: 1.3
  disk_size_gb: 10

cpu represents the number of cores (it can be less than a fraction). memory_gb is RAM in GB.

The beauty of Firebase-Queue is that multiple queue workers can be initialized on multiple machines and Firebase-Queue will ensure that only one worker is processing a single queue task at a time. So using Google App Engine is a nice way to autoscale your firebase worker-queues.

And that’s how you can effortlessly scale firebase queue workers with Google App engine. Got any questions? Ping me on Twitter: @TimAsiimwe, I’d be happy to chat!

Note: In order to use Flexible VMs, you need to enable billing for your project in the Google Cloud Console.

For more on how to configure your app engine app, read the documentation.

References: https://expressjs.com/en/starter/installing.html https://github.com/firebase/firebase-queue/blob/master/docs/guide.md