Writing an API - a mobile developer story

Last Updated: 17 Aug 2023

Alistair Sykes

This blog post is part of the Writing An API series

How a mobile app developer writes an API 1

This in-depth blog series offers a step-by-step guide for mobile development teams.

A new project

Recently, I had a project which required me to create a backend. As it’s not usually my area of expertise, I relished the chance of learning something new and getting stuck into some new platforms and technologies.

The scope of the backend was a relatively simple concept - to take some regularly updated text files and serve them to an app.

Reality sets in

As it transpired it took me five attempts to nail the functionality of that simple concept. But as any software developer will tell you, trial and error is part and parcel of our job. Everyday I try things out, look at the results and adapt.

In the spirit of knowledge sharing, I wanted to write about my experience of creating this backend from my perspective as an Android developer. Along the way, I’ve written a few supporting articles to explain in more detail certain things I learnt.

Let’s get started.

1. I know Firebase… right?

I am familiar with the Firebase platform, having used various parts of its offering. Firebase Database offers an amazing SDK. It handles syncing and offline support with little boilerplate code on the client side. This project felt like a great opportunity to utilise Firebase, so I started there.

Firebase Database would hold the data I need and Cloud Functions would provide the mechanism for reading in a txt file and populating/updating the database.

Firebase Database

Firebase Realtime Database is a cloud-hosted database. Data is stored as JSON and synchronised in real-time to every connected client. Designing data structures for this database can be very difficult. To make the data queryable, you need to de-normalise. For more details, we've written a post dedicated to explaining Firebase Database.

Firebase Cloud Functions

Firebase Cloud Functions are a mechanism of running backend code (written in JavaScript) which can be triggered in various ways. In this instance, I wanted to trigger functions based on a new file, so opted to use them in combination with Firebase Cloud Storage. Whenever a file gets uploaded my code is triggered.

I’ve written a lot more about how I triggered those cloud functions in this article.

Failed :(

Unfortunately, this approach quickly hit issues.

I began to realise that this wasn’t going to work when I started trying to add bigger datasets. I began to hit timeout issues. Firebase cloud functions only allow for up to 540 secs per function (default of 60 secs). To combat this issue I began exploring running promises (Javascript concept) in parallel. Although parallelism is possible, when trying to handle errors correctly, and working within those timeouts, it appeared this was never going to work for large datasets. I needed a more fully fledged backend, able to run tasks within queues.

2. More Google must be better…

I decided to give the Google Cloud Platform a go since I was already using Firebase and wanted something that would integrate quickly and easily with our existing database. I wanted something that was reasonably out of the box, without any server config.

I spent a lot of time setting up App Engine and working out how to queue tasks with Firebase.

Failed :(

Although at first this seemed great and a workable solution, I started looking at how to secure the servlets so that no one else could call them. You can secure these servlets with Google accounts via a web user interface, but it isn’t possible to secure them without any UI. I needed a more formalised and securable API.

3. Adding Endpoints

Next, I began to explore Google Cloud Endpoints as a way of adding a layer on top of the servlets to protect them. I wanted to wrap our servlets in a clear API which could only be called by authorised clients. Take a look at how to do this in this posts, 'Using Google Cloud Endpoints'.

Cloud Endpoints

Cloud Endpoints is a distributed API management system. It provides an API console, hosting, logging, monitoring and other features to help you create, share, maintain, and secure your APIs. (https://cloud.google.com/endpoints/)

Failed :(

Ok, I finally have all the data in a Firebase Database, using update API’s which are secure. So what’s the problem?

I came across two issues which instigated a bit of a pivot for us.

One, a key selling point of App Engine was the instance scaling mentioned above. Since our text files will be updated once a day we will have high load once a day. The rest of the time the server doesn’t need to be running. Unfortunately App Engine, when using Firebase SDK, does not support this scaling (https://cloud.google.com/solutions/mobile/firebase-app-engine-android-studio#configuring_the_app_engine_backend_to_use_manual_scaling).

A potential solution for this was to switch to using the REST API, instead of the SDK. This, however, wasn’t feasible because of the HTTP limits imposed on App Engine (https://cloud.google.com/appengine/quotas) and the large number of requests required to replace the SDK.

The other and more well known issue with Firebase Database is its queriability. Doing queries, which in SQL are simple, are notoriously difficult in Firebase. I spent a long time planning our data structure, de-normalising and flattening, but still couldn’t achieve the queries I needed. With that in mind, there were two options. Grab all the data and query it ourselves client side, or integrate with something like ElasticSearch. I experimented with client-side queries, but as suspected, the dataset was just too large to do this at the kind speed our users required. The prospect of integrating with ElasticSearch was not an exciting one and seemed like a time expense we didn’t want.

4. Bye, bye Firebase

It was time for a bold decision to step away from Firebase Database. I began to look for a more familiar SQL option. However, this came with another challenge. I would also need a client-facing API for the client to interface with this SQL database.

I chose to use Google Cloud SQL since we were already within that ecosystem. It integrates nicely with App Engine and would be simpler from a billing perspective.

Cloud SQL

Cloud SQL is a fully-managed database service. It makes it easy to set up, maintain, manage, and administer your relational MySQL database in the cloud.

Check out our post about how we implemented and populated the Cloud SQL database and also about how we solved the need for User Authentication.

Failed :(

Seriously, how is this not the finished solution? Well instances, even without the Firebase SDK, still weren’t scaling as I'd expect. I’ve written about that issue and how to understand app engine scaling.

5. Getting there…

App engine still wasn’t scaling as I expected. I had removed the Firebase SDK and, therefore, was expecting the instances to scale according to the basic scaling config.

After long investigations and consulting with Google support, I worked out that logging, of all things, was keeping background threads running.

The Google support team suggested I move to automatic-scaling because it does not support background threads at all. This means that no background thread nonsense could stop our instances from scaling down. Although it felt like a workaround, I took on their suggestion.

This solved it!

Conclusion

A combination of app engine (with automatic scaling) and cloud SQL means you can have a scaling backend allowing you to support high load, infrequent tasks for relatively low costs. Using a Google Cloud solution means you get a stable platform with support available.

I am happy with our solution and have built two great apps off the back of this backend which have been successful. It’s not the end of the road for my backend learning by any means. As with everything here at Brightec, we are regularly looking to continuously improve what we do and how we do it.

Advice

Having shared my story, I’d like to offer a suggestion on how to get started with a new project.

Starting from this sample (https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/appengine/endpoints-frameworks-v2/backend) will give you the groundwork for a successful backend. Download it and use it as the base for your project.
Switch to automatic scaling straight away. Only move to basic if you find a reason why you need it.
Be wary of the Firebase/App Engine combo. This is a real shame, as I love the Firebase platform. But we need scaling to be out of the box so that App Engine is cost effective for our clients.
The Google documentation on its cloud platform is extensive but not very organised. It’s very easy to get lost. Try to keep a record of the exact pages that you use or find helpful.

A note on Firebase Cloud Firestore

Cloud Firestore is the new database recently released by the Firebase team. Like Firebase Realtime Database, it keeps your data in sync across client apps and offers offline support. Would this have helped with our project? The only component this solves is the queriability of the database. The Firebase team has worked really hard to try to solve the more fuzzy but common queries we all love from SQL. However, it still requires background threads on App Engine. So it will still stop scaling from working as intended.

Sadly, Firebase is still a no-go when it comes to App Engine with scaling.