Scheduling Google Compute Instances with Cloud Scheduler

November 18, 2019

Scheduling Google Compute Instances with Cloud Scheduler

Our company needs to save some cost for the running staging instances. Since the instance and any related resources are used only when we are at work, we need the instance to be available only at working hours, or from Monday to Friday, 9 to 5. It approximately reduces our cost to roughly 45%. A huge diff.

The Ingredients

We need the following components from Google Cloud:

Google Cloud Scheduler: to run a function at an exact day/time.
Google Cloud Functions: to start and stop our compute instances.
Google Cloud Pub/Sub: as a bridge or messenger from the Cloud Scheduler to the Cloud Functions.
Lastly, the Google Cloud Compute Engine Instances.

Each of those will cost us money, but the good news is we will work under the fair usage policy so it can be said that all of those will be free of charge. For example, now Google Cloud gives us 93x each month for Cloud Functions. That means if we only use it twice a day (to start and stop the instance), it will cost us nothing.

We also can use the Google Cloud HTTP Functions as an alternative for the Google Cloud Pub/Sub, but notice that there is no authentication so basically it is less secure than the Pub/Sub component.

Prepare the Compute Engine Instances

Create our compute engine instance, or edit the existing ones. You can work at any zone location, specs or anything. The most important things for this tutorial is the label of our instances should be "env=dev". You can use any value but make sure it is unique enough because we need the label as an identifier.

Setup Google Cloud Functions with Cloud Pub/Sub

Go to your Google Cloud Functions page, and create a new function. This function will be used to start the instance.

Set the name to "startInstancePubSub"
Set the 'Trigger" to "Cloud Pub/Sub"
Set the Topic to "start-instance-event"
Set "Runtime" to "Node.js 8"

IMPORTANT. Set the starter code with the following code. The current code used as an example at the official page is obsolete for node.js 8. This way, we will start all instance with "env=dev" label. We don't care about the zone, etc.

Here is the content of our index.js.

/**
 * Copyright 2018, Google, Inc.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

// [START functions_start_instance_pubsub]
// [START functions_stop_instance_pubsub]
const Buffer = require('safe-buffer').Buffer;
const Compute = require('@google-cloud/compute');
const compute = new Compute();
// [END functions_stop_instance_pubsub]

/**
 * Starts a Compute Engine instance.
 *
 * Expects a PubSub message with JSON-formatted event data containing the
 * following attributes:
 *  zone - the GCP zone the instances are located in.
 *  label - the label of instances to start.
 *
 * @param {!object} event Cloud Function PubSub message event.
 * @param {!object} callback Cloud Function PubSub callback indicating
 *  completion.
 */
exports.startInstancePubSub = (event, context, callback) => {
  try {
    const payload = _validatePayload(
      JSON.parse(Buffer.from(event.data, 'base64').toString())
    );
    const options = {filter: `labels.${payload.label}`};
    compute.getVMs(options).then(vms => {
      vms[0].forEach(instance => {
          compute
            .zone(instance.zone.id)
            .vm(instance.name)
            .start()
            .then(data => {
              // Operation pending.
              const operation = data[0];
              return operation.promise();
            })
            .then(() => {
              // Operation complete. Instance successfully started.
              const message = 'Successfully started instance ' + instance.name;
              console.log(message);
              callback(null, message);
            })
            .catch(err => {
              console.log(err);
              callback(err);
            });
      });
    });
  } catch (err) {
    console.log(err);
    callback(err);
  }
};
// [END functions_start_instance_pubsub]

// [START functions_start_instance_pubsub]

/**
 * Validates that a request payload contains the expected fields.
 *
 * @param {!object} payload the request payload to validate.
 * @return {!object} the payload object.
 */
function _validatePayload(payload) {
//   if (!payload.zone) {
//     throw new Error(`Attribute 'zone' missing from payload`);
//   } else 
  if (!payload.label) {
    throw new Error(`Attribute 'label' missing from payload`);
  }
  return payload;
}
// [END functions_start_instance_pubsub]
// [END functions_stop_instance_pubsub]

At the package.json tab, copy and paste the following code:

{
  "name": "cloud-functions-schedule-instance",
  "version": "0.0.2",
  "private": true,
  "license": "Apache-2.0",
  "author": "Google Inc.",
  "repository": {
    "type": "git",
    "url": "https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git"
  },
  "engines": {
    "node": ">=8.13.0"
  },
  "scripts": {
    "test": "mocha test/*.test.js --timeout=20000"
  },
  "devDependencies": {
    "@google-cloud/nodejs-repo-tools": "^3.3.0",
    "mocha": "^6.0.0",
    "proxyquire": "^2.0.0",
    "sinon": "^7.0.0"
  },
  "dependencies": {
    "@google-cloud/compute": "^0.12.0",
    "safe-buffer": "^5.1.2"
  }
}

Theoretically, the code will

Search for any instances by label (our label is "env=dev"). It is not limited to 1 instance.
Start the instances.

I think it doesn't matter if we have no experience using node.js before.

For "Function to execute", enter "startInstancePubSub"

Next, we are going to set the "stop instance function"

Create another function named "stopInstancePubSub"
Set everything else the same way we set up the start instance function above.
For pub-sub topic, set it to "stop-instance-event"

Here is our index.js code.

/**
 * Copyright 2018, Google, Inc.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

// [START functions_start_instance_pubsub]
// [START functions_stop_instance_pubsub]
const Buffer = require('safe-buffer').Buffer;
const Compute = require('@google-cloud/compute');
const compute = new Compute();
// [END functions_stop_instance_pubsub]

/**
 * Starts a Compute Engine instance.
 *
 * Expects a PubSub message with JSON-formatted event data containing the
 * following attributes:
 *  zone - the GCP zone the instances are located in.
 *  label - the label of instances to start.
 *
 * @param {!object} event Cloud Function PubSub message event.
 * @param {!object} callback Cloud Function PubSub callback indicating
 *  completion.
 */
exports.startInstancePubSub = (event, context, callback) => {
  try {
    const payload = _validatePayload(
      JSON.parse(Buffer.from(event.data, 'base64').toString())
    );
    const options = {filter: `labels.${payload.label}`};
    compute.getVMs(options).then(vms => {
      vms[0].forEach(instance => {
          compute
            .zone(instance.zone.id)
            .vm(instance.name)
            .start()
            .then(data => {
              // Operation pending.
              const operation = data[0];
              return operation.promise();
            })
            .then(() => {
              // Operation complete. Instance successfully started.
              const message = 'Successfully started instance ' + instance.name;
              console.log(message);
              callback(null, message);
            })
            .catch(err => {
              console.log(err);
              callback(err);
            });
      });
    });
  } catch (err) {
    console.log(err);
    callback(err);
  }
};
// [END functions_start_instance_pubsub]
// [START functions_stop_instance_pubsub]

/**
 * Stops a Compute Engine instance.
 *
 * Expects a PubSub message with JSON-formatted event data containing the
 * following attributes:
 *  zone - the GCP zone the instances are located in.
 *  instance - the name of a single instance.
 *  label - the label of instances to start.
 *
 * Exactly one of instance or label must be specified.
 *
 * @param {!object} event Cloud Function PubSub message event.
 * @param {!object} callback Cloud Function PubSub callback indicating completion.
 */
exports.stopInstancePubSub = (event, context, callback) => {
  try {
    const payload = _validatePayload(
      JSON.parse(Buffer.from(event.data, 'base64').toString())
    );
    const options = {filter: `labels.${payload.label}`};
    compute.getVMs(options).then(vms => {
      vms[0].forEach(instance => {
          compute
            .zone(instance.zone.id)
            .vm(instance.name)
            .stop()
            .then(data => {
              // Operation pending.
              const operation = data[0];
              return operation.promise();
            })
            .then(() => {
              // Operation complete. Instance successfully stopped.
              const message = 'Successfully stopped instance ' + instance.name;
              console.log(message);
              callback(null, message);
            })
            .catch(err => {
              console.log(err);
              callback(err);
            });
      });
    });
  } catch (err) {
    console.log(err);
    callback(err);
  }
};
// [START functions_start_instance_pubsub]

/**
 * Validates that a request payload contains the expected fields.
 *
 * @param {!object} payload the request payload to validate.
 * @return {!object} the payload object.
 */
function _validatePayload(payload) {
  // ignore the zone
  //  if (!payload.zone) {
  //    throw new Error(`Attribute 'zone' missing from payload`);
  //  } else 
  if (!payload.label) {
    throw new Error(`Attribute 'label' missing from payload`);
  }
  return payload;
}
// [END functions_start_instance_pubsub]
// [END functions_stop_instance_pubsub]

Then, replace the package.json tab to the following code:

{
  "name": "cloud-functions-schedule-instance",
  "version": "0.0.2",
  "private": true,
  "license": "Apache-2.0",
  "author": "Google Inc.",
  "repository": {
    "type": "git",
    "url": "https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git"
  },
  "engines": {
    "node": ">=8.13.0"
  },
  "scripts": {
    "test": "mocha test/*.test.js --timeout=20000"
  },
  "devDependencies": {
    "@google-cloud/nodejs-repo-tools": "^3.3.0",
    "mocha": "^6.0.0",
    "proxyquire": "^2.0.0",
    "sinon": "^7.0.0"
  },
  "dependencies": {
    "@google-cloud/compute": "^0.12.0",
    "safe-buffer": "^5.1.2"
  }
}

Then, we need to set "Function to Execute" to "stopInstancePubSub".

We can optionally test the function using the following Triggering Event:

We can use this command on our console to make the base64-encoding.

echo '{"label":"env=dev"}' | base64
eyJsYWJlbCI6ImVudj1kZXYifQo=

Input the following string into the "Triggering Event" textarea :

{"data":"eyJsYWJlbCI6ImVudj1kZXYifQo="}

Setup the Google Cloud Scheduler

Our functions will not work if we don't trigger them. That's why we need the Google Cloud Scheduler component: to trigger our start/stop functions at an exact day/time.

Go to the Google Scheduler page and create a new job named "startup-dev-instances".
Set the Frequency to your preference. 0 9 * * 1-5. That means we will start the instance from the 1st day of the week to 5th day of the week (1-5), at 9 AM (0 9).
Select our related PubSub topic as the Target
For payload, enter the following:
{"label":"env=dev"}

Then, create the exact job as the start one, except for the name of the scheduler and Frequency.

Set the Frequency to 0 17 * * 1-5. That's it, we just create a scheduler to call the PubSub topic from Monday to Friday, at 17:00 or 5 PM.
That's it. We are almost done.

Bear in mind that the functions will run at the exact time, but with a random order if we have multiple instances to be started/stopped. In our case, we need to start the A instance first, before the B and C running. That way, I create another Google Scheduler which starts 15 minutes earlier to the B and C instances.

We can modify the flow or the starter code to be more suitable to our needs. I hope you enjoy the article. Don't hesitate to ask a question and give me your comment below. Thank you!

Search This Blog

Prabowo Murti