Don’t save data to the scheduling server

I would suggest to not save or use data in the same place you are doing the scheduling. Use a service like BigQuery (bigQueryR) or googleCloudStorageR (googleCloudStorageR) to first load any necessary data, do your work then save it out again. This may be a bit more complicated to set up, but will save you tears if the VM or service goes down - you still have your data.

To help with this, on Google Cloud you can authenticate with the same details you used to launch a VM to authenticate with the storage services above (as all are covered under the http://www.googleapis.com/auth/cloud-services scope) - you can access this auth when on a GCE VM in R via googleAuthR::gar_gce_auth()

An example skeleton script is shown below that may be something you are scheduling.

It downloads authentication files, does an API call, then saves it up to the cloud again:

library(googleAuthR)
library(googleCloudStorageR)

gcs_global_bucket("my-bucket")
## auth on the VM
options(googleAuthR.scopes.selected = "https://www.googleapis.com/auth/cloud-platform")
gar_gce_auth()

## use the GCS auth to download the auth files for your API
auth_file <- "auth/my_auth_file.json"
gcs_get_object(auth_file, saveToDisk = TRUE)

## now auth with the file you just download
gar_auth_service(auth_file)

## do your work with APIs etc.
.....

## upload results back up to GCS (or BigQuery, etc.)
gcs_upload(my_results, name = "results/my_results.csv")

Set up a schedule for logs too

Logs are important for scheduled jobs, so you have some idea on whats happened when things go wrong. To help with scheduling debugging, most googleAuthR packages now have a timestamp on their output messages.

You can send the output of your scripts to log files, if using cron and RScript it looks something like this:

RScript /your-r-script.R > your-r-script.log

…where > sends the output to the new file.

Over time though, this can get big and (sigh) fill up your disk so you can’t log in to the VM (speaking from experience here!) so I now set up another scheduled job that every week takes the logs and uploads to GCS, then deletes the current ones.

Consider using Docker for environments

Several of the methods below use Docker.

The reasons for that is Docker provides a nice reproducible way to define exactly what packages and dependencies you need for your script to run, which can run on top of any type of infrastructure as Docker has quickly become a cloud standard.

For instance, migrating from Google Cloud to AWS is much easier if both can be deployed using Docker, and below Docker is instrumental in allowing you to run on multiple solutions.

Bear in mind that when a Docker container relaunches it won’t save any data, so any non-saved state will be lost (you should make a new container if you need it to contain data), but you’re not saving your data to the docker container anyway, aren’t you?

Handy tools for Docker - containerit and Build Triggers

Here we introduce Docker images, which may have been more a technical barrier for some before (but worth knowing, I think)

Recipe

Putting it all together then, documentation of this workflow for scheduling R scripts is found here.

1. If you don’t already have one, start up a scheduler VM using gce_vm_scheduler
2. Create a Dockerfile either manually or using containerit that will run your script upon execution
3. Upload the Dockerfile to a git repo (private or public)
4. Setup a build trigger for that Dockerfile
5. Once built, set a script to schedule within that Dockerfile with gce_schedule_docker

This is still in beta at time of writing but should be stable by the time googlecomputeEngineR hits CRAN 0.2.0.

Setting up your script for App Engine

There are some requirements for the container that need configuring so it can run:

• You can not use googleAuthR::gar_gce_auth() so will need to upload the auth token within the Dockerfile.
  
• AppEngine expects a web service to be listening on port 8080, so your schedule script needs to be triggered via HTTP requests.

For authentication, I use the system environment arguments (i.e. those usually set in .Renviron) that googleAuthR packages use for auto-authentication. Put the auth file (such as JSON or a .httr-oauth file) into the deployment folder, then point to its location via specifying in the app.yaml. Details below.

To solve the need for being a webservice on port 8080 (which is then proxied to normal webports 80/443), plumber is a great service by Jeff Allen of RStudio, which already comes with its own Docker solution. You can then modify that Dockerfile slightly so that it works on App Engine.

Recipe

To then schedule your R script on app engine, follow the guide below, first making sure you have setup the gcloud CLI.

1. Create a Google Appengine project in the US region (only region that supports flexible containers at the moment)
2. Create a scheduled script e.g. schedule.R - you can use auth from environment files specified in app.yaml.
3. Make an API out of the script by using plumber - example:

library(googleAuthR)         ## authentication
library(googleCloudStorageR)  ## google cloud storage
library(readr)                ## 
## gcs auto authenticated via environment file 
## pointed to via sys.env GCS_AUTH_FILE

#* @get /demoR
demoScheduleAPI <- function(){
  
  ## download or do something
  something <- tryCatch({
      gcs_get_object("schedule/test.csv", 
                     bucket = "mark-edmondson-public-files")
    }, error = function(ex) {
      NULL
    })
      
  something_else <- data.frame(X1 = 1,
                               time = Sys.time(), 
                               blah = paste(sample(letters, 10, replace = TRUE), collapse = ""))
  something <- rbind(something, something_else)
  
  tmp <- tempfile(fileext = ".csv")
  on.exit(unlink(tmp))
  write.csv(something, file = tmp, row.names = FALSE)
  ## upload something
  gcs_upload(tmp, 
             bucket = "mark-edmondson-public-files", 
             name = "schedule/test.csv")
  
  message("Done", Sys.time())
}

4. Create Dockerfile. If using containerit then replace FROM with trestletech/plumber and add the below lines to use correct AppEngine port:

Example:

library(containerit)

dockerfile <- dockerfile("schedule.R", copy = "script_dir", soft = TRUE)
write(dockerfile, file = "Dockerfile")

Then change/add these lines to the created Dockerfile:

EXPOSE 8080
ENTRYPOINT ["R", "-e", "pr <- plumber::plumb(commandArgs()[4]); pr$run(host='0.0.0.0', port=8080)"]
CMD ["schedule.R"]

Example final Dockerfile below. This doesn’t need to be built in say a build trigger as its built upon app engine deployment.

FROM trestletech/plumber
LABEL maintainer="mark"
RUN export DEBIAN_FRONTEND=noninteractive; apt-get -y update \
 && apt-get install -y libcairo2-dev \
    libcurl4-openssl-dev \
    libgmp-dev \
    libpng-dev \
    libssl-dev \
    libxml2-dev \
    make \
    pandoc \
    pandoc-citeproc \
    zlib1g-dev
RUN ["install2.r", "-r 'https://cloud.r-project.org'", "readr", "googleCloudStorageR", "Rcpp", "digest", "crayon", "withr", "mime", "R6", "jsonlite", "xtable", "magrittr", "httr", "curl", "testthat", "devtools", "hms", "shiny", "httpuv", "memoise", "htmltools", "openssl", "tibble", "remotes"]
RUN ["installGithub.r", "MarkEdmondson1234/googleAuthR@7917351", "hadley/rlang@ff87439"]
WORKDIR /payload/
COPY [".", "./"]

EXPOSE 8080
ENTRYPOINT ["R", "-e", "pr <- plumber::plumb(commandArgs()[4]); pr$run(host='0.0.0.0', port=8080)"]
CMD ["schedule.R"]

5. Specify app.yaml for flexible containers as detailed here. Add any environment vars such as auth files, that will be included in same deployment folder.

Example:

runtime: custom
env: flex

env_variables:
  GCS_AUTH_FILE: auth.json

6. Specify cron.yaml for the schedule needed:

cron:
- description: "test cron"
  url: /demoR
  schedule: every 1 hours

7. You should now have these files in the deployment folder:

   • app.yaml - configuration of general app settings
   • auth.json - an authentication file specified in env arguments or app.yaml
   • cron.yaml - specification of when your scheduling is
   • Dockerfile - specification of the environment
   • schedule.R - the plumber version of your script containing your endpoints

Open the terminal in that folder, and deploy via gcloud app deploy --project your-project and the cron schedule via gcloud app deploy cron.yaml --project your-project.

It will take a while (up to 10 mins) the first time.

8. The App Engine should then be deployed on https://your-project.appspot.com/ - every GET request to https://your-project.appspot.com/demoR (or other endpoints you have specified in R script) will run the R code. The cron example above will run every hour to this endpoint.

Logs for the instance are found here.

This approach is the most flexible, and offers a fully managed platform for your scripts. Scheduled scripts are only the beginning, since deploying such actually gives you a way to run R scripts in response to any HTTP request from any language - triggers could also include if someone updates a spreadsheet, adds a file to a folder, pushes to GitHub etc. which opens up a lot of exciting possibilities. You can also scale it up to become a fully functioning R API.

Summary

Hopefully this has given you an idea on your options for R on Google Cloud regarding scheduling. If you have some other easier workflows or suggestions for improvements please put them in the comments below!