Takes a generated API function and lets you page through results

A helper function to help with the common task of paging through large API results.

gar_api_page(
  f,
  page_f = function(x) x$nextLink,
  page_method = c("url", "param", "path", "body"),
  page_arg = NULL,
  body_list = NULL
)

Arguments

f	a function created by gar_api_generator
page_f	A function that will extract the next page information from f(). Should return NULL if no paging is required, or the value for page_arg if it is.
page_method	Method of paging: url will fetch by changing the fetch URL; param will fetch the next page via a parameter set in page_arg; path will change a path variable set in page_arg
page_arg	If page_method="param", you need to set this to the parameter that will change for each API page.
body_list	If page_method="body", you need to set the body that will be used in each API call, including the top level parameter page_arg that will be modified by page_f

Value

A list of the API page responses, that you may need to process further into one object.

Details

The page_f function operates on the object returned from the data_parse_function of the function f

If using page_method="url" then then page_f function needs to return the URL that will fetch the next page of results. The default finds this via x$nextLink. This is the easiest to implement if available and is recommended.

If using page_method = "param", then page_f needs to extract the parameter specified in page_arg that will fetch the next page of the results, or NULL if no more pages are required. e.g. if response is x, page_f should extract the next value for the parameter of page_arg that fetches the next results. It should also return NULL if no (more) paging is necessary. See examples. Remember to add the paging argument (e.g. start-index) to the generated function too, so it can be modified.

Examples

if (FALSE) {
# demos the two methods for the same function.
# The example is for the Google Analytics management API, 
#  you need to authenticate with that to run them. 


# paging by using nextLink that is returned in API response
ga_segment_list1 <- function(){

  # this URL will be modified by using the url_override argument in the generated function
  segs <- gar_api_generator("https://www.googleapis.com/analytics/v3/management/segments",
                            "GET",
                            pars_args = list("max-results"=10),
                            data_parse_function = function(x) x)


  gar_api_page(segs,
               page_method = "url",
               page_f = function(x) x$nextLink)

}

# paging by looking for the next start-index parameter

## start by creating the function that will output the correct start-index
paging_function <- function(x){
  next_entry <- x$startIndex + x$itemsPerPage

  # we have all results e.g. 1001 > 1000
  if(next_entry > x$totalResults){
    return(NULL)
  }

  next_entry
  }

## remember to add the paging argument (start-index) to the generated function too, 
##  so it can be modified.    
ga_segment_list2 <- function(){

  segs <- gar_api_generator("https://www.googleapis.com/analytics/v3/management/segments",
                           "GET",
                            pars_args = list("start-index" = 1,
                                             "max-results"=10),
                            data_parse_function = function(x) x)

  gar_api_page(segs,
               page_method = "param",
               page_f = paging_function,
               page_arg = "start-index")

  }


identical(ga_segment_list1(), ga_segment_list2())


}