Creating your own cookieless analytics tool with GTM Server Side, Cloud Run, BigQuery and Shiny

Making a browser side tag to send in the events.

This will send in GTM events as per traditional GTM setups.

I choose not to ride on the back of the GA events like the Baz Analytics example, so will need to send to my own endpoint. The tag will instead send it to its own GTM server side endpoint /edmondlytica, reading the event information from the GTM trigger the tag is set up for.

It includes a timestamp and the eventId as they could be useful.

const log = require('logToConsole');
const copyFromDataLayer = require('copyFromDataLayer');
const encodeUri = require('encodeUri');
const sendPixel = require('sendPixel');
const getReferrerUrl = require('getReferrerUrl');

log('data =', data);

const gtm = copyFromDataLayer('gtm');

const the_url = data.transport_url + '/edmonlytica' +
    '?event=' + copyFromDataLayer('event') + 
    '&referrer=' + getReferrerUrl() + 
    '&ts=' + gtm.start + 
    '&eventId=' + data.gtmEventId +
    '&custom=' + data.custom +
    '&page=' + data.pagepath;

log('the_url:', encodeUri(the_url));

sendPixel(encodeUri(the_url), data.gtmOnSuccess(), data.gtmOnFailure());

This code is set via a tag template, with a data.transport_url field to swap out which GTM server-side instance you want to use. My default is set to https://gtm2.markedmondson.me.

I also have a field for variables that can be set with GTM macros such as {{Click URL}} and instead of hardcoding the page that will be sent in the tag use by default the macro {{Page URL}}. This gives some flexibility on precisely what information is sent, such as stripping out parameters or not, via GTM trigger configuration.

Edmonlytica GTM Browser tag template data fields

Making the tag instances + using the custom field

Edmonlytica only has one custom field, which is used to send in unique data for the trigger you are using.

By default with no custom field you are only collecting page views.

I also added two other instances - one linked to my External Link Clicks trigger (all those linkCicks that do not go to my own website), and the Scroll Depth trigger. See the other Built-in GTM variables for other ideas, or use JavaScript to make your own.

I now create instances of my tag and choose what triggers it - for me pageviews, external link triggers and scroll events for 50% and 90% of the page length, but you could add any other GTM trigger you have created. The tag will capture the event details for that trigger.

Edmonlytica GTM Browser-side tag instances

Publishing the tag or in preview mode I can see the logs showing the URL is being formed correctly, but getting a 400 error as we haven’t setup the server-side yet. In my case the triggers for a pageview = gtm.js and for an external link click = gtm.linkClick

Logs in Web Console Network tab - (click to make larger)

Now I need the GTM Server Side Client that will accept the /edmonlytica endpoint, process it and send it into BigQuery.

Self Hosting GTM itself

As mentioned above, make sure to create a “Google Tag Manager: Web Container” as specified in the GTM Server Side documentation.

The GTM Server Side Client

The GTM Server Side Client accepts the endpoint request from the browser-side tag, puts it into an event format then runs it against the server-side tag. If successful, it returns a 200 response back to the browser as a pixel.

It includes a hashing of an anonymised ip address for a session/userId. This isn’t as reliable as using cookies for identifying sessions and users, but deemed fine for the scope of this use case. See these 4000 words on data privacy engineering and GTM server-side for background.

const claimRequest = require('claimRequest');
const getRequestPath = require('getRequestPath');
const getRequestQueryParameters = require('getRequestQueryParameters');
const log = require('logToConsole');
const runContainer = require('runContainer');
const returnResponse = require('returnResponse');
const setPixelResponse = require('setPixelResponse');
const sha256Sync = require('sha256Sync');
const getRemoteAddress = require('getRemoteAddress');

if(getRequestPath() == "/edmonlytica"){
  
  claimRequest();
  
  const ips = getRemoteAddress().split(".");
  const anon_ip = ips[1]+"."+ips[2]+"."+ips[3]+".000";
  
  const queryParameters = getRequestQueryParameters();
  const hashedip = sha256Sync(anon_ip + 'salted');
  
  // create event schema 
  const event = {
    event_name: "edmonlytica",
    event:  queryParameters.event,
    referrer:  queryParameters.referrer,
    ts:  queryParameters.ts,
    eventId: queryParameters.eventId,
    custom: queryParameters.custom,
    sessionId: hashedip,
    page: queryParameters.page
  };
  
  log("Edmonlytica request event:", event);
  
  setPixelResponse();
  
  runContainer(event, () => returnResponse());

}

Remember to set the appropriate permissions as documented in the server side tagging APIs.

The GTM Server Side Tag

The Server Side tag will do the actual sending to BigQuery of your data.

const log = require('logToConsole');
const getAllEventData = require('getAllEventData');
const BigQuery = require('BigQuery');
const JSON = require('JSON');

// need a list of event data
const event = getAllEventData();

// this should match the BigQuery schema
let row = [];
row.push(event);

log('array to push to BigQuery: ', row);

BigQuery.insert({
  projectId: data.projectId,
  datasetId: data.dataset,
  tableId: data.table,
}, row, {}, () => {
  log('BigQuery Success');
  data.gtmOnSuccess();
}, (errors) => {
  log('BigQuery Failure');
  log(JSON.stringify(errors));
  data.gtmOnFailure();
});

Its possible to push many rows at once, but for this application it only does one at a time.

Once you have created the template, deploy the instances of them to the Clients and Tags and create a trigger:

Handy SQL

This is where off-the-shelf analytics tools earn their keep by making these aggregations for you, but for simple data models like this it doesn’t take too long to extract some useful data tables. The common ones could be made into their own tables or views for convenience.

• Number of pageviews per YYYY-mm-dd hour

SELECT FORMAT_TIMESTAMP("%Y%m%d-%H00",
         TIMESTAMP_MILLIS(CAST(ts AS INT64))) AS timestamp, 
         count(1) as pageviews
FROM `edmonlytica.code_markedmondson_me` 
WHERE event = "gtm.js" 
group by timestamp

• Number of external link clicks per YYYY-mm-dd

SELECT custom as URL,
       FORMAT_TIMESTAMP("%Y%m%d",TIMESTAMP_MILLIS(CAST(ts AS INT64))) AS timestamp, 
       count(1) AS linkClicks 
FROM `edmonlytica.code_markedmondson_me` 
WHERE event = "gtm.linkClick" 
group by URL, timestamp

• Number of unique hashed ips visiting from referrers per landing page

SELECT
  referrer, page,
  FORMAT_TIMESTAMP("%Y%m%d %H00",TIMESTAMP_MILLIS(CAST(ts AS INT64))) AS timestamp,
  COUNT(DISTINCT sessionId) AS visits
FROM
  `edmonlytica.code_markedmondson_me`
WHERE
  event = "gtm.js" 
  AND page is not null 
  AND referrer IS NOT NULL AND referrer != ""
  AND NOT STARTS_WITH(referrer, "https://code.markedmondson.me")
GROUP BY
  referrer, page, timestamp

You could connect to BigQuery with Data Studio or lots of other visualisation products, but I would like to connect via R Shiny.

Shiny deployment on Cloud Run

The Shiny app code is here: https://github.com/MarkEdmondson1234/edmonlytica

I used the SQL examples above to help build the datasets for Shiny to process and display.

bigQueryR was used within my Shiny app, then since Cloud Run is running on GCP in the same project as the BigQuery dataset I could use googleAuthR::gar_gce_auth() to use the default authentication service account without needing to upload any auth keys.

I used highcharter as was used in my previous Shiny app (real-time forecasting dashboard using GTM, GCP and Shiny) as it makes pretty plots, and also kept the forecasting capabilities from that post for my trended hits via forecast.

The deployment followed the deploy Shiny on Cloud Run instructions and once the Dockerfile was built could be deployed by googleCloudRunner via:

library(googleCloudRuner)

# deploy to Cloud Run
cr_run(sprintf("gcr.io/%s/shiny-edmonlytica:latest",cr_project_get()),
       name = "shiny-edmonlytica",
       concurrency = 80,
       max_instances = 1,
       memory = "2Gi")