The big idea

At work, we use all these amazing tools like Jest, Flow, npm etc. which greatly enhance working in a large team. The visibility of these tools, however, is not that great. Sure, they will run on git hooks or in your editor but seeing actual metrics for the whole project or a trend of how the project evolved over time is difficult or time consuming.

I wanted to come up with something different and since I love Grafana dashboards, decided it was time I put a couple of days work into it and try to get a usable dashboard on the big screen.

We use Jenkins as a CI and I had an InfluxDB container running to collect metrics. Jenkins is not really my favorite tool, so instead of making something tailor-made to Jenkins (like a plugin), I looked for something pluggable which I could easily transition to another CI or any pipeline for that matter. Luckily, the InfluxDB HTTP REST client is easy enough to use for this idea.

Groovy (Jenkins' scripting lang) was somewhat difficult to understand, but fortunately we could spawn new shell scripts from it, which made everything easier to do in Bash. The latter (or sh if you like that) is compatible with almost every headless environment.

Defining our collection data

The project I wanted to collect metrics for is a React Native application, but these collection 'samples' are common for most Javascript projects so you could adapt it to your React web or Node project as well:

Code quality

Test coverage: collect via Jest's provided test coverage
Flow coverage: collect via flow-coverage-report
Eslint warnings: collect via eslint dumping to a file
FIXME, TODO count: use git to perform a project-wide search for this string

Dependencies

Outdated dev deps: via npm outdated
Outdated deps: via npm outdated

Build

Build failures: collect via Jenkins error handling

Writing a script + function to insert metrics

With the help of bash + curl I managed to write a simple script which takes the metrics and maps them to the HTTP payload expected by the InfluxDB HTTP REST interface. Writing this in Bash meant I could easily test it locally by running the script, instead of triggering new Jenkins builds each time:

bash#!/usr/bin/env bash
set -e

function display_help() {
  echo ""
  echo "Usage:"
  echo "  Pass fields as arguments in the format of field=value to this script"
  echo "    e.g: ./write_influxdb.sh my_measurement myfield=somevalue otherfield=3"
  echo ""
}

# Convenience wrapper around curl to insert tags + fields in InfluxDB.
# This script gracefully exits with 0 even if writing to database is unsuccessful.
function write_to_influxdb() {
  local database="orbit_metrics" # This is your InfluxDB database
  local jobBranch author tags fields

  if [ -z "$1" ]; then
    echo ""
    echo "No measurement provided to write_to_influxdb.sh!"
    display_help
    exit 1
  fi

  # Non PR builds do not have the CHANGE_BRANCH env
  if [ -z "$CHANGE_BRANCH" ]; then
    jobBranch="$BRANCH_NAME"
  else
    jobBranch="$CHANGE_BRANCH"
  fi

  # Branch builds do not have CHANGE_AUTHOR env
  if [ -z "$CHANGE_AUTHOR" ]; then
    author="jenkins"
  else
    author="$CHANGE_AUTHOR"
  fi

  tags="project=my_project,branch=$jobBranch,author=$author,job=$BUILD_ID"

  for field in "${@:2}"; do
    key=$(echo "$field" | sed 's/=.*//g')
    value=$(echo "$field" | awk -F= '{print $2}')
    echo "Field: { $key: $value }"
    # Basically: Join key + value into array, trim leading whitespace, join by comma
    fields=$(echo "$fields $key=$value" | sed 's/^ //g' | sed 's/ /,/g')
  done

  echo "Writing measurement to $database.$1"
  echo "  Tags: $tags"
  echo "  Fields: $fields"
  echo ""
  echo "HTTP Payload: $1,$tags $fields"

  curl -i -X POST \
    "$INF_HOST/write?db=$database&u=$INF_USER&p=$INF_PASS" \
    --data-binary "$1,$tags $fields" || echo "Could not insert measurement into database..."
}

write_to_influxdb "$@"

In this process I added some logging via echo calls so we could see where something went wrong. The InfluxDB container was also not running at high availability but in a Google Cloud playground so it would fail writing from time to time. To avoid having the whole build fail because the curl call exits with non-zero, I added a simple echo failsafe to keep the Jenkins pipeling running.

Jenkins

From Jenkins' side we then needed some way to interact with the new script. The easiest way was to write a Groovy function that would call the script in our pipeline and pass it the metrics:

groovydef post_influxdb(measurement, fields) {
  withCredentials([
    string(credentialsId: 'ORBIT_INFLUX_HOST', variable: 'INF_HOST'),
    string(credentialsId: 'ORBIT_INFLUX_USER', variable: 'INF_USER'),
    string(credentialsId: 'ORBIT_INFLUX_PASS', variable: 'INF_PASS'),
  ]) {
    sh (label: "post_influxdb", script: "bash scripts/write_influxdb.sh ${measurement} ${fields}")
  }
}

Using the withCredentials wrapper will make credentials defined on Jenkins available as regular envs in our script!

And then finally, call it in our pipeline to have it insert the development metrics into InfluxDB:

groovytry {

  stage('Checkout') {
    checkout scm
  }
  
  stage('Dependencies') {
    sh "npm -v"
    sh "npm ci"
    
    outdatedDep = sh (returnStdout: true, script: "npm outdated --long | grep \"dependencies\" | grep -v \"git\" | wc -l").trim()
    outdatedDev = sh (returnStdout: true, script: "npm outdated --long | grep \"devDependencies\" | grep -v \"git\" | wc -l").trim()
    post_influxdb('dependencies', "npm_dep_outdated=${outdatedDep} npm_devdep_outdated=${outdatedDev}")
  }
  
  stage('Lint') {
    sh "npm run lint > lint-results.log"
    sh "cat lint-results.log"
    
    warnings = sh (returnStdout: true, script: "grep -E \"([0-9]+ warnings)\" lint-results.log | awk '{ print \$2 }'").trim()
    fixme_count = sh (returnStdout: true, script: "git grep -EI \"FIXME\" src | wc -l").trim()
    todo_count = sh (returnStdout: true, script: "git grep -EI \"TODO\" src | wc -l").trim()
    post_influxdb('code_quality', "fixme_count=${fixme_count} todo_count=${todo_count} linter_warnings=${warnings}")
  }
  
  stage('Tests') {
    sh "mkdir -p coverage"
    sh "npm run test:coverage > coverage/test-coverage.log"
    sh "cat coverage/test-coverage.log"
    
    def testPercentage = sh (returnStdout: true, script: "grep 'All files' coverage/test-coverage.log | cut -c 122-126").trim()
    post_influxdb('code_quality', "test_coverage=${testPercentage}")
  }

  post_influxdb('jenkins', 'build_succeeded=1')
  
} catch(error) {
  
  echo "ERROR: $error"
  
  if (error != 'hudson.AbortException: script returned exit code 143') {
    post_influxdb('jenkins', 'build_failed=1')
  }
  
  throw error
  
}

def post_influxdb(measurement, fields) {
  withCredentials([
    string(credentialsId: 'ORBIT_INFLUX_HOST', variable: 'INF_HOST'),
    string(credentialsId: 'ORBIT_INFLUX_USER', variable: 'INF_USER'),
    string(credentialsId: 'ORBIT_INFLUX_PASS', variable: 'INF_PASS'),
  ]) {
    sh (label: "post_influxdb", script: "bash scripts/write_influxdb.sh ${measurement} ${fields}")
  }
}

Et voila! The data will now be sent to InfluxDB on the next triggered job. Using the Grafana explorer or InfluxQL you can now easily create visible dashboards and alerts that will show the team how the project's quality is evolving over time or if something needs attention.

We have this running for about 3 months now and it does a great job. I am now exploring gathering metrics via Telegraf to collect even more useful metrics like Github data, Pull Request information from Bitbucket and contribution stats.

Any thoughts? Let me know via Twitter or email 👋

The code examples for this can also be found on my Github demo repo

Thibault Maekelbergh

📈 Gathering project metrics using Jenkins and InfluxDB