How does json logging workΒΆ

You can turn JSON, or structured, logging on by setting CORDITE_LOG_MODE=json in the docker environment variables. If you do this, beware of a few things:

  • each log line is a flat JSON object with strings mapping to strings
  • the severities are mapped on to stack driver severity levels
  • MDC properties tip up as entries in the JSON object
  • you can put as many [key=value] bits in your log message and each one will be stripped out of the message and will show up in the JSON object

You might well be asking why on earth we want to do this. A number of reasons:

  • in this way we can get stackdriver not to interpret every message as INFO level!
  • we can use the MDC for tracking requests across the entire cluster that we are monitoring - perhaps using session id, or some other discrim like txnId
  • we can mark up various log messages as having certain types - we can then use this to slice and dice the messages - in particular we could figure out how much time we spend in certain blocks, or how often we call a method, but across a whole cluster, or within a certain type of transaction.

It sounds fairly innocuous, but when I used to run a 40,000 node grid, this became a key way of understanding what our system was doing.