Skip to main content

Counter map

Posted by evanx on June 7, 2012 at 7:39 AM PDT

We herewith begin a saga about monitoring with this series entitled "Timestamped: a trilogy in a few parts," this being the first part, where we introduce a map to count key things, and ensure we can order it by its integer values.

 Counter map: A part of "Timestamped: a trilogy in a few parts."

We will be analysing logs in this unwinding series. Ultimately we gonna hook up a remote Log4j appender to digest our logs in order to gather stats, and make some judgement calls as to the rapidly changing status of our app.

As you can imagine we will need to count so many key things like the number of successes and failures of one thing or another. For this purpose, we introduce the following so-called "counter map."

public class IntegerCounterMap extends TreeMap {
    private int totalCount = 0;

    public int getInt(K key, int defaultValue) {
        if (!containsKey(key)) {
            return defaultValue;
        } else {
            return get(key);
        }
    }

    public int getInt(K key) {
        return getInt(key, 0);
    }
   
    public void add(K key, int augend) {
        totalCount += augend;
        put(key, new Integer(getInt(key) + augend));
    }
   
    public void increment(K key) {
        add(key, 1);
    }

    public int getTotalCount() {
        return totalCount;
    }

    public int calculateTotalCount() {
        int total = 0;
        for (K key : keySet()) {
            total += getInt(key);
        }
        return total;
    }
   
    public LinkedList descendingValueKeys() {
        return Maps.descendingValueKeys(this);
    }
}

where since we are often counting things one by one e.g. as we digest log messages, we provide that increment() convenience method.

Our typical key type is String e.g. the name of something we are counting e.g. "ERROR".

    @Test
    public void testCounterMap() {
        IntegerCounterMap counterMap = new IntegerCounterMap();
        counterMap.increment("INFO");
        counterMap.increment("ERROR");
        counterMap.increment("ERROR");
        Assert.assertEquals(100*counterMap.getInt("ERROR")/counterMap.getTotalCount(), 66);
        Assert.assertEquals(counterMap.getInt("WARN"), 0);
        Assert.assertEquals(counterMap.getInt("INFO"), 1);
        Assert.assertEquals(counterMap.getInt("ERROR"), 2);
        Assert.assertEquals(counterMap.size(), 2);
        Assert.assertEquals(counterMap.getTotalCount(), 3);
        Assert.assertEquals(counterMap.getTotalCount(), counterMap.calculateTotalCount());
   }

where getTotalCount() is used to get a percentage of the total e.g. a 66% error rate, D'oh!

Note that since WARN is not incremented like the others, it's not put into the map, and so the size of map is only the two keys, namely INFO and ERROR.

We have a descendingValueKeys() method for when we wanna display counters in descending order, to see the biggest numbers and/or worst culprits. This delegates to the following util class.

public class Maps {  

    public static LinkedList descendingValueKeys(Map map) {
        return keyLinkedList(descendingValueEntrySet(map));
    }
   
    public static NavigableSet>
            descendingValueEntrySet(Map map) {
        TreeSet set = new TreeSet(new Comparator>() {

            @Override
            public int compare(Entry o1, Entry o2) {
                return o2.getValue().compareTo(o1.getValue());
            }
        });
        set.addAll(map.entrySet());
        return set;
    }

    public static LinkedList keyLinkedList(NavigableSet> entrySet) {
        LinkedList keyList = new LinkedList();
        for (Map.Entry entry : entrySet) {
            keyList.add(entry.getKey());
        }
        return keyList;
    }

    public static V getMinimumValue(Map map) {
        return getMinimumValueEntry(map).getValue();
    }
    ...
}

where courtesy of a TreeMap, we sort the map's entries by value, and put the thus ordered keys into a LinkedList to iterate over e.g. in a for each loop.

See also stackoverflow.com which helped me with that.

Let's test this ordering.

    @Test
    public void testDescendingMap() {
        IntegerCounterMap counterMap = new IntegerCounterMap();
        counterMap.add("BWARN", 0);
        counterMap.add("DEBUG", 1000000);
        counterMap.add("ERROR", 5000);
        counterMap.add("INFO", 1);
        Assert.assertEquals(counterMap.size(), 4);
        Assert.assertEquals(counterMap.descendingValueKeys().getFirst(), "DEBUG");
        Assert.assertEquals(counterMap.descendingValueKeys().getLast(), "BWARN");
        Assert.assertEquals(Maps.getMinimumValue(counterMap).intValue(), 0);
        Assert.assertEquals(Maps.getMaximumValue(counterMap).intValue(), 1000000);
        for (String key : Maps.descendingValueKeys(counterMap)) {
            System.out.printf("%d %s\n", counterMap.get(key), key);
        }
     }

where we use "BWARN" instead of "WARN" to be sure we aren't picking up the natural ordering. (Yes, that was hiding a bug that bit me in the bum!)

After some DEBUG'ing (with the help of a lot logging), we eventually get what we would have expected...

1000000 DEBUG
1001 ERROR
1 INFO
0 BWARN

As so often seems to be the case, we have a million DEBUG messages, a thousand and one ERRORs, with very little INFO to go on, and no bloody warning! "Put that in your internet, put that in your twitter right there."

Resources

https://code.google.com/p/vellum/ - where i will collate these articles and their code.

Coming soon

In the next installment, we'll properly introduce the namesake of this series, a so-called Timestamped interface, for our "time series" or what-have-you.

public interface Timestamped {
    public long getTimestampMillis();   
}
Related Topics >>