 |
Logging of causal distributed application behavior
Posted by mranga on December 21, 2003 at 06:49 PM | Comments (6)
I've been hacking at building distributed applications for some time now (both in Java and C++) and, because I was invented before debuggers were invented, I dont use debuggers. Besides my code tends to be multi-threaded and messy and open source and bugs occur on other people's machines that I cannot access. Thus, my code carries its own debugging facility, which in my bag of tricks means liberal logging. I usually resort to :
- Writing huge log files
- . In java I do new Exception().printStackTrace liberally in order to capture runtime execution.
- . Staring at multiple log filesgenerted at multiple nodes and correlating log files to source code until I become quite bad tempered and cross eyed.
So how do we improve the situation?
Theres things like log4j that nicely write things to log files although I am not fascinated by the mere ability to write to a file. However, I have often thought it would be very very cool to have the ability to do causal (vector) timestamps on messages so that when a socket writes out a message for example, if logging is enbabled, the outgoing message can be stamped with a causal clock. ( A vector clock is simply a clock with N fields for N nodes, each node increments its counter when it sends/receives a message. A vector clock allows you to determine relative ordering of events in a distributed application. Its actually a very old idea employed in distribtued systems such as ISIS.)
Then the log files from different nodes can be gathered and one can re-play execution in causal sequence. What with the nice introspection features of java, one might even be able to record and replay a distributed system. Now, protocols like SIP (with which I've been hacking of late), do capture causality in message sequences through the notions of transactions, sequence nunbers etc and we bult some nice log file viewing tools around it, but is there a more general way to do this using vector timestamps? So where is this leading ?
- 1. Lets add a standard log format to capture causality in distributed applications.
- 2. Lets build tools for correlation of distributed logs without needing to synchronize wall clock time.
and finally, I would love to not employ the infamous new Exception().printStackTrace to capture stack traces in log files (which invariably frightens unsuspecting users). It would be nice to have a better means of doing this (perhaps there is and I dont know about it).
Bookmark blog post:  del.icio.us  Digg  DZone  Furl  Reddit
Comments
Comments are listed in date ascending order (oldest first) | Post Comment
-
Re: printStackTrace
StackTraceElement?
Posted by: zoe_info on December 22, 2003 at 12:44 PM
-
Useful distributed logging technique
In the cajo project there is a class called MonitorItem. It is a very illustrative approach to logging the interaction between distributed objects
I've been able to develop wonderfully elaborate distributed systems, only by using it.
Worth a look,
John
Posted by: cajo on December 22, 2003 at 02:08 PM
-
Useful distributed logging technique
Very cool stuff. I have to deal with lower level IO abstractions (such as sockets though). A source code viewer that can integrate vector causal timestamps (ie. vector lamport clocks) with the log would be nice indeed. Just something as simple as looking at a bunch of log files and bringing up source code (with highlighted line numbers) corresponding to the causal order in which the messages were exchanged would be a nice post mortem debugging tool.
Posted by: mranga on December 23, 2003 at 07:34 AM
-
Useful distributed logging technique
Ah,
Really low-level stuff. I used to do elaborate socket based protocols, but as the applications grew more complex, as they inevitably do, they became extremely difficult to debug.
As soon as I could get everyone to commit to using Java; RMI clearly became the best technique. However, I did have to endure a similar 'culture-shock' to the one I had getting my teams to use compiled languages more than a decade ago.
Whether it was code or bandwidth:
Yes it's less efficient, yes it uses more memory, but the symbolic representations are far more intuitive.
I believe RMI is a compiler for network interaction.
John
Posted by: cajo on December 23, 2003 at 09:19 AM
-
Useful distributed logging technique
Your point is well taken but it does not help if one is imlementing an actual distributed protocol such as SIP or JXTA where a message may take multiple hops and trigger multiple events before a response to that message actually returns.
Incidentally, when I first saw C, I thought, Ah! thats a nice assembly macro language and indeed that was the history of C before it became C.
Posted by: mranga on January 02, 2004 at 10:32 AM
-
网络营销软件
网络营销软件
网络营销软件
群发软件
群发软件
---
群发软件
网络营销软件
论坛群发软件
网站排名软件
群发软件
推广小助手破解版
论坛群发软件
网站排名软件
群发软件
推荐给你很好的群发软件和信息群发软件和供求群发软件
推荐给你很好的群发软件和信息群发软件和供求群发软件博客群发软件网络营销软件网络营销软件
网站排名软件网站排名软件网站优化软件信息群发软件信息群发软件信息群发软件论坛群发软件网站推广软件网站推广软件博客群发软件博客群发软件
群发软件群发软件博客群发软件论坛群发软件网络营销软件论坛群发软件
信息群发软件推广软件网站推广软件网络营销软件网站推广软件群发软件网站排名软件网站推广软件博客群发软件论坛群发软件群发软件网站排名软件网站推广软件博客群发软件论坛群发软件
网站排名软件
博客群发软件
网站排名软件
网站推广软件
群发软件信息群发软件
免费论坛群发软件
论坛群发软件
网站排名软件
免费博客群发软件
网站推广软件
群发软件
博客群发软件
网站排名软件
网站推广软件
群发软件信息群发软件
免费论坛群发软件
论坛群发软件
网站排名软件
免费博客群发软件
博客群发软件
信息群发软件
论坛群发软件
信息群发软件
博客群发软件
qq群发软件
邮件群发软件
博客群建软件
企业名录搜索软件
信息群发软件
邮件群发软件
论坛群发软件
博客群发软件
网站推广软件
网络营销软件
全能营销破解版
网络营销软件
论坛群发软件
论坛群发软件
论坛群发软件
网络营销软件
信息群发软件
信息群发软件
信息群发软件
群发软件
论坛群发软件
群发软件
网络营销软件
网站推广软件
Posted by: u89io98 on December 25, 2007 at 11:52 PM
|
java.net RSS Feeds
