 |
Sorcerer: a better source-code cross referencing tool
Posted by kohsuke on September 19, 2006 at 02:33 PM | Comments (27)
My latest hobby project went online today. This project, named "sorcerer" generates HTML files from Java source code, and it does so in a way better than existing tools, thanks to the excellent javac tree API in JDK6. Just take a look at the sample report and see it for yourself.
There have been a few source code cross reference generators, like JXR in Maven and OpenGrok, but the problem I found with all of those tools is that they don't understand Java semantics. It can do lexical analysis, but none of them do the semantic analysis of the Java code. So the end result is that the generated HTML doesn't have much intelligence. The best it can do is to do syntax-coloring or do some keyword match (the tool basically just guess that this token MySAXParserFilterImpl probably means org/acme/MySAXParserFilterImpl.java.) I'm used to modern Java IDEs, so my expectation was higher.
This is where sorcerer does a better job. It actually builds the AST of Java source files, just like javac does (by using javac tree API.) So when it sees something like p.println(5) in your source file, it knows that "p" is a local variable defined in line 95 as PrintWriter, the println method invocation is PrintWriter.println(int), and so on.
Armed with this rich information, sorcerer can generate all kinds of useful navigation features that you often use with IDEs. In the version 0.1 I posted today, it does:
- jump to sub-classes, super-classes, etc.
- jump to methods that are overriding the current method
- jump to method overridden by the current method
- jump to definition of variables, methods, types, packages, etc.
I plan to add more navigations in the future — the top on the list is the "find usage" features. Another thing that I'd like to work on is to integrate source code from multiple projects into a single coherent view, so that you can navigate between projects seamlessly.
Meanwhile, the project should be already useful. It comes with CLI, ant task, and maven plugin. In particular, if you are using Maven2, it's dead-easy to set up. Read up the documentation on the website.
Bookmark blog post:  del.icio.us  Digg  DZone  Furl  Reddit
Comments
Comments are listed in date ascending order (oldest first) | Post Comment
-
One thing that is missing in jumping to variable / field / method declaration is highlighting it. It's not immediately clear that it's on the first line (usually in IDE you get it in the top portion, but not on the first line). Great tool.
Posted by: kirillcool on September 19, 2006 at 03:17 PM
-
Good point. In fact I tried a little to figure out if there's anyway to show the declaration in the middle of the page (obviously, IDEs do that for a reason), but I couldn't figure out how.
I'll see if I can highlight the line.
Posted by: kohsuke on September 19, 2006 at 03:23 PM
-
Perhaps you could add a target for every line (instead of every field / variable / ... ) and "jump" to a location that is 5-10 lines before the actual one.
Posted by: kirillcool on September 19, 2006 at 03:58 PM
-
Wow, this looks really useful. I spen quite some time setting up OpenGrok but also felt a bit disappointed with its intelligence. Well, now's the time to try sorcerer! Thanks for all the great tools like this one and like hudson.
Posted by: vsizikov on September 19, 2006 at 04:13 PM
-
Kirill — Yeah, I guess that's a possibility, too. You don't want to use line numbers because it won't make much sense when you paste them to e-mails or IRCs, but I can always have 5-10 lines above the actual one.
Posted by: kohsuke on September 19, 2006 at 04:32 PM
-
vsizikov — thanks! I'm always looking for help on any of my projects. If you feel like hacking code with me, let me know!
Posted by: kohsuke on September 19, 2006 at 04:32 PM
-
Hi--I tried running this on the Flying Saucer source, and got an exception--I posted to the user's mailing list. Look forward to trying it out, nice job! Cheers Patrick
Posted by: pdoubleya on September 20, 2006 at 04:58 AM
-
psoubleya — thanks. talk to you on the users list.
Posted by: kohsuke on September 20, 2006 at 07:49 AM
-
Great work, seems already very useful for a 0.1 version!
Posted by: xhanin on September 20, 2006 at 08:31 AM
-
The sample report looks very cool! One change I'd like to see is some differentiation between links that are going to take me somewhere else in the source, and links that are to outside javadoc. I'm pretty sure you could do that with a css hover style that changes the icon, or pops up a tooltip or something. All in all, great work!
Posted by: tfenne on September 20, 2006 at 09:36 AM
-
May I make a plea?
All of these source code mark up/cross reference tools are all lovely.
But they fail at a very basic task, something I've been look for for quite some time.
They fail to show me dead methods. Methods in my code base that are simply not called any more.
They don't show me where the code is called from. I can go backwards, find the source of a method or class, but not the other way around.
I appreciate it will never have perfect coverage, esepecially with things like reflection and what not, but that doesn't mean what information can be gathered isn't usefui.
Those kinds of metrics are much more interesting to me.
Posted by: whartung on September 20, 2006 at 10:26 AM
-
This looks great. Thanks
Posted by: rameshm on September 20, 2006 at 10:47 AM
-
Very nice. Nice sub/super navigation, too.
For scrolling to the middle of the doc, it should also be possible to use DOM coordinates and scrolling functions. For semantics, you could leave the link pointing to the right line. Just override the default like behavior in JavaScript.
Also, I noticed that a second field on a line doesn't get highlighted in blue. Great job using CSS, by the way.
Posted by: tompalmer on September 20, 2006 at 11:28 AM
-
Oh, and I agree with whartung that if you could fit reference hierarchies in there, that would be super cool. Maybe could be done in JavaScript since storing them all as pregenerated HTML would be ugly. (Or does could there be both static file and dynamic server versions of this? I think I'd put static file as the priority which it seems you've done here.) Also note that I say "reference" hierarchies unlike Eclipse's limited "call" hierarchies. No need for separate concepts here, and no need for a separate search feature without hierarchy support. (Also please sort the hierarchy results unlike Eclipse.)
Posted by: tompalmer on September 20, 2006 at 11:32 AM
-
Oh, and my link was on the wrong sentence. Should have been on the sentence before-hand. Didn't preview first.
Posted by: tompalmer on September 20, 2006 at 11:34 AM
-
It looks wonderfull! Ever thought about creating a diff tool that would make use of AST? It would make a great extension to the project. I'm currently looking for one to track changes when refactoring is done on code.
Posted by: simontrudeau on September 20, 2006 at 12:53 PM
-
looks nice. can't wait for a Maven plugin for it :)
Posted by: jimisola on September 20, 2006 at 02:35 PM
-
tfenne — I recorded issue #2 for that. If you can come up with hand-edited mock up HTML that does what you want to do, it would be very easily to modify sorcerer to generate that.
whartung — with "find usage" information, it will be really easy to spot methods that are not called, again just like modern IDEs do. So stay tund for future enhancements.
tompalmer — thanks for spotting a bug. I filed issue #3 for that. If you know that JavaScript to scroll into the right place, please tell me more about it. I'd love to learn, but I couldn't find anything like that in JavaScript so far. I'm also not that familiar with Eclipse, and some of the concepts you are describing (like reference/call hierarchy) is new to me. If you have some pointers to those handy, those would be much appreciated (time to install the latest Eclipse, I know...)
simontrudeau — yes, I briefly thought about that, but it didn't "click" to me. What kind of things do you think you'd do if you have AST-based diff tools?
Posted by: kohsuke on September 20, 2006 at 02:41 PM
-
jimisola — a maven2 plugin is already there. Go to the project website, read the doc, try it out, and let me know how it goes.
Posted by: kohsuke on September 20, 2006 at 02:42 PM
-
kohsuke: I noticed that when I went to the project site. nice work! might have to install a JDK6 :)
Posted by: jimisola on September 20, 2006 at 03:00 PM
-
Regarding "find usage", I'm kind of looking for a "bulk" thing I can run across the source base, not necessarily a browser. The browser is important, but I've got a 5+ year old 1M+ line source base that's been touched by over 40 different people.
Large, sweeping, bulky reports are my friend.
thanx!
Posted by: whartung on September 20, 2006 at 03:14 PM
-
Wow! we should generate GlassFish docs using this! Great work, as usual!
Posted by: jfarcand on September 20, 2006 at 07:26 PM
-
Jean-Francois — Yes, that's on my TODO list. I'm thinking about offline version of this tool and a web application version of this tool, and especially now that I have access to glassfishwiki.org, I can easily run the web app version there.
Posted by: kohsuke on September 20, 2006 at 07:43 PM
-
Lovely looking app - very impressive.
However, why would I use this rather than Intellij? It's got everything that whartung posted about as well.
Posted by: archangel on September 21, 2006 at 12:46 AM
-
archangel — Here's my use case. I work on the JAXB/WS RI, but occasionally I need to see the Glassfish source code. To do that, I'd have to check out the workspace (takes 1/2hr at least), and it doesn't even have IntelliJ project files (even if it has, it will need a huge chunk of memory just to load them up.)
If we have sorcerer generate Glassfish source code and available online, I can just point my browser to it. I'll be browsing the source code in seconds, not in 1/2 hr. That's a big difference to me.
Or consider another situation. I'm in a conference room with a bunch of other engineers discussing a design problem. We quickly need to see how a particular portion of Glassfish looks like. I don't think you'd want to wait for minutes to load the IDE.
Just replace "Glassfish" with your favorite project. This tool is not for developers to browser their projects. It's a courtesy for users of your project so that they can see your code quickly.
Posted by: kohsuke on September 21, 2006 at 10:11 AM
-
Great work! We were using Java2Html till now and this is a Quantum Leap.
Posted by: g_korland on September 22, 2006 at 03:21 AM
-
Does anyone know what is the software behind this page
http://www.java2s.com/Open-Source/Java-Document/JDK-Core/lang/java/lang/Byte.java.htm
Posted by: asdf263 on September 26, 2007 at 08:51 AM
|
java.net RSS Feeds
