Mike Loukides's Blog

The Problem with Scripting

A few days ago, I needed to write a program that was essentially a variant of the well-known UNIX diff. I had to find out which lines had been added and deted between two versions of a file--the trick (which made diff inapplicable) was that the order of the lines had been changed.

Well, I'd been wanting an excuse to learn Ruby for some time, but the more I thought about it--I could write this sucker in Java in a few minutes, and be done with it. I needed it it RIGHT NOW. I didn't want to have get stuck in the semantics of a new language.

Bottom line--I still have to learn Ruby. I realized I could (and did) write the program in less time than it would have taken me to find one of my Ruby books. (Sorry, Dave and Andy--it's still on the shelf...) It weighs in at around 25 lines, with somewhat more exception handling than I usually bother to do.

So I really have to ask: what's the "scripting language mystique" all about? You don't have to run javac? Rubbish--I don't believe that's a big issue. You don't have to write declarations? Bull--writing declarations means (to me) that it's easier to debug misspelled variable names. That's a HUGE time saver. As Brett McLaughlin has often said, "whenever you can get the compiler to do the debugging, it's a win." Is it that scripting languages are better "glue" languages for calling UNIX utilities and other external programs? Well, I don't see why it's so difficult to call Runtime.exec(). The fact is--if you have some documentation, you can use Java to call into the guts of other Java programs, not just their external command-line interfaces. That's some pretty powerful glue. If you understand reflection or have been watching what's going on with lightweight containers like Spring, you've got real computation superglue.

Or is it that this could have been written in fewer lines? I'm sure this program could have been compressed into a single line of Perl, or maybe a dozen lines of Python. Sorry, I just don't see "number of lines" as a good metric of anything. When I was in high school, I loved APL because just about anything could be compressed into a single, inscrutable line. I didn't believe in comments then, either. When I was a child, I thought as a child... I admit to being biased--as an O'Reilly editor, I read much more code than I write. So being readable (and I think Java is very readable) is very valueable to me.

The really sad thing is: there's another programming task I need to do. I've got to change a fairly simple line-oriented file format into a sort of bastardized xml-like format. It's trivial in Java. I've really set this task aside as 'the program I'll write in Ruby'. But, really--in Java, it's 10 minutes of work. And I'm tired of setting it aside.

Ultimately, the only criterion is getting the job done.

Comments

• I agree with you, I'm not a big scripting guy either. However, it can be useful for allowing custimization of applications. It can be argued that Java classes can be used in that case too, but it's nice to not require a customer have access to a java compiler to customize your applications.

  Posted by: rabbe on January 03, 2005 at 09:04 AM

• Hello. with the recent winds of dynamic languages, i wanted to give a shot to ruby too. but, then, i find myself bored reading the documents. And the idea of developing it in a text editor other than an advanced IDE made me less enthusiastic. java is not perfect, but the advanced editors really help a lot where language lacks. Sorry but it will take time for me to even try one of those languages.. there are web sites comparing language quality with line counts. i just smile.. give me a 10 line of java code any day doing the same thing that a cryptic one line perl code does.. reuasbility is the real problem both approaches should solve.

  Posted by: ahmetaa on January 03, 2005 at 09:11 AM

• Knowing your shell and available commands and using a little forethought makes this task trivial: sort file1 > file1.tmp; sort file2 > file2.tmp; diff file1.tmp file2.tmp

  One of the key reasons I use scripting languages are the availability of libraries (or in this case simple commands). If you know the "language" well enough its always easier to write in it. When I'm writing Perl, everything looks like a regular expression. When I'm writing Java code, everything looks like a collection. When I'm writing SQL, everything looks like a join.

  If you have the extra tools ready to go, then life becomes easier. But don't learn things just because they might be useful someday... Learn them because you need them. I learned AWK because I couldn't do some things in shell script. I learned PERL because there were some things I couldn't do in AWK (now I never use AWK). I learned SQL because I needed it to stay employed.

  Posted by: acaldwel on January 03, 2005 at 09:52 AM

• Being able to write a one-off program in some scripting language is a really short-sighted reason for using scripting languages. The real benefit actually comes from programs which must be maintained for a long period of time and which must grow and change during that time.

  I agree that ultimately developing software is about getting the job done. However I think that by limiting your toolbox you are doing yourself a disservice because with languages like Perl, Python, PHP and Ruby in your toolkit you can get the job done faster, which means that you get more done, and these days that matters quite a bit.

  Sincerely,
  Anthony Eden

  Posted by: aeden on January 03, 2005 at 10:03 AM

• A couple of comments, on most of the above.

  I'm pretty well familiar with the Unix toolkit. Ahem, you could almost say I wrote (part of) the book on it (UNIX Power Tools). What I didn't say is that there was some order in the files that I wanted to preserve--that doesn't happen with sort f1 > tmp1; sort f2 > tmp2; diff tmp1 tmp2. Anyway, since I didn't state that, it's not fair to have expected you to know it.

  As far as "different tools in your toolbox": yeah, I know the "to a man with a hammer, everything is a nail" thing. I partly buy it. But I buy it for things like EJB, Spring, JDO, Hibernate, and the such. Languages are big, bulky tools that largely do the same thing. So we're not talking screwdrivers, planes, hammers, and drills here. We're talking a BIG drillpress, a slightly smaller drillpress, a BIG HUMONGOUS drill press with automatic feed--not loads and loads of difference. Some are faster, some are cooler, all do about the same thing.

  That isn't to say that there aren't special-purpose languages, and when you need those, you really need them. But I really don't put Java, Perl, Python, or Ruby into those categories.

  I will confess that, when I was in college, I did call a hammer a "universal unfastening tool", and regularly used it when cooking. Usually to help cut meat that I hadn't bothered to defrost.

  Mike Loukides

  Posted by: mikel on January 03, 2005 at 10:16 AM

• I do have to admit that as the Java library becomes larger, I'm less inclined to slip into another language. With the inclusion of Regular Expressions in Java, the only time I degenerate to shell scripts is when I know I'm writing for an environment where Java isn't likely to be available.

  Sometimes it easier to write things in Bourne shell than to navigate the politics of getting Java/Perl/Ruby/Python installed on a system.

  Your final sentence is important: Ultimately, the only criterion is getting the job done.

  Posted by: acaldwel on January 03, 2005 at 11:32 AM

• First of all I don

  Posted by: applebanana8 on January 04, 2005 at 12:47 AM

• I believe "not having to run javac" and not having to write type declarations make you indeed gain a lot of time when writing programs, when you adopt interactive programming techniques. I wont try to explain why those features are not just mere details, because I dont think those things can be discussed in the abstract. Languages are like chairs, if you have not really used a particular chair, you can not really have a serious opinion of it.

  Posted by: sergiogarcia on January 04, 2005 at 12:57 AM

• 
  Traditionally the benefit of scripted languages over compiled was that they gave better error messages when they crashed (indeed, the fact that they gave some form of error message when they malfunctioned at all was a benefit). With Java this is less of an issue, as the JVM catches and reports all errors, and so only the bugs in the JVM itself remain cryptic.

  Once you no longer have the excuse of better error trapping/reporting 'built in', then scripting languages do become less attractive. Other signature attributes of scripting languages, as you correctly pointed out, don't really amount to much apart from a little less typing. Naturally scripting's ability to 'just run' code is a boon in some areas - how popular would JavaScript (aka ECMAScript) be on the Web if developers and newbie coders needed to download and install development tools before they could get going? (Actually, given the amount of eye candy and dodgey scripts out there perhaps that's not such a good argument in favour of scripting? :-)

  Java is unusual as it sits between traditional compiled languages and scripting. At development time it looks and feels like a compiled language - but at runtime its focus on checking, trapping and reporting errors makes it feel more like a scripting language. If you had to re-write your program again, but your choices were not Ruby versus Java, but Ruby versus C, I suspect you'd find Ruby a bit more attractive (especially if you'd already read that book you've got :-)
  

  Posted by: javakiddy on January 04, 2005 at 04:00 AM

• I definitely agree that lines of codes is not a measure, but there is the right tool for the right job. I generally use scripting languages (Bource Shell and Perl) for small programs.

  
  I've use Perl for years, and IMHO it is the best language to do text processing. I can quickly and easily write a script to parse a text file.

  Posted by: scholnicks on January 04, 2005 at 07:11 AM

• I mostly agree with your points. Perhaps we are the few odd birds, but I find Java to be generally more productive as well.

  "I don't see why it's so difficult to call Runtime.exec()"

  Runtime.exec is not adequately defined. More generally, I take that to be a claim that Java plays well with others. It doesn't even play with other JVMs well enough.

  
  Runtime.exec is too ill defined.
  JNI is much harder than it needs to be
  finding and creating other JVMs is too hard
  RMI rocks, but Java should have facilities for talking to non-JVM processes with objects

  
  - Curt

  Posted by: coxcu on January 04, 2005 at 07:15 AM

• javac/package/deploy can take over 13 minutes on a project I'm working on when the PC is low on physical RAM. The organization simply wouldn't buy more memory. Using BeanShell removes javac/package/deploy and allows me to work efficiently. Eventually, more RAM showed up and the (sometimes) 13 minute job shrunk down to 2 minutes and change. Still, two whole minutes totally wasted to test a change.

  Things are getting much worse with bad tools like Maven that force you to run all of your tests on every compile. (Please correct me if you know how to stop this) This adds a growing wait cycle with no bounds.

  As projects get larger and more folks switch to bad tools like Maven things are just going to get worse.

  Posted by: markswanson on January 04, 2005 at 08:43 AM

• For me, the attraction of scripting (and other weakly-typed, expressive languages) is that of focus -- my own. When I'm solving a problem, be it a one-off script or something significant, I'm generally thinking big picture. Sometimes I'm not even thinking about programming at all. Scripting languages let to sketch out some ideas quickly, without a lot of worrying about data typing and test harnesses and long-term structure, and that means I can keep my main focus where it belongs -- on my problem.

  I don't know about you, but I never write any system just once. I generally plan to write everything at least twice -- the first time to figure out what it is I'm writing, and the second time to produce something that will "stand the test of time". For that first writing, a lot of structure in the language is a major productivity hit -- not just because of the extra keystrokes, but also because of the split of my focus between my problem and its implementation.

  For instance, I've spent the last six months writing a compiler in Ruby. Yes, I'm insane. But here's the thing -- over those months, I have iteratively refactored and rewritten the whole thing a half dozen times. I've swapped out designs and data structures, changed coding standards, moved stuff from one compiler pass to another. Weak typing and expressive syntax made it possible for me to do that while still keeping most of my brain space for figuring out how to turn the input into the output. And when I do occasionally stumble across a data typing problem, I insert a call to a routine that checks type, and usually have the bug killed in a few minutes. Yes, if I'd it written it in Java, I would probably have cleaner code, and it would definitely run faster. I'm just pretty sure it wouldn't be nearly as close to finished.

  Of course, every approach has its costs and its benefits, and different people allocate their brain power in different ways. This approach is what works for me, and I don't promise it will work for anyone else. In any event, what I'm thinking of now, in terms of language design, is a language that combines static typing with type inferencing and optional dynamic typing. I want a language that supports the way I work -- that lets me play dangerously when I'm grappling with the problem, and then lets me set the design in stone later on, once the hard thinking has been done.

  - Chris

  Posted by: chris_poirier on January 04, 2005 at 09:21 AM

• "We're talking a BIG drillpress, a slightly smaller drillpress, a BIG HUMONGOUS drill press with automatic feed--not loads and loads of difference. Some are faster, some are cooler, all do about the same thing."

  Using that analogy applied to woodworking a jig saw does the same thing as a table saw. Have you ever tried to cut an 8 foot piece of wood with a jig saw and maintain a straight line?

  Anyhow, I must not have gotten my point across the first time, so I'm going to try one more time. The benefit of scripting languages is not for one off short little scripts (although they work really well for that), the real benefit is when your applications grow over time and when requirements change as a project progresses.

  Posted by: aeden on January 04, 2005 at 11:05 AM

• It's a case of choosing the right tool for the job. It is possible to create big complicated programs with shell scripts, but you soon run into maintainability and portablility issues, which mean that other tools such as Java are better suited to the task.
  In the unix world, most programmers have a common ground in shell scripts whether they are normally a C/C++ or a Java programmer they can probably both understand and maintain simple shell scripts.
  Recently I wrote a program in Perl then I realised that there was nothing I had done with Perl that I couldn't do with Java, so I converted the whole thing to Java and I am happy my Perl manual is safely back on the shelf.

  Posted by: mel on January 04, 2005 at 12:57 PM

• This is precisely the kind of things I like to prototype in BeanShell, or in the case of a one-time-use, just get done with a bsh script faster. In this case (you are a power developer) I agree, it's not about saving lines of code - it's about not writing pro-forma stuff or using tools that you don't need. First, with BeanShell you don't have to learn a new language, it's just Java. The diffences are small but important: work procedurally if you want to, no need to create a class unless you want to, no need to create a main method unless you want to, no need to declare types unless you *want* to, no need to catch exceptions unless you *want* to, no need to compile... Just write the heart of your algorithm procedurally - the body of a method - and run it as needed. And when you're done, if you want a compilable Java class just beef up your script then or cut and paste.

  Some people will say - "My IDE does all that for me". I agree. But you were probably playing with Unix diff for a while there, right? Does your IDE look good on that platform? How many times did you switch windows to get into and out of the IDE? How man times did you have to go between the command line where your data file was and the IDE's navigator. Did you use grep? How did you pass arguments to your class? I bet you could have typed the important 10 of your 25 lines in your favorite editor a lot faster and been done with it.

  Those are all gut, coder arguments. BeanShell is about more than that... It's about having this kind of Java syntax flexibility in post-deployment situations, for novices, for end-users, for application extension languages, macros, and configuration files. It's about *scaleable syntax* that runs from 100% static Java down to loose script syntax. And the best thing is that you already know the language. Just loosen up accordingly.

  To run a BeanShell script on the command line just grab the jar file and use 'java bsh.Interpreter myfile.bsh'. You can also launch a script from the command line to a web container or remote application for debugging or tweaking without redeployment.

  
  - Pat Niemeyer
  

  Posted by: pat on January 04, 2005 at 04:45 PM

• I agree with the your statement "Ultimately, the only criterion is getting the job done", we should not use a tool or a language because it is hip.

  Can java get the job done always? Many times these scriptish (glue) apps are for nightly and batch jobs, they are developed by programmers, but finally used by system admins, deployment and support teams, who are not that familiar to java. If the glue app is in a scriptish language, they also can hack around if needed. They should not be used for an enterprise level application, and be limited only to these glue tasks.

  However as mentioned by Pat, there is a use for poking around languages to play with a running JVM, to clearout cache, reload some stale data, dynamically change logging levels on ever running server processes with out restarting them.

  Posted by: pulihora on January 04, 2005 at 06:25 PM

• I use both a scripting language (Perl) and a compiled language (Java) on almost a daily basis and I completely disagree with the premise of this article, even if the minor points are valid. For text processing tasks, I write and execute in short rapid iterations to make sure I'm getting what I want. With loose typing, no compilation, concise constructs, exceptionless execution, and transparent access to the shell I save some time with each cycle, and it all adds up. Java continues to get richer, so I do agree it isn't a comparison of capability. However, your antecdote seems to speak to your own knowledge set, not the general case. After all, if Spanish is your weak second language, you'll be able to express yourself faster in English, verdad?On the other hand, I also disagree with commenter aeden: if it's going to stick around with changing requirements, I want compiled code and a strong OO design. 10 KLOC of oft-updated Perl code for a mission-critical enterprise app is a nightmare I've seen and wouldn't wish on my worst enemies. I keep all my scripts around for posterity, but I usually only need to run them a few times before I move on.

  Posted by: mentata on January 05, 2005 at 10:03 PM

• I don't agree with you.
  Scripting in java adds fundamental value to many applications.
  The problems really are when to script ? how to script ?
  what to use ?
  you know what we need ?

  we need a framework with clear rules on how the interaction between scripts and code works which encorauges modular design and leads to well-crafted products.
  So have a look at Seppia and hopefully change your mind.
  Seppia gains from the sinergy of java and javascript

  
  

  Posted by: lorenzop on January 16, 2005 at 02:24 PM

• Scripting is throw-away-after-usage code. It's for those many times when you spent hours writing and debugging a Java class rather than running a few command lines. So for that matter, number of LOC do matter.

  Posted by: alexismp on February 06, 2005 at 12:45 PM

• I don't quite understand the reason for this post... who's putting a gun to your head forcing you to solve your problem in Ruby?
  In the time you typed that post, you could have solved your 2nd task in Java (assuming your time estimate was right).

  And why do you spread your misconception that *all* code written in scripting languages must resemble your college APL code (ie. consist of only one-liners)? Check out a RubyOnRails application (or the Rails code itself) to see some proper coding style.

  Also: use proper tools then some of the dynamic problems will disappear:

  editors can emit warnings if 2 variables appear, that have only minor spelling differences (prevents typing errors)

  autocomplete prevents typos in the first place

  editors will show warnings for non-used variables or variables that are only assigned and never read (again, to prevent typo errors)

  unit testing for interfaces of classes

  etc.

  
  And if you use Beanshell http://beanshell.org/ you won't even have to learn a new language, but can reap the benefits of incrementally building up your code. Not to mention that you can still use your Java library knowledge to get the job done.

  And if you do get around to reading the Programming Ruby book, there's JRuby http://jruby.sourceforge.net/

  Posted by: murphee on December 05, 2005 at 09:48 AM

This work is licensed under a Creative Commons License.