Skip to main content

Serialization surprises

Posted by survivant on August 6, 2010 at 7:12 AM PDT

CONTEXT

I want to share a problem that we had in our project. We were doing a real-time "Profits and Loss" server (P&L). The server sends stock updates to all the users subscribed to the stocks, basically as Google Finance or Yahoo Finance.

SIMPLE IMPLEMENTATION

I will used a basic approach (no aggregation and no optimization) to explain the problem that we had with Serialization and Hessian.

The server always keep the last value on the stocks, because when a user ask for a quote we want to send back as soon as possible the
last value (from the cache) that we had on that stock).

We kept the stock prices (BID and ASK) in a simple object : SymbolSerializable.

When a update is received for a stock, we update the values in his SymbolSerializable and send back the updated values to all
client subscribed to this stock.

[prettify]
...

symbol.setBid(update.getBid());
symbol.setAsk(update.getAsk());

...

for(Client client : clientList){
    client.sendUpdate(symbol);
}

....
[/prettify]
It can't be more simpler than that. The client will received each updates on his stocks.

The problems that we had was surprising. The clients were receiving the always the same price on a stock !

WHY ?

here an example :

stock : ABC

Updates :

1 : bid=10.25$, ask=10.50$
2 : bid=11.50$, ask=11.75$
3 : bid=12.00$, ask=12.15$

and the clients received :

1 : bid=10.25$, ask=10.50$
2 : bid=10.25$, ask=10.50$
3 : bid=10.25$, ask=10.50$

To debug our server we printed the values sent to the client, and we saw in the server's logs that the values were corrects.

value sent to client :
bid=10.25$, ask=10.50$

value sent to client :
bid=11.50$, ask=11.75$

value sent to client :
bid=12.00$, ask=12.15$

Everything look fine, why it doesn't work on the client side ?

INVESTIGATION/SOLUTION

The only thing that could cause the problem were our Serialization in the server.

I'll show you four different test cases. See the implementations below.

  • #1 : Reference implementation using Serializable (failed)
  • #2 : Changed Serializable for Externalizable interface (failed)
  • #3 : Serializable using new (passed)
  • #4 : Serializable using reset (passed)
#1 : Serializable : Reference #2 : Externalizable #3 : Serializable with new #4 : Serializable with reset
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • All items read
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • bid= 0.0 ask= 0.0
  • All items read
  • bid= 0.0 ask= 0.0
  • bid= 1.0 ask= 1.0
  • bid= 2.0 ask= 2.0
  • bid= 3.0 ask= 3.0
  • bid= 4.0 ask= 4.0
  • bid= 5.0 ask= 5.0
  • bid= 6.0 ask= 6.0
  • bid= 7.0 ask= 7.0
  • bid= 8.0 ask= 8.0
  • bid= 9.0 ask= 9.0
  • All items read
  • bid= 0.0 ask= 0.0
  • bid= 1.0 ask= 1.0
  • bid= 2.0 ask= 2.0
  • bid= 3.0 ask= 3.0
  • bid= 4.0 ask= 4.0
  • bid= 5.0 ask= 5.0
  • bid= 6.0 ask= 6.0
  • bid= 7.0 ask= 7.0
  • bid= 8.0 ask= 8.0
  • bid= 9.0 ask= 9.0
  • All items read

Here the Serializable pojo

[prettify]
package ca.sebastiendionne.demo.model;

import java.io.Serializable;

public class SymbolSerializable implements Serializable {

    private static final long serialVersionUID = 7853892880704628717L;

    Double bid = null;
    Double ask = null;

    public SymbolSerializable(){
    }

    public Double getBid() {
        return bid;
    }
    public void setBid(Double bid) {
        this.bid = bid;
    }
    public Double getAsk() {
        return ask;
    }
    public void setAsk(Double ask) {
        this.ask = ask;
    }

    @Override
    public String toString() {
        return "bid = " + bid + "  ask=" + ask;
    }

}
[/prettify]
Here the Externalizable pojo
[prettify]
package ca.sebastiendionne.demo.model;

import java.io.Externalizable;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;

public class SymbolExternalizable implements Externalizable {
    private static final long serialVersionUID = 1853892880704628717L;

    Double bid = null;
    Double ask = null;
    
    public SymbolExternalizable(){
        
    }
    
    public Double getBid() {
        return bid;
    }
    public void setBid(Double bid) {
        this.bid = bid;
    }
    public Double getAsk() {
        return ask;
    }
    public void setAsk(Double ask) {
        this.ask = ask;
    }
    
    @Override
    public String toString() {
        return "bid = " + bid + "  ask=" + ask;
    }

    @Override
    public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
        bid = (Double)in.readObject();
        ask = (Double)in.readObject();
    }

    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
        out.writeObject(bid);
        out.writeObject(ask);
    }
    
}
[/prettify]

For the test cases I use a for loop to create Symbol and serialized on the hard drive and unserialized them back.

Here the code for the test case #1 (used as reference)

[prettify]
package ca.sebastiendionne.demo;

import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

import ca.sebastiendionne.demo.model.SymbolSerializable;

public class SerializationFailed {

    /**
     * @param args
     */
    public static void main(String[] args) throws Exception {
        
        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser"))); 
        
        
        SymbolSerializable symbol = new SymbolSerializable();
        
        // dummy values
        symbol.setAsk(new Double(-1));
        symbol.setBid(new Double(-1));
        
        for(int i=0;i<10;i++){
            // update values
            symbol.setAsk(new Double(i));
            symbol.setBid(new Double(i));
            
            oos.writeObject(symbol);
            oos.flush();
        }
        
        oos.flush();
        oos.close();
        
        // read
        
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));

        try {
            Object obj = null;
            while((obj = ois.readObject())!=null){
                System.out.println((SymbolSerializable)obj);
            }
        } catch (EOFException e) {
            System.out.println("All items read");
        } catch(Exception e){
            e.printStackTrace();
        } finally {
            ois.close();
        }
        
        
    }

}
[/prettify]
Here the code for the test case #2
[prettify]
package ca.sebastiendionne.demo;

import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

import ca.sebastiendionne.demo.model.SymbolExternalizable;

public class SerializationFailed2 {

    
    /**
     * @param args
     */
    public static void main(String[] args) throws Exception {
        
        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser"))); 
        
        
        SymbolExternalizable symbol = new SymbolExternalizable();
        
        // dummy values
        symbol.setAsk(new Double(-1));
        symbol.setBid(new Double(-1));
        
        for(int i=0;i<10;i++){
            // update values
            symbol.setAsk(new Double(i));
            symbol.setBid(new Double(i));
            
            oos.writeObject(symbol);
            oos.flush();
        }
        
        oos.flush();
        oos.close();
        
        // read
        
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));

        try {
            Object obj = null;
            while((obj = ois.readObject())!=null){
                System.out.println((SymbolExternalizable)obj);
            }
        } catch (EOFException e) {
            System.out.println("All items read");
        } catch(Exception e){
            e.printStackTrace();
        } finally {
            ois.close();
        }
        
        
    }

}
[/prettify]
Here the code for the test case #3
[prettify]
package ca.sebastiendionne.demo;

import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

import ca.sebastiendionne.demo.model.SymbolSerializable;

public class SerializationWithNew {

    /**
     * @param args
     */
    public static void main(String[] args) throws Exception {
        
        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser"))); 
        
        
        SymbolSerializable symbol = new SymbolSerializable();
        
        // dummy values
        symbol.setAsk(new Double(-1));
        symbol.setBid(new Double(-1));
        
        for(int i=0;i<10;i++){
            // update values
            symbol.setAsk(new Double(i));
            symbol.setBid(new Double(i));
            
            SymbolSerializable symbol2 = new SymbolSerializable();
            symbol2.setAsk(symbol.getAsk());
            symbol2.setBid(symbol.getBid());
            
            oos.writeObject(symbol2);
            oos.flush();
        }
        
        oos.flush();
        oos.close();
        
        // read
        
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));

        try {
            Object obj = null;
            while((obj = ois.readObject())!=null){
                System.out.println((SymbolSerializable)obj);
            }
        } catch (EOFException e) {
            System.out.println("All items read");
        } catch(Exception e){
            e.printStackTrace();
        } finally {
            ois.close();
        }
        
        
    }

}
[/prettify]
Here the code for the test case #4
[prettify]
package ca.sebastiendionne.demo;

import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

import ca.sebastiendionne.demo.model.SymbolSerializable;

public class SerializationWithReset {

    /**
     * @param args
     */
    public static void main(String[] args) throws Exception {
        
        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser"))); 
        
        
        SymbolSerializable symbol = new SymbolSerializable();
        
        // dummy values
        symbol.setAsk(new Double(-1));
        symbol.setBid(new Double(-1));
        
        for(int i=0;i<10;i++){
            // update values
            symbol.setAsk(new Double(i));
            symbol.setBid(new Double(i));
            
            oos.writeObject(symbol);
            oos.flush();
            oos.reset(); // magic line
        }
        
        oos.flush();
        oos.close();
        
        // read
        
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));

        try {
            Object obj = null;
            while((obj = ois.readObject())!=null){
                System.out.println((SymbolSerializable)obj);
            }
        } catch (EOFException e) {
            System.out.println("All items read");
        } catch(Exception e){
            e.printStackTrace();
        } finally {
            ois.close();
        }
        
        
    }

}
[/prettify]

I can't say what is the performance impact if we use reset() on each updates sent. It is better to use new or reset ?

You can follow me on Twitter : http://twitter.com/survivant

Related Topics >>

Comments

reset() or writeUnshared() seem to be the only ways

We use serialization a lot, so this perplexed me.

After running some tests, it seems the two ways to have one instance, change the fields, and have the modified fields written to the object output stream is to call reset() on the object output stream after each write, or use writeUnshared(). Just adding hash/equals based on the fields didn't work Calling flush() didn't work either.

This is with a basic test serializing to/deserializing from a byte array (out/in) stream.

A better approach is to new() the container each time.

Don't mess with reset.

Don't mess with reset. Serialize a separate instance of Symbol in each iteration of the loop. Or, even better, serialize a single ArrayList<Symbol>:

 ArrayList<Symbol> syms = new ArrayList<Symbol>(); for (int i=0;i<10;i++) {
Symbol symbol = new Symbol(); // New instance for each symbol
symbol.setAsk(new Double(i));
symbol.setBid(new Double(i));
syms.add(symbol); }
oos.writeObject(syms);

In general, my recommendation is to serialize a single object. Then you only need to use your general Java knowledge to debug any problems with shared references.

BTW, the serialization protocol is well documented, for example in Core Java vol. 2 ch. 1.

 

Cheers,

Cay

so what you suggest is to

so what you suggest is to serialize clientList   instead of looping.

 

 

What's the JVM used?

What's the JVM used? is it Sun's JVM, JRockit, or IBM's.??

in this example, I used Sun

in this example, I used Sun JVM, but at my last job it was Sun JVM and IBM (from Websphere 6 and 7)

Most likely equals/hashCode based caching

My guess would be that serialization has cached the object. Since you did not override equals() and hashCode() the change to the fields will not change the object from a serialization point of view, so it skips doing anything. That's just a theory, but try adding some standard equals(..) and hashCode() methods and see if it helps.

I think so...

I would vote for your guess. It is written in documentation that the serialization mechanism can make use of writing of handles instead of "full" objects for the instances that were already wrote in the stream. That implies that some form of hash table cache is probablz used. With missing equals() and hashcode() it could make the mess you are describing. It is always strongly recommended practise to implement those methods. As well as it is good practice to make the object immutable if there is no serious reason why the object should not be immutable.

So if your Symbol object would be immutable, you will need to create new instances and the serialization will go ok (even with missing equals()/hashcode() as the identity of instances will be different as they are different instances... but still, having such methods there is really important for further placing of instances into collections, etc.)

Better solution

Make Symbol immutable.

Or, if for whatever stupid reason you can't, at least re-instantiate it every time, instead of re-instantiating the stream (duh!).

thanks.  Look like using

thanks.  Look like using

oos.writeUnshared(symbol);

 

works fine too in my example.  Too bad that I can't test that with the real application, I left this job.

 

We read the javadoc and google about our problem, but didn't find an anwser.  So we fool around.

writeUnshared is not as

writeUnshared is not as useful as it appears. The "unshared" bit is only for the immediate object referenced. So the effect is dependent upon the internal form of the object. Immutability is best.

how would you make Symbol

how would you make Symbol immutable in this example ?