Serialization surprises
Posted by survivant on August 6, 2010 at 10:12 AM EDT
CONTEXT
I want to share a problem that we had in our project. We were doing a real-time "Profits and Loss" server (P&L). The server sends stock updates to all the users subscribed to the stocks, basically as Google Finance or Yahoo Finance.
SIMPLE IMPLEMENTATION
I will used a basic approach (no aggregation and no optimization) to explain the problem that we had with Serialization and Hessian.
We kept the stock prices (BID and ASK) in a simple object : SymbolSerializable.
When a update is received for a stock, we update the values in his SymbolSerializable and send back the updated values to all client subscribed to this stock.
...
symbol.setBid(update.getBid());
symbol.setAsk(update.getAsk());
...
for(Client client : clientList){
client.sendUpdate(symbol);
}
....
It can't be more simpler than that. The client will received each updates on his stocks.
The problems that we had was surprising. The clients were receiving the always the same price on a stock !
WHY ?
here an example :
stock : ABC
Updates :
1 : bid=10.25$, ask=10.50$
2 : bid=11.50$, ask=11.75$
3 : bid=12.00$, ask=12.15$
and the clients received :
1 : bid=10.25$, ask=10.50$
2 : bid=10.25$, ask=10.50$
3 : bid=10.25$, ask=10.50$
To debug our server we printed the values sent to the client, and we saw in the server's logs that the values were corrects.
value sent to client :
bid=10.25$, ask=10.50$
value sent to client :
bid=11.50$, ask=11.75$
value sent to client :
bid=12.00$, ask=12.15$
Everything look fine, why it doesn't work on the client side ?
The only thing that could cause the problem were our Serialization in the server. I'll show you four different test cases. See the implementations below.
- #1 : Reference implementation using Serializable (failed)
- #2 : Changed Serializable for Externalizable interface (failed)
- #3 : Serializable using new (passed)
- #4 : Serializable using reset (passed)
| #1 : Serializable : Reference | #2 : Externalizable | #3 : Serializable with new | #4 : Serializable with reset |
|---|---|---|---|
|
|
|
|
package ca.sebastiendionne.demo.model;
import java.io.Serializable;
public class SymbolSerializable implements Serializable {
private static final long serialVersionUID = 7853892880704628717L;
Double bid = null;
Double ask = null;
public SymbolSerializable(){
}
public Double getBid() {
return bid;
}
public void setBid(Double bid) {
this.bid = bid;
}
public Double getAsk() {
return ask;
}
public void setAsk(Double ask) {
this.ask = ask;
}
@Override
public String toString() {
return "bid = " + bid + " ask=" + ask;
}
}
Here the Externalizable pojo
package ca.sebastiendionne.demo.model;
import java.io.Externalizable;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;
public class SymbolExternalizable implements Externalizable {
private static final long serialVersionUID = 1853892880704628717L;
Double bid = null;
Double ask = null;
public SymbolExternalizable(){
}
public Double getBid() {
return bid;
}
public void setBid(Double bid) {
this.bid = bid;
}
public Double getAsk() {
return ask;
}
public void setAsk(Double ask) {
this.ask = ask;
}
@Override
public String toString() {
return "bid = " + bid + " ask=" + ask;
}
@Override
public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
bid = (Double)in.readObject();
ask = (Double)in.readObject();
}
@Override
public void writeExternal(ObjectOutput out) throws IOException {
out.writeObject(bid);
out.writeObject(ask);
}
}
For the test cases I use a for loop to create Symbol and serialized on the hard drive and unserialized them back.
Here the code for the test case #1 (used as reference)
package ca.sebastiendionne.demo;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import ca.sebastiendionne.demo.model.SymbolSerializable;
public class SerializationFailed {
/**
* @param args
*/
public static void main(String[] args) throws Exception {
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser")));
SymbolSerializable symbol = new SymbolSerializable();
// dummy values
symbol.setAsk(new Double(-1));
symbol.setBid(new Double(-1));
for(int i=0;i<10;i++){
// update values
symbol.setAsk(new Double(i));
symbol.setBid(new Double(i));
oos.writeObject(symbol);
oos.flush();
}
oos.flush();
oos.close();
// read
ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));
try {
Object obj = null;
while((obj = ois.readObject())!=null){
System.out.println((SymbolSerializable)obj);
}
} catch (EOFException e) {
System.out.println("All items read");
} catch(Exception e){
e.printStackTrace();
} finally {
ois.close();
}
}
}
Here the code for the test case #2
package ca.sebastiendionne.demo;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import ca.sebastiendionne.demo.model.SymbolExternalizable;
public class SerializationFailed2 {
/**
* @param args
*/
public static void main(String[] args) throws Exception {
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser")));
SymbolExternalizable symbol = new SymbolExternalizable();
// dummy values
symbol.setAsk(new Double(-1));
symbol.setBid(new Double(-1));
for(int i=0;i<10;i++){
// update values
symbol.setAsk(new Double(i));
symbol.setBid(new Double(i));
oos.writeObject(symbol);
oos.flush();
}
oos.flush();
oos.close();
// read
ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));
try {
Object obj = null;
while((obj = ois.readObject())!=null){
System.out.println((SymbolExternalizable)obj);
}
} catch (EOFException e) {
System.out.println("All items read");
} catch(Exception e){
e.printStackTrace();
} finally {
ois.close();
}
}
}
Here the code for the test case #3
package ca.sebastiendionne.demo;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import ca.sebastiendionne.demo.model.SymbolSerializable;
public class SerializationWithNew {
/**
* @param args
*/
public static void main(String[] args) throws Exception {
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser")));
SymbolSerializable symbol = new SymbolSerializable();
// dummy values
symbol.setAsk(new Double(-1));
symbol.setBid(new Double(-1));
for(int i=0;i<10;i++){
// update values
symbol.setAsk(new Double(i));
symbol.setBid(new Double(i));
SymbolSerializable symbol2 = new SymbolSerializable();
symbol2.setAsk(symbol.getAsk());
symbol2.setBid(symbol.getBid());
oos.writeObject(symbol2);
oos.flush();
}
oos.flush();
oos.close();
// read
ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));
try {
Object obj = null;
while((obj = ois.readObject())!=null){
System.out.println((SymbolSerializable)obj);
}
} catch (EOFException e) {
System.out.println("All items read");
} catch(Exception e){
e.printStackTrace();
} finally {
ois.close();
}
}
}
Here the code for the test case #4
package ca.sebastiendionne.demo;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import ca.sebastiendionne.demo.model.SymbolSerializable;
public class SerializationWithReset {
/**
* @param args
*/
public static void main(String[] args) throws Exception {
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(new File("test.ser")));
SymbolSerializable symbol = new SymbolSerializable();
// dummy values
symbol.setAsk(new Double(-1));
symbol.setBid(new Double(-1));
for(int i=0;i<10;i++){
// update values
symbol.setAsk(new Double(i));
symbol.setBid(new Double(i));
oos.writeObject(symbol);
oos.flush();
oos.reset(); // magic line
}
oos.flush();
oos.close();
// read
ObjectInputStream ois = new ObjectInputStream(new FileInputStream(new File("test.ser")));
try {
Object obj = null;
while((obj = ois.readObject())!=null){
System.out.println((SymbolSerializable)obj);
}
} catch (EOFException e) {
System.out.println("All items read");
} catch(Exception e){
e.printStackTrace();
} finally {
ois.close();
}
}
}
I can't say what is the performance impact if we use reset() on each updates sent. It is better to use new or reset ?
You can follow me on Twitter : http://twitter.com/survivant
Related Topics >>
Blog Links >>
- Login or register to post comments
- Printer-friendly version
- survivant's blog
- 1660 reads
Comments
reset() or writeUnshared() seem to be the only ways
by cobrien - 2010-08-09 08:59
We use serialization a lot, so this perplexed me.After running some tests, it seems the two ways to have one instance, change the fields, and have the modified fields written to the object output stream is to call reset() on the object output stream after each write, or use writeUnshared(). Just adding hash/equals based on the fields didn't work Calling flush() didn't work either.
This is with a basic test serializing to/deserializing from a byte array (out/in) stream.
A better approach is to new() the container each time.
Don't mess with reset.
by cayhorstmann - 2010-08-08 09:03
Don't mess with reset. Serialize a separate instance of Symbol in each iteration of the loop. Or, even better, serialize a single ArrayList<Symbol>:
In general, my recommendation is to serialize a single object. Then you only need to use your general Java knowledge to debug any problems with shared references.
BTW, the serialization protocol is well documented, for example in Core Java vol. 2 ch. 1.
Cheers,
Cay
so what you suggest is to
by survivant - 2010-08-08 10:43
so what you suggest is to serialize clientList instead of looping.
What's the JVM used?
by melbeltagy - 2010-08-08 01:16
What's the JVM used? is it Sun's JVM, JRockit, or IBM's.??in this example, I used Sun
by survivant - 2010-08-08 06:27
in this example, I used Sun JVM, but at my last job it was Sun JVM and IBM (from Websphere 6 and 7)
Most likely equals/hashCode based caching
by peterbecker - 2010-08-07 04:08
My guess would be that serialization has cached the object. Since you did not override equals() and hashCode() the change to the fields will not change the object from a serialization point of view, so it skips doing anything. That's just a theory, but try adding some standard equals(..) and hashCode() methods and see if it helps.I think so...
by vlasak - 2010-08-09 06:16
I would vote for your guess. It is written in documentation that the serialization mechanism can make use of writing of handles instead of "full" objects for the instances that were already wrote in the stream. That implies that some form of hash table cache is probablz used. With missing equals() and hashcode() it could make the mess you are describing. It is always strongly recommended practise to implement those methods. As well as it is good practice to make the object immutable if there is no serious reason why the object should not be immutable.So if your Symbol object would be immutable, you will need to create new instances and the serialization will go ok (even with missing equals()/hashcode() as the identity of instances will be different as they are different instances... but still, having such methods there is really important for further placing of instances into collections, etc.)
Better solution
by ronaldtm - 2010-08-06 09:34
Make Symbol immutable.Or, if for whatever stupid reason you can't, at least re-instantiate it every time, instead of re-instantiating the stream (duh!).
thanks. Look like using
by survivant - 2010-08-06 10:11
thanks. Look like using
oos.writeUnshared(symbol);
works fine too in my example. Too bad that I can't test that with the real application, I left this job.
We read the javadoc and google about our problem, but didn't find an anwser. So we fool around.
writeUnshared is not as
by tackline - 2010-08-06 11:55
writeUnsharedis not as useful as it appears. The "unshared" bit is only for the immediate object referenced. So the effect is dependent upon the internal form of the object. Immutability is best.how would you make Symbol
by survivant - 2010-08-06 12:02
how would you make Symbol immutable in this example ?