Skip to main content

Handling Poison Messages with Glassfish

Posted by felipegaucho on September 24, 2009 at 4:06 AM PDT

Poison messages are basically delivery deadlocks caused
by a continuous redelivery of a message to a JMS Queue or Topic. That
usually happens due to a code bug or configuration problems in the
project.

How to reproduce poison messages

The easiest way of reproducing the poison messages issue
is to create a Message Driven Bean and then to throw an exception in its
href="http://java.sun.com/javaee/5/docs/tutorial/doc/bnbpo.html#bnbpp">onMessage
method, like the example below.

@MessageDriven(activationConfig = {@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")}, mappedName = "MyQueue")
public class RegistrationMessageBean implements MessageListener {
@Override
public void onMessage(Message registration) {
throw new RuntimeException("poison message");
}
}

Fixing the JMS deadlock

What happens is that JMS relies on href="http://java.sun.com/products/jms/faq.html#relship_transac">transactions
to guarantee that all messages in a Queue will be delivered despite any
temporary problems in the message consumer. If a message consumer (MDB)
throws an exception, the JMS server tries to redeliver the message (transaction rollback). Only if a method consuming the message finishes without error the transaction will be committed and then the message will be removed from the Queue (acknowledged). In
the sample above, the message consumer will always throw an exception
and due to that the server will always redeliver the message - a deadlock.

The workarounds to fix such a problem are:

  1. Fix all bugs from your code: that's the
    general best solution for poison messages, but as you know bugs are
    intrinsic to any software and it is not a surprise the time you will
    loose fighting against the poison messages :)
  2. Try-Catch and digest the exceptions: for the
    paranoid, a good choice is to surround the whole onMessage
    code with a try{...} catch(Exception e){ ... }. Even if
    you strongly believe your code is sound, it is a recommended practice
    to do that. So, rewriting our sample code in a safe way, it looks like
    this:
    @Override
    public void onMessage(Message registration) {
    try {
    throw new RuntimeException("poison message");
    } catch(Exception error) {
    logger.severe("I am ignoring the JMS exception: " + e.getMessage());
    }
    }

The second solution is a robust way of guaranteeing the consume
of a message despite any problems. Not so elegant, but without
exceptions in the onMessage method, the Message
will be acknowledged and the JMS transaction will be href="http://72.5.124.55/javaee/5/docs/api/javax/jms/Session.html#commit()">committed.

A more complicated scenario with sub-transactions

The previous solution works for the general case, a simple Java
code inside the onMessage method. Problem is, JMS uses the

Java
Transaction API (JTA)
to control the message transactions and the JTA
API supports sub-transactions. So imagine if the onMessage
calls a JPA transactional method:

@PersistenceUnit(name = "arenapuj")
protected EntityManagerFactory emf;

@Override
public void onMessage(Message registration) {
try {
create(new MyJpaEntity());
} catch(Exception error) {
logger.severe("I am ignoring the JMS exception: " + e.getMessage());
}
}

public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
EntityManager manager = emf.createEntityManager();
try {
manager.persist(entity);
manager.flush();
return entity;
} finally {
if (manager != null && manager.isOpen()) {
manager.close();
}
}
}

The robust try-catch block is still there but guess
what: if the method JPA Transaction of the method create is
rolled back, the JMS transaction is also rolled back, causing the poison
message problem despite the try-catch block on the
onMessage code. So, the comfortable robustness provided by the try-catch
block is actually a trap, a silent killer in the JMS sub-transactions
scenario.

How to avoid poison messages caused by sub-transactions?

You should annotate the method with the first sub-transaction
with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
to avoid the dependencies between the JMS Transaction and its
sub-transactions. Notice that only the first sub-transaction needs to be
decoupled from the JMS one to avoid poison messages.

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
EntityManager manager = emf.createEntityManager();
try {
manager.persist(entity);
manager.flush();
return entity;
} finally {
if (manager != null && manager.isOpen()) {
manager.close();
}
}
}

Done, now your JMS method is decoupled from the new JPA
transaction and if the JPA code rolls back, the JMS message will be
acknowledged anyway.

Disclaimer: I am abusing the terms "

JMS
transaction
" and "JPA transactions" for the sake of
clarification here. Actually there are no such things, we have only
the transactions defined in the href="http://java.sun.com/javaee/technologies/jta/index.jsp">Java
Transaction API (JTA). For me it seems simpler to visualize the problem
thinking about the methods and their transactions scope separately but
after all it is all about JTA Transactions
:)

Handling problems outside the code

Other common JMS scenario is to have resources problems, like
connection failures, database down, etc. If we have a message consumer
down and a message producer working, the producer will try to send a
message, fail and try to send the message again. It would cause another
type of deadlock, but the Java EE containers provide a set of
configuration options to prevent such problems. The href="http://docs.sun.com/app/docs/doc/820-7692/gbtvf?a=view">ActivationSpec
for Message Driven Beans specifies two annotations to workaround
activating problems:

  1. endpointExceptionRedeliveryAttempts: Number
    of times to redeliver a message when MDB throws an exception during
    message delivery
  2. sendUndeliverableMsgsToDMQ: Place message in
    dead message queue when MDB throws a runtime exception and number of
    redelivery attempts exceeds the value of
    endpointExceptionRedeliveryAttempts? If false, the Message Queue broker
    will attempt redelivery of the message to any valid consumer, including
    the same MDB.

You may check those configurations in your code review, but it is
not so critical since their default values are reasonable for the common
scenarios. And you need to know container specific attributes. The href="http://docs.sun.com/app/docs/doc/820-7692/gbtvf?a=view">Glassfish
V3 activation properties changed a bit, and I suppose for the other
containers we will find different names. Fortunately we can expect that
all defaults work fine and we are not forced to dig in product-specific
details all the time.

Summary

JMS is one of the most powerful java EE resources available for
developers and architects, but it is very important that anyone
designing such applications knows in deep the specification and also
some implementation tricks. Poison messages can make your server and
eventually the whole host machine to hang for hours - forcing a machine to
restart. It is a common problem, in my opinion weakly supported by the
containers and a problem we should know about. The goal of this blog
entry is just to give you a chance to identify the poison messages
problem of your application. to know more about JMS and its details, you
need to read more and the links below seem to be a good starting point:

  • href="http://java.sun.com/javaee/5/docs/tutorial/doc/bncfu.html">Creating
    Robust JMS Applications - The Java EE 5 Tutorial
  • href="http://today.java.net/pub/a/today/2008/01/22/jms-messaging-using-glassfish.html">JMS
    Messaging Using GlassFish by Deepa Sobhana
  • href="http://software.intel.com/en-us/articles/bitter-messages-java-messaging-anti-patterns/">Bitter
    Messages: Java* Messaging Anti-Patterns by Bruce A. Tate
  • href="http://forums.java.net/jive/message.jspa?messageID=334997">Thread:
    how can i setup glassfish to deal with "poison message"

Aknowledgment: a special thanks for
Marina Vatkina and Nigel Deakin for their friendly support through the href="https://glassfish.dev.java.net/servlets/SummarizeList?listName=users&by=date">Glassfish's
mailing list. And a kudo to "The Professor" and his wise hints on this
topic.

Related Topics >>