Monday, August 12, 2013

How To Solve JSON infinite recursion Stackoverflow
(with Spring and Jackson annotations)

Infinite recursion picture

Coding in Spring 3 with JPA and Jackson is supposed to semplify your life, sparing you from writing thousand lines of code. Sometimes, though, you'll notice that all these annotations hide the real core of your application, preventing you to really understand what your code is doing (and what it is supposed to do).
This means that if you want to develop a good java application or website, you need to carefully read the docs or (more realistically) you need to understand how to read your error logs and combine them with a well-written search engine query.

This is what I had to do when I first met the problem I'm about to write about.



The problem.

I was getting a wrong JSON response while using a many-to-many relationship, that caused my application to enter in an infinite loop giving as output an infinite recursion Stackoverflow error.

Here is part of what I found in my Eclipse console:

Request processing failed; nested exception is org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: Infinite recursion (StackOverflowError) (through reference chain: [here is the loop]) with root cause
java.lang.StackOverflowError
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:567)
 at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:143)
 at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:118)
 at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:24)
 at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:180)
 at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:544)
 at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:551)
 ...

and so on.

Since I was using Datatables jQuery plugin, I also had this output in my browser window:

DataTables warning (table id = 'products'): DataTables warning: JSON data from server could not be parsed. This is caused by a JSON formatting error.

What I noticed is that the same database query, without JSON, was working well. So the problem (as the console in fact said) was with Jackson and Json serialization. As the docs state (see references), bi-directional references for ORM-managed beans (iBatis, Hibernate) would cause serialization to failure since they are cyclic dependencies. Since Jackson 1.6, this problem has been solved by the introduction of two new annotations: @JsonManagedReference and @JsonBackReference (and see the end of this post to give a look at the @JsonIdentityInfo annotation).

How does serialization work? (The solution in theory)
In computer science, in the context of data storage and transmission, serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and resurrected later in the same or another computer environment.

For Jackson to work well, one of the two sides of the relationship should not be serialized, in order to avoid the annoying infinite recursive loop that causes our stackoverflow error.

So, Jackson takes the forward part of the reference, for example an attribute of a java class (i.e. List<Role> roles in User class), and converts it in a json-like storage format; this is the so-called marshalling process.
Then, Jackson looks for the back part of the reference (i.e. List<User> users in Role class) and leaves it as it is, not serializing it. This part of the relationship will be re-constructed during the deserialization (unmarshalling) of the forward reference.

The solution in practice.

It's very simple. Assuming that your database query already works without JSON, all you have to do is this:

  1. Add the @JsonManagedReference In the forward part of the relationship (i.e. User.java class):
    @Entity
    public class User implements java.io.Serializable{
     
     @Id
     @GeneratedValue(strategy=GenerationType.IDENTITY)
     private long id;
     
     @Column(name="name")
     private String name;
    
     @ManyToMany
     @JoinTable(name="users_roles",joinColumns=@JoinColumn(name = "user_fk"),
     inverseJoinColumns=@JoinColumn(name = "role_fk"))
     @JsonManagedReference
     private Set<Role> roles = new HashSet<Role>();
    
    ...
    
  2. Add the @JsonBackReference In the back part of the relationship (i.e. Role.java class):
    @Entity
    public class Role implements java.io.Serializable {
    
     @Id 
     @GeneratedValue(strategy=GenerationType.IDENTITY)
     private int id;
    
     @ManyToMany(mappedBy="roles")
     @JsonBackReference
     private Set<User> users = new HashSet<User>();
    
    ...
    

The work is done. If you take a look at your firebug logs, you'll notice that the infinite recursive loop has disappeared.


Edit (02/09/2013)

Another useful annotation you could check is @JsonIdentityInfo: using it, everytime Jackson serializes your object, it will add an ID (or another attribute of your choose) to it, so that it won't entirely "scan" it again everytime. This can be useful when you've got a chain loop between more interrelated objects (for example: Order -> OrderLine -> User -> Order and over again).

In this case you've got to be careful, since you could need to read your object's attributes more than once (for example in a products list with more products that share the same seller), and this annotation prevents you to do so. I suggest to always take a look at firebug logs to check the Json response and see what's going on in your code.


References

66 comments:

  1. Hi,
    The solution is great and simple. But it does not work for me in case when an object has a self-reference(in a tree-like structure). Have you ever encountered the same problem and how have you solved it?

    Thanks,
    Kostya

    ReplyDelete
    Replies
    1. Hi Kostya.
      If by "self-reference" you mean a hyerarchical structure with parent-child relationships between objects of the same class, these annotations should work as well. For example: if you've got a Category class and you want to nest categories (e.g.: "smartphone" is a child of "technology") you could add a @JsonBackReference to the parent property and a @JsonManagedReference to the child property:

      @OneToOne
      @JsonBackReference
      private Category parent;

      @OneToMany
      @JsonManagedReference
      private List children;

      Another useful annotation you could check is @JsonIdentityInfo: using it, everytime jackson serializes your object, it will add an ID (or another attribute of your choose) to it, so that it won't entirely "scan" it again everytime. This can be useful when you've got a chain loop between more interrelated objects (for example: Order -> OrderLine -> User -> Order and over again ).
      In this case you've got to be careful, since you could need to read your object's attributes more than once (for example in a products list with more products that share the same seller), and this annotation prevents you to do so.

      I hope I've answered your question :)

      Delete
    2. I've just found the root of my problem. It was our custom model generator that messed things up. As for Jackson, it works as expected.

      Thanks for your help!

      Delete
    3. This comment has been removed by the author.

      Delete
  2. Your post is well explained and really helped me. The @JsonManagedReference and @JsonBackReference worked for me. Thanks

    ReplyDelete
  3. I had the same problem with user and role and followed your solution. It solved the problem but I still have one question left. If I read the user object I get the user including the roles. But if I want to read the role object how can I ensure that I also could get all users of a role. What I mean is, using @JsonManagedReference and @JsonBackReference solves the problem in one direction but I think not in the other direction, right?

    ReplyDelete
    Replies
    1. I think it should solve the problem in both directions.
      If you've got a Role and you want to retrieve its Users you can simply do that because of the bidirectional relationship between Role and User objects.
      Using json, those annotations ensure that when you'll get the Users they won't try to give you back the Roles again (so that you won't enter a loop).

      Delete
    2. But If I have the Users only, how can I access the role?

      Delete
  4. I just want to say THANK YOU !! I became crazy about an infinite recursion in my Json, I came here by Googled to solve the problem and your post is very helpful, @JsonManagedReference and @JsonBackReference worked fine for me ! :)

    ReplyDelete
    Replies
    1. You're very welcome, I'm glad I've been useful to you :)

      Delete
  5. Thank you! Perfect in my case, too. Thank you, man.

    ReplyDelete
    Replies
    1. You're welcome! Thank you for your kind comment :)

      Delete
  6. Fantastic !
    It fixed my problem ... I have been struggling several days to fix this one.
    Thanks.

    ReplyDelete
  7. You Are Awesome man...God bless you...Tons of thanks

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  8. This works for us, too. Thank you for the great post! We've spent hours making hacks just waiting for the day we could have a solution. Much thanks.

    ReplyDelete
  9. Just thought I'd comment to let you know how much I appreciate this, and that it's a working solution for me!

    ReplyDelete
    Replies
    1. Thank you Alex, I'm glad it worked. I really thank you for your feedback! :)

      Delete
  10. Hi, It really works. But in my case I also need the parent Id in the the child json as child class contains the parent reference.But child json totally ignores the parent class. How can I get the parent ID in the child json?

    ReplyDelete
    Replies
    1. Did you found the solution? I have the same problem.

      Delete
    2. This worked for me.. now I can access the parent object with none infinite recursion.

      @JsonIdentityInfo(
      generator = ObjectIdGenerators.PropertyGenerator.class,
      property = "id")
      public class Product {

      @Id
      @GeneratedValue(strategy = GenerationType.IDENTITY)
      private Long id;

      @OneToMany(fetch = FetchType.LAZY, mappedBy = "parentProduct")
      @JsonManagedReference
      private Set childProducts;

      @ManyToOne(fetch = FetchType.LAZY)
      //@JsonBackReference
      private Product parentProduct;
      }

      Delete
  11. for me it didnt work .I tried both @JsonManagedReference,backreference and @JsonIgnore

    ReplyDelete
  12. could any one help??
    http://stackoverflow.com/q/27688466/4183085

    ReplyDelete
  13. Hi, How to use these annotations @JsonManagedReference and @JsonBackReference ( I am stuck with Jackson v1.9) when third party classes have cyclic dependencies, I have no control on these classes, I use them as instance variables in my bean class which I am trying to serealize.

    ReplyDelete
  14. Hi, first of all thanks for your post! It solved me the problem with infinite recursion!

    But now I'm no longer able to post JSON objects with contains the entities annotated with @JsonManagedReference and @JsonBackReference to my server.. it gives me a 415 (Unsupported Media Type) in browser.. does anyone have a clue?

    ReplyDelete
    Replies
    1. Hi,

      First of all, great info in this blog post! It helped me solving a problem I was facing on reading data!

      However, like Felipe I had the 415 error (on a create operation). To solve that I have changed the @JsonBackReference to @JsonIgnore in one of the sides of the many-to-many relations and got it working. Still need to further validate this, but it seems to solve the problem.

      Cheers!

      Delete
  15. Hi, thanks for the post, this is a great fix. It works perfectly for me when done right.

    ReplyDelete
  16. I put those annotations (@JsonManagedReference and @JsonBackReference) but it seems they are being ignored by Wildfly 8.2.0. I wrote a META-INF/jboss-deployment-structure.xml file inside my EJB .jar, but it didnt help also. When I deployed a reduced version of my application inside Tomcat 8 the annotations worked fine. Three days around this problem and nothing worked. Any clues?

    ReplyDelete
  17. From the post above, let me be more specific: I have an EJB which has the two entities with bi-directional references. The are annotated with @JsonManagedReference and @JsonBackReference. My rest resources are inside a web application and they consume the EJB methods that return the entities. I put a jboss-deployment-structure.xml (that says Wildfly to ignore jackson1 and uses jackson2 provider) in both the EJB jar file and the webapp war file. It seems the annotations are being ignored when the rest resource tries marshalling the entities to a JSON representation. It´s the rest resource in the web application who marshalls the entities, but the JSON annotations that says jackson to ignore infinite recursion are in the entities inside the EJB module! How wildfly should be configured in this case? Thanks for any clu and congratulation for this page! It began to show me the solution.

    ReplyDelete
  18. Man you saved my day. I think this is really stupid, but that is and thank to you I've finally solved this struggling issue!

    ReplyDelete
  19. P.S. I was using a OneToMany and ManyToOne annotations

    ReplyDelete
  20. I have @JsonManagedReference and @JsonBackReference, however when I invoke @PUT method, I am getting Can not handle managed/back reference.

    I have posted my question here.
    http://stackoverflow.com/questions/31825622/java-lang-illegalargumentexception-can-not-handle-managed-back-reference

    ReplyDelete
  21. I can now get a user and the associated battles.
    But i cannot do the vice versa. I can get a battle, BUT
    i cannot get the associated users.

    Any ideas?

    ReplyDelete
  22. Not able to access parent from child object. Any idea ?

    ReplyDelete
    Replies
    1. have u got the solution?? coz i m also stuck in the same problem. thanku in advance. :)

      Delete
  23. I solved my problem with infinite loop to delete in higher entity obsolete definitions pointer (@OneToMany....) to subordinate tables. Those columns are virtual and has no real binding in DB table. Subordinate records I can fetch in next step using primary key and subornate table fetch (according foreign key).

    ReplyDelete
  24. This comment has been removed by the author.

    ReplyDelete
  25. still my problem with post Method..

    ReplyDelete
  26. i've read through this a bunch of times now, and i think i must be missing something. i posted my slight variation of this problem here - http://stackoverflow.com/questions/35346794/infinite-recursion-with-jackson-json-and-hibernate-jpa-issue-yet-another.

    if anyone can help suggest what i'm *still* doing wrong, i would be enormously grateful.

    ReplyDelete
  27. Your solution works beautiful to solve the infinite loop issue. But now im having an issue and is that the child has no access to the parent whatsoever.

    So in my scenario, I have a User who writes Posts... So when querying for a User, everything works like a charm. It brings the User + the bunch of Posts he has written (with no User reference loop on each post). But then, if I query for a specific Post, the User is removed, so I have no idea whom wrote that.

    I could start writing my own DTOs and convert stuff, but it will be annoying as post has other relations.

    Thanks...

    ReplyDelete
    Replies
    1. did you solved this problem? I have a similar one.

      Delete
  28. great job, helpful resource God bless you

    ReplyDelete
  29. The solution you have provided work like a charm. Thank you

    ReplyDelete
  30. It helped me as well. Thanks

    ReplyDelete
  31. Nice it help me a lot! Thanks

    ReplyDelete
  32. @JsonManagedReference and @JsonBackReference solves the problem in one direction but not in the other direction. Fof bidirectional cause use @JsonIdentityInfo
    Here is some info: http://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion

    ReplyDelete
  33. I have a similar strutcture but it fails with the infinite error when I try to get the data
    Parent class
    @OneToMany(fetch = FetchType.LAZY, cascade=CascadeType.ALL, mappedBy="project")
    public List getUnitDetails() {
    return unitDetails;
    }

    Child Class
    @ManyToOne(fetch = FetchType.EAGER, cascade=CascadeType.ALL)
    @JoinColumn(name="id", insertable=true, updatable=false)
    public Project getProject() {
    return project;
    }

    I have also annotated both the classes as below
    @JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "id")

    Anyone help

    ReplyDelete
  34. Thank you @JsonBackReference and @JsonManagedReference worked for me

    ReplyDelete
  35. I understand the problem and I understand the solutions referenced here, but for my project I don't want to have to monkey with the auto generated classes which represent the database (i.e., add these annotations). Is there a solution where I would not have to modified auto generated java classes?

    ReplyDelete
  36. It was really helpful and a way better than @JsonIgnore. Thank you

    ReplyDelete
  37. Thank you so much for this! Solved my Problem in 5 Seconds :)

    ReplyDelete
  38. Just wanted to let you know, that you solution was very helpful for me. One question left for me: In your example I saw that you were mixing jpa and jackson annotations in one "model" object. This is especially dangerous if you have two-way relations. Shouldn't both be seperated to a domain AND DTO object to particulary avoid the issue you described here?

    ReplyDelete
  39. THANKS! spent the whole day trying to figure this one out

    ReplyDelete
  40. Thanks ! Works very well !
    But if you have to post your entity, you will endup with a 415 error, in this case, you will need to remove the @JsonManagedReference and @JsonBackReference, and use a @JsonIgnore in one of your entities.

    ReplyDelete
  41. Thank you very much for your post! One small adition: In the simple case that you only have a unidirectional reference you must use "@JsonBackReference" (and not @JsonManagedReferences").

    Example:

    @Entity
    class NodeModel {
    @id
    long id

    @ManyToOne
    @JsonBackRefereence
    NodeModel parentNode;

    ...
    }

    ReplyDelete
  42. very nice , my prob;le also solved with the use of @JsonBackRresponse on parent table cloumn mapping and @jsonManageResponse at child ,,,
    thanks a alot

    ReplyDelete
  43. Don't forget to implements java.io.Serializable in the class

    ReplyDelete