The website has moved

Note: the website moved to www.keenformatics.com. You're being redirected.

Monday, August 12, 2013

How To Solve JSON infinite recursion Stackoverflow
(with Spring and Jackson annotations)

Infinite recursion picture

Coding in Spring 3 with JPA and Jackson is supposed to semplify your life, sparing you from writing thousand lines of code. Sometimes, though, you'll notice that all these annotations hide the real core of your application, preventing you to really understand what your code is doing (and what it is supposed to do).
This means that if you want to develop a good java application or website, you need to carefully read the docs or (more realistically) you need to understand how to read your error logs and combine them with a well-written search engine query.

This is what I had to do when I first met the problem I'm about to write about.



The problem.

I was getting a wrong JSON response while using a many-to-many relationship, that caused my application to enter in an infinite loop giving as output an infinite recursion Stackoverflow error.

Here is part of what I found in my Eclipse console:

Request processing failed; nested exception is org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: Infinite recursion (StackOverflowError) (through reference chain: [here is the loop]) with root cause
java.lang.StackOverflowError
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:567)
 at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:143)
 at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:118)
 at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:24)
 at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:180)
 at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:544)
 at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:551)
 ...

and so on.

Since I was using Datatables jQuery plugin, I also had this output in my browser window:

DataTables warning (table id = 'products'): DataTables warning: JSON data from server could not be parsed. This is caused by a JSON formatting error.

What I noticed is that the same database query, without JSON, was working well. So the problem (as the console in fact said) was with Jackson and Json serialization. As the docs state (see references), bi-directional references for ORM-managed beans (iBatis, Hibernate) would cause serialization to failure since they are cyclic dependencies. Since Jackson 1.6, this problem has been solved by the introduction of two new annotations: @JsonManagedReference and @JsonBackReference (and see the end of this post to give a look at the @JsonIdentityInfo annotation).

How does serialization work? (The solution in theory)
In computer science, in the context of data storage and transmission, serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and resurrected later in the same or another computer environment.

For Jackson to work well, one of the two sides of the relationship should not be serialized, in order to avoid the annoying infinite recursive loop that causes our stackoverflow error.

So, Jackson takes the forward part of the reference, for example an attribute of a java class (i.e. List<Role> roles in User class), and converts it in a json-like storage format; this is the so-called marshalling process.
Then, Jackson looks for the back part of the reference (i.e. List<User> users in Role class) and leaves it as it is, not serializing it. This part of the relationship will be re-constructed during the deserialization (unmarshalling) of the forward reference.

The solution in practice.

It's very simple. Assuming that your database query already works without JSON, all you have to do is this:

  1. Add the @JsonManagedReference In the forward part of the relationship (i.e. User.java class):
    @Entity
    public class User implements java.io.Serializable{
     
     @Id
     @GeneratedValue(strategy=GenerationType.IDENTITY)
     private long id;
     
     @Column(name="name")
     private String name;
    
     @ManyToMany
     @JoinTable(name="users_roles",joinColumns=@JoinColumn(name = "user_fk"),
     inverseJoinColumns=@JoinColumn(name = "role_fk"))
     @JsonManagedReference
     private Set<Role> roles = new HashSet<Role>();
    
    ...
    
  2. Add the @JsonBackReference In the back part of the relationship (i.e. Role.java class):
    @Entity
    public class Role implements java.io.Serializable {
    
     @Id 
     @GeneratedValue(strategy=GenerationType.IDENTITY)
     private int id;
    
     @ManyToMany(mappedBy="roles")
     @JsonBackReference
     private Set<User> users = new HashSet<User>();
    
    ...
    

The work is done. If you take a look at your firebug logs, you'll notice that the infinite recursive loop has disappeared.


Edit (02/09/2013)

Another useful annotation you could check is @JsonIdentityInfo: using it, everytime Jackson serializes your object, it will add an ID (or another attribute of your choose) to it, so that it won't entirely "scan" it again everytime. This can be useful when you've got a chain loop between more interrelated objects (for example: Order -> OrderLine -> User -> Order and over again).

In this case you've got to be careful, since you could need to read your object's attributes more than once (for example in a products list with more products that share the same seller), and this annotation prevents you to do so. I suggest to always take a look at firebug logs to check the Json response and see what's going on in your code.


References

87 comments:

  1. Hi,
    The solution is great and simple. But it does not work for me in case when an object has a self-reference(in a tree-like structure). Have you ever encountered the same problem and how have you solved it?

    Thanks,
    Kostya

    ReplyDelete
    Replies
    1. Hi Kostya.
      If by "self-reference" you mean a hyerarchical structure with parent-child relationships between objects of the same class, these annotations should work as well. For example: if you've got a Category class and you want to nest categories (e.g.: "smartphone" is a child of "technology") you could add a @JsonBackReference to the parent property and a @JsonManagedReference to the child property:

      @OneToOne
      @JsonBackReference
      private Category parent;

      @OneToMany
      @JsonManagedReference
      private List children;

      Another useful annotation you could check is @JsonIdentityInfo: using it, everytime jackson serializes your object, it will add an ID (or another attribute of your choose) to it, so that it won't entirely "scan" it again everytime. This can be useful when you've got a chain loop between more interrelated objects (for example: Order -> OrderLine -> User -> Order and over again ).
      In this case you've got to be careful, since you could need to read your object's attributes more than once (for example in a products list with more products that share the same seller), and this annotation prevents you to do so.

      I hope I've answered your question :)

      Delete
    2. I've just found the root of my problem. It was our custom model generator that messed things up. As for Jackson, it works as expected.

      Thanks for your help!

      Delete
    3. This comment has been removed by the author.

      Delete
    4. Thank you, it was a really nice solution.

      Delete
  2. Your post is well explained and really helped me. The @JsonManagedReference and @JsonBackReference worked for me. Thanks

    ReplyDelete
  3. I had the same problem with user and role and followed your solution. It solved the problem but I still have one question left. If I read the user object I get the user including the roles. But if I want to read the role object how can I ensure that I also could get all users of a role. What I mean is, using @JsonManagedReference and @JsonBackReference solves the problem in one direction but I think not in the other direction, right?

    ReplyDelete
    Replies
    1. I think it should solve the problem in both directions.
      If you've got a Role and you want to retrieve its Users you can simply do that because of the bidirectional relationship between Role and User objects.
      Using json, those annotations ensure that when you'll get the Users they won't try to give you back the Roles again (so that you won't enter a loop).

      Delete
    2. But If I have the Users only, how can I access the role?

      Delete
    3. @Bernfried Howe

      Did you solve this ? I have the same problem

      Delete
    4. @PC

      Did you solve this ? I have the same problem

      Delete
  4. I just want to say THANK YOU !! I became crazy about an infinite recursion in my Json, I came here by Googled to solve the problem and your post is very helpful, @JsonManagedReference and @JsonBackReference worked fine for me ! :)

    ReplyDelete
    Replies
    1. You're very welcome, I'm glad I've been useful to you :)

      Delete
  5. Thank you! Perfect in my case, too. Thank you, man.

    ReplyDelete
    Replies
    1. You're welcome! Thank you for your kind comment :)

      Delete
  6. Fantastic !
    It fixed my problem ... I have been struggling several days to fix this one.
    Thanks.

    ReplyDelete
  7. You Are Awesome man...God bless you...Tons of thanks

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  8. This works for us, too. Thank you for the great post! We've spent hours making hacks just waiting for the day we could have a solution. Much thanks.

    ReplyDelete
  9. Just thought I'd comment to let you know how much I appreciate this, and that it's a working solution for me!

    ReplyDelete
    Replies
    1. Thank you Alex, I'm glad it worked. I really thank you for your feedback! :)

      Delete
  10. Hi, It really works. But in my case I also need the parent Id in the the child json as child class contains the parent reference.But child json totally ignores the parent class. How can I get the parent ID in the child json?

    ReplyDelete
    Replies
    1. Did you found the solution? I have the same problem.

      Delete
    2. This worked for me.. now I can access the parent object with none infinite recursion.

      @JsonIdentityInfo(
      generator = ObjectIdGenerators.PropertyGenerator.class,
      property = "id")
      public class Product {

      @Id
      @GeneratedValue(strategy = GenerationType.IDENTITY)
      private Long id;

      @OneToMany(fetch = FetchType.LAZY, mappedBy = "parentProduct")
      @JsonManagedReference
      private Set childProducts;

      @ManyToOne(fetch = FetchType.LAZY)
      //@JsonBackReference
      private Product parentProduct;
      }

      Delete
  11. for me it didnt work .I tried both @JsonManagedReference,backreference and @JsonIgnore

    ReplyDelete
  12. could any one help??
    http://stackoverflow.com/q/27688466/4183085

    ReplyDelete
  13. you are awesome thanks maan

    ReplyDelete
  14. Hi, How to use these annotations @JsonManagedReference and @JsonBackReference ( I am stuck with Jackson v1.9) when third party classes have cyclic dependencies, I have no control on these classes, I use them as instance variables in my bean class which I am trying to serealize.

    ReplyDelete
  15. Hi, first of all thanks for your post! It solved me the problem with infinite recursion!

    But now I'm no longer able to post JSON objects with contains the entities annotated with @JsonManagedReference and @JsonBackReference to my server.. it gives me a 415 (Unsupported Media Type) in browser.. does anyone have a clue?

    ReplyDelete
    Replies
    1. Hi,

      First of all, great info in this blog post! It helped me solving a problem I was facing on reading data!

      However, like Felipe I had the 415 error (on a create operation). To solve that I have changed the @JsonBackReference to @JsonIgnore in one of the sides of the many-to-many relations and got it working. Still need to further validate this, but it seems to solve the problem.

      Cheers!

      Delete
  16. Hi, thanks for the post, this is a great fix. It works perfectly for me when done right.

    ReplyDelete
  17. I put those annotations (@JsonManagedReference and @JsonBackReference) but it seems they are being ignored by Wildfly 8.2.0. I wrote a META-INF/jboss-deployment-structure.xml file inside my EJB .jar, but it didnt help also. When I deployed a reduced version of my application inside Tomcat 8 the annotations worked fine. Three days around this problem and nothing worked. Any clues?

    ReplyDelete
  18. From the post above, let me be more specific: I have an EJB which has the two entities with bi-directional references. The are annotated with @JsonManagedReference and @JsonBackReference. My rest resources are inside a web application and they consume the EJB methods that return the entities. I put a jboss-deployment-structure.xml (that says Wildfly to ignore jackson1 and uses jackson2 provider) in both the EJB jar file and the webapp war file. It seems the annotations are being ignored when the rest resource tries marshalling the entities to a JSON representation. It´s the rest resource in the web application who marshalls the entities, but the JSON annotations that says jackson to ignore infinite recursion are in the entities inside the EJB module! How wildfly should be configured in this case? Thanks for any clu and congratulation for this page! It began to show me the solution.

    ReplyDelete
  19. Man you saved my day. I think this is really stupid, but that is and thank to you I've finally solved this struggling issue!

    ReplyDelete
  20. P.S. I was using a OneToMany and ManyToOne annotations

    ReplyDelete
  21. I have @JsonManagedReference and @JsonBackReference, however when I invoke @PUT method, I am getting Can not handle managed/back reference.

    I have posted my question here.
    http://stackoverflow.com/questions/31825622/java-lang-illegalargumentexception-can-not-handle-managed-back-reference

    ReplyDelete
  22. I can now get a user and the associated battles.
    But i cannot do the vice versa. I can get a battle, BUT
    i cannot get the associated users.

    Any ideas?

    ReplyDelete
  23. Not able to access parent from child object. Any idea ?

    ReplyDelete
    Replies
    1. have u got the solution?? coz i m also stuck in the same problem. thanku in advance. :)

      Delete
  24. I solved my problem with infinite loop to delete in higher entity obsolete definitions pointer (@OneToMany....) to subordinate tables. Those columns are virtual and has no real binding in DB table. Subordinate records I can fetch in next step using primary key and subornate table fetch (according foreign key).

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. still my problem with post Method..

    ReplyDelete
  27. i've read through this a bunch of times now, and i think i must be missing something. i posted my slight variation of this problem here - http://stackoverflow.com/questions/35346794/infinite-recursion-with-jackson-json-and-hibernate-jpa-issue-yet-another.

    if anyone can help suggest what i'm *still* doing wrong, i would be enormously grateful.

    ReplyDelete
  28. Your solution works beautiful to solve the infinite loop issue. But now im having an issue and is that the child has no access to the parent whatsoever.

    So in my scenario, I have a User who writes Posts... So when querying for a User, everything works like a charm. It brings the User + the bunch of Posts he has written (with no User reference loop on each post). But then, if I query for a specific Post, the User is removed, so I have no idea whom wrote that.

    I could start writing my own DTOs and convert stuff, but it will be annoying as post has other relations.

    Thanks...

    ReplyDelete
    Replies
    1. did you solved this problem? I have a similar one.

      Delete
    2. HI I have the same problem, have you solved this?

      Delete
  29. great job, helpful resource God bless you

    ReplyDelete
  30. The solution you have provided work like a charm. Thank you

    ReplyDelete
  31. It helped me as well. Thanks

    ReplyDelete
  32. Nice it help me a lot! Thanks

    ReplyDelete
  33. @JsonManagedReference and @JsonBackReference solves the problem in one direction but not in the other direction. Fof bidirectional cause use @JsonIdentityInfo
    Here is some info: http://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion

    ReplyDelete
  34. I have a similar strutcture but it fails with the infinite error when I try to get the data
    Parent class
    @OneToMany(fetch = FetchType.LAZY, cascade=CascadeType.ALL, mappedBy="project")
    public List getUnitDetails() {
    return unitDetails;
    }

    Child Class
    @ManyToOne(fetch = FetchType.EAGER, cascade=CascadeType.ALL)
    @JoinColumn(name="id", insertable=true, updatable=false)
    public Project getProject() {
    return project;
    }

    I have also annotated both the classes as below
    @JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "id")

    Anyone help

    ReplyDelete
  35. Thank you @JsonBackReference and @JsonManagedReference worked for me

    ReplyDelete
  36. I understand the problem and I understand the solutions referenced here, but for my project I don't want to have to monkey with the auto generated classes which represent the database (i.e., add these annotations). Is there a solution where I would not have to modified auto generated java classes?

    ReplyDelete
  37. It was really helpful and a way better than @JsonIgnore. Thank you

    ReplyDelete
  38. Thank you so much for this! Solved my Problem in 5 Seconds :)

    ReplyDelete
  39. Just wanted to let you know, that you solution was very helpful for me. One question left for me: In your example I saw that you were mixing jpa and jackson annotations in one "model" object. This is especially dangerous if you have two-way relations. Shouldn't both be seperated to a domain AND DTO object to particulary avoid the issue you described here?

    ReplyDelete
  40. Great !! Its help me, thank you

    ReplyDelete
  41. THANKS! spent the whole day trying to figure this one out

    ReplyDelete
  42. Thanks ! Works very well !
    But if you have to post your entity, you will endup with a 415 error, in this case, you will need to remove the @JsonManagedReference and @JsonBackReference, and use a @JsonIgnore in one of your entities.

    ReplyDelete
  43. Thank you very much for your post! One small adition: In the simple case that you only have a unidirectional reference you must use "@JsonBackReference" (and not @JsonManagedReferences").

    Example:

    @Entity
    class NodeModel {
    @id
    long id

    @ManyToOne
    @JsonBackRefereence
    NodeModel parentNode;

    ...
    }

    ReplyDelete
  44. very nice , my prob;le also solved with the use of @JsonBackRresponse on parent table cloumn mapping and @jsonManageResponse at child ,,,
    thanks a alot

    ReplyDelete
  45. Don't forget to implements java.io.Serializable in the class

    ReplyDelete
  46. It was much helpful and concise. The details are also not missed and the reason for which we have to use this strategy of implementation. Thanks for sharing such information.

    ReplyDelete
  47. Ohh my goodness..!! This solution just eased my problem. Thats a real good catch on how to deal with Jackson/JSON. Thanks alot for helping my problem to resolve...

    ReplyDelete
  48. I am facing this kind of problem now, but I am in a condition that prevent me to modify the entity. Do you have a solution for this kind of problem ?
    Thank you.

    ReplyDelete
  49. Hi,
    I have a Play! framework application in which I have used Ebean ORM (which is similar to Hibernate JPA). I have one modal class 'User' and it has a self table foreign key i.e. user table is having a foreign key called 'user_manager_id' and this id is primary key of user table itself. The modal class looks like this:


    class User{


    Other fields:
    .
    .
    .
    .
    .
    @OneToOne
    private User user_manager;
    .
    .
    .
    }

    Now when performing 'update' operation on some other table, in which 'user_id' is a foreign key, the Ebean update operation throws an error which is as under:

    java.lang.RuntimeException: java.lang.IllegalArgumentException: Infinite recursion (StackOverflowError) (through reference chain: org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]-......


    Inside the exception trace, I found for this Jackson exception as well:

    com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"].......


    Can anybody help me please in identifying what thing is causing this exception to occur and also what would be the best solution for resolving this. Please help. Thanks!

    ReplyDelete
  50. Hi,
    I have a Play! framework application in which I have used Ebean ORM (which is similar to Hibernate JPA). I have one modal class 'User' and it has a self table foreign key i.e. user table is having a foreign key called 'user_manager_id' and this id is primary key of user table itself. The modal class looks like this:


    class User{


    Other fields:
    .
    .
    .
    .
    .
    @OneToOne
    private User user_manager;
    .
    .
    .
    }

    Now when performing 'update' operation on some other table, in which 'user_id' is a foreign key, the Ebean update operation throws an error which is as under:

    java.lang.RuntimeException: java.lang.IllegalArgumentException: Infinite recursion (StackOverflowError) (through reference chain: org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]-......


    Inside the exception trace, I found for this Jackson exception as well:

    com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"]->org.org_name.user.models.User["user_manager"].......


    Can anybody help me please in identifying what thing is causing this exception to occur and also what would be the best solution for resolving this. Please help. Thanks!

    ReplyDelete
  51. Im facing issues with call the mockito/mvc on the test side, with jasonBackReference and @JsonManagedReferenceit is not working the test, any idea?

    ReplyDelete
  52. The documentation for the @JsonBackReference annotation (https://fasterxml.github.io/jackson-annotations/javadoc/2.2.0/com/fasterxml/jackson/annotation/JsonBackReference.html) says "Value type of the property must be a bean: it can not be a Collection, Map, Array or enumeration.". I tried to use @JsonBackReference in a many-to-many relationship in conjunction with @JsonManagedReference as in the example, but I noticed some weird behaviour in the relationship, i.e. when adding further ForwardEntity-BackwardEntity relationships one of the pre-existing relationships was removed. I solved the problem by replacing @JsonBackReference with @JsonProperty(access=JsonProperty.Access.WRITE_ONLY) and dropping @JsonManagedReference in the other Entity. Any idea why this could have happened and the article's example works?

    ReplyDelete
  53. This comment has been removed by a blog administrator.

    ReplyDelete
  54. Thanks, My issue has been resolved.

    ReplyDelete
  55. THAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANKS MAN!!!!!!!!!!!!

    ReplyDelete
  56. This comment has been removed by a blog administrator.

    ReplyDelete
  57. how to solve this issue with third party classes. I can not modify them

    ReplyDelete
  58. Thank Man! Worked like charm..!!:)

    ReplyDelete
  59. I had this error on creating Json from DTOs, i was confused that how can i use @JsonManagedReference and @JsonBackReference on my DTOs that does not have any annotations about relationships, but i could solve my problem by using @JsonIdentityInfo on UserDto that had ManyToMany relationship.
    thanks for your complete answer on stackOverFlow and more complete answer here :) this was very helpful.

    ReplyDelete
  60. Hey! Sorry to resurrect the post. But Man, you saved me with this post. So I only wanted to thank you for your time and energy, as i can see you helped a lot of people.
    Regards

    ReplyDelete
    Replies
    1. Hi Erik, thank you for your kind message! I'm glad I could help :)

      Delete