Add support for pagination

[andye2004]

Just wondering if there is any appetite for adding pagination support? E.g. configure things to use PagedResources instead of Resources and also to add some additional methods to follow first, next, prev and last links etc.

From my own perspective it would be nice to be able to have a getPage() method in the client where we could specify page size and offset etc.

As before I'm happy to do some work on this and issue a PR.

[hdpe]

Hi, thanks for raising this important issue. Basically my thinking is: yes, we definitely want to support Spring Data REST pagination; but no, we don't want to use PagedResources or add methods to the Client that are very specific to Spring Data REST. Spring HATEOAS's approach to pagination is very opinionated - the HAL spec definitely doesn't require page metadata to be encapsulated in a object keyed with page or anything like that, and we only want to add things to the core library if they have broad applicability.

Instead I think we want to use the existing Bowman constructs like @LinkedResource to support this. Ideally, something like:

public abstract class ResultPage {

  private List<Entity> content;

  private org.springframework.hateoas.PagedResources.PageMetadata metadata;

  @EmbeddedResources
  public List<Entity> getContent() {
    return content;
  }

  public void setContent(List<Entity> content) {
    this.content = content;
  }

  @JsonProperty("page")
  public org.springframework.hateoas.PagedResources.PageMetadata getMetadata() {
    return metadata;
  }

  @LinkedResource
  public abstract ResultPage next();

  @LinkedResource
  public abstract ResultPage previous();
}

and then you could do:

ResultPage page1 = client.get(URI.create("http://.../entities?page=0&size=10",
    ResultPage.class));

ResultPage page2 = page1.next();
// etc

None of this is possible at the moment for all manner of reasons, but I'm currently putting together a proof of concept.

Does this seem like it would get you at least part way to what you want?

There may be some scope for adding a Spring Data REST-specific extension module to provide utilities to make some of this easier. I just don't want the core library getting bloated with SDR-specific stuff.

[andye2004]

Hi Ryan, what you're proposing isn't too dissimilar to what I have currently and I totally understand you're point re SDR / Spring HATEOAS independence. I actually had the same conversation with a colleague over the PagedResources vs Page question and said I expected you to be reluctant to use the PagedResources class 😄

If you have a look in the forked branch you'll see a class called Page and an extra couple of methods added to the Client class. There are also integration tests to cover the new functionality. Relevant code below to save you going looking but all I'm currently missing is the next() and previous() methods.

In addition I added some convenience methods to the Page class instead of returning the PageMetaData.

This code is currently working for us but I'm wondering what some of the other issues are that you allude to, maybe it's something I can look into?

public class Page<T> {

	private final List<T> resources;
	private final PageMetadata pageMetadata;

	public Page(List<T> resources, PageMetadata pageMetadata) {
		this.resources = resources;
		this.pageMetadata = pageMetadata;
	}

	public List<T> getResources() {
		return resources;
	}

	public long getElementsPerPage() {
		return pageMetadata.getSize();
	}

	public long getTotalElements() {
		return pageMetadata.getTotalElements();
	}

	public long getTotalPages() {
		return pageMetadata.getTotalPages();
	}

	public long getCurrentPage() {
		return pageMetadata.getNumber();
	}
}

public class Client<T> {
...
	public long getCount() {
		return getPage(0, 1).getTotalElements();
	}

	public Page<T> getPage(int page, int size) {
		URI uri = UriComponentsBuilder
				.fromUri(baseUri)
				.queryParam("page", page)
				.queryParam("size", size)
				.build().toUri();

		PagedResources<Resource<T>> resources = restOperations.getPagedResources(uri, entityType);

		List<T> result = new ArrayList<>();
		for (Resource<T> resource : resources) {
			result.add(proxyFactory.create(resource, restOperations));
		}

		return new Page<>(result, resources.getMetadata());
	}
...
}

[hdpe]

What you've got looks good but unfortunately it's far too tightly coupled to SDR endpoints ("page", "size" params etc.) for our needs.

To get to something more like what I've suggested above I think there's a few steps we need to get through.

1. Supporting deserialisation of abstract classes.

I think this is important because in the example above we do want content to be a JavaBean-style property (so it can be set by the deserializer but then read by the user), but then next I don't think we want to require users to provide an implementation for that is never used. I've had a stab at this and I think this is pretty much done and working in d58c712.

2. Supporting using the proxy-creating deserializer (currently called InlineAssociationDeserializer) to handle _embedded values.

This just doesn't work when using Spring HATEOAS's HalResourcesDeserializer at the moment. I've raised an issue for it with Spring HATEOAS. Let's see what they say.

If this worked, then you could do something like:

@RemoteResource("/entities")
public abstract class ResultPage {
	
	private List<Entity> content;
	
	@JsonProperty("_embedded")
	@JsonDeserialize(using = HalResourcesDeserializer.class, contentUsing = InlineAssociationDeserializer.class,
		contentAs = Entity.class)
	public List<Entity> getContent() {
		return content;
	}
	
	public void setContent(List<PageableEntity> content) {
		this.content = content;
	}
	
	@LinkedResource
	public abstract ResultPage next();
}

and navigate the result set using a Client<ResultPage>. Not ideal, but a good first step: it should work to retrieve pages and navigate between them by following their links.

3. InlineAssocationDeserializer would then need a better name (ProxyDeserializer?) and the old name deprecating.

4. Do something about the Client<ResultPage> madness.

Currently clients correspond to a specific entity class which has to be annotated with @RemoteResource. We don't really want to create clients just for result pages; the client for the entity should suffice.

Need to have a think about how best to handle this. Adding a method Client.get(URI uri, Class<?> clazz) and relaxing the requirement mandating a @RemoteResource annotation would work, but that doesn't feel to me that it sits nicely with the functionality and type-specificity of the Client as it stands. I wonder if we're now actually talking about different types of clients: SimpleClient which is type-less, and CrudClient which is typed - or some such. More thought required.

5. It would be nice to not have to use

@JsonProperty("_embedded")
@JsonDeserialize(using = HalResourcesDeserializer.class, contentUsing = InlineAssociationDeserializer.class,
	contentAs = Entity.class)
public List<Entity> getContent() { ... }

on embedded resources, or

@JsonDeserialize(contentAs = Entity.class)
public List<Entity> getContent() { ... }

on inline associations, and instead just be able to use our own annotations @EmbeddedResources, @InlineResource etc.

I think we could do this by implementing a Jackson AnnotationIntrospector.

6. Having done 5, we could achieve an unrelated long term goal of mine: put the annotations in a separate artifact so users only need a dependency on the "annotation" artifact from their client model. Then you could limit the dependency on the "client" artifact to where the client was actually invoked.

7. If the link-navigating pagination described above is too onerous for users, we should consider adding a "bowman-client-sdr" artifact or similar for Spring Data REST specific utilities in helping with this. This may be its own client subclass or just a set of utilities for helping to use the standard one. Again, I'm not sure at all what this would look like yet, but once we've done all the steps above this might become more clear.

I think I'll see what the response is on the Spring HATOEAS issue before embarking on any of this. It may change how we tackle some of the other bits.

I would be interested what you thought about all of this though and in particular point 4. Shout out to previous contributors @datenbrille & @aelstad - if you have any thoughts on any of this, please add them here!

Thanks

[andye2004]

Sorry @hdpe, been really busy and not had time to get back to this. I will attempt to digest your comments over the next couple of days and get back to you.

[xfrancois]

Hello,
Is there any update on this issue ? I just checked the issue on Spring Hateoas side but it doesn't seem resolved yet.

[andye2004]

Hi @xfrancois, sorry but no I haven't made any progress on this. I'm no longer working on the team in my organisation that was looking after the project using this library and so as a result my attention has been elsewhere.

I'd love to say I can pick this back up again but at the moment that's unlikely given my current workload.