Lazy loading of tables in JSF – working with large amount of data

The challenge working with a large amount of data in user-presentation is to serve the user an easy and effective way of getting to the information he/she is looking for. If you have a table with 50.000 rows, you would in some way need to chop the information in smaller pieces, for example having multiple pages displaying 50 rows each, and then having a way of navigating through the pages.

But still, there are some considerations to do this in a way that opts for performance. If the user selects do display page 2, meaning rows 50-100, wouldn’t you like to fetch only those, not all the 50.000 rows? And like in JSF, if you have multiple components (dataTables, dataScrollers) that are dependent on/using the same data, you would like that the data is only retrieved once, not one time for each component. Doing web applications, being able to limit the amount of data that is fetched from the backend is a key to success when it comes to performance and operability issues.

My example is based on Cagatay Civici’s post http://cagataycivici.wordpress.com/2006/07/10/jsf_datatable_with_custom_paging/,
so I will not go through the technical solution and how the “lazy” data model works once more, I suggest you read his post for that. But I give a complete example of how to use his PagedListDataModel, and in addition how to make sure the data is retrieved only once by using phaseListeners.

To keep it simple, let’s imagine that we have a Dictionary in the backend, and a jsp page with static links with letters like ‘A’, ‘B’, ‘C’ and so on throughout the alphabet. Pressing on one of the letters would display all the words that starts with it in the dataTable, and we can use a dataScroller to navigate through multiple pages.

This is my jsp, in which I’ve removed the columns of the dataTable as they are of no importance. Note that the dataTable is set with a valueBinding, and that the value of it is pointing to a dataModel. Also note that we do not render the table if no one has selected a letter. I also just give one link as an example of how to update the property we can refer to as the “selected letter”:

<h:commandLink styleClass="myLink" value="A">
	<t:updateActionListener value="A" property="#{myDataModelHandler.selectedLetter}"></t:updateActionListener>
</h:commandLink>
<h:panelGrid id="mainPanel" width="100%">
	<t:dataTable
	  id="htmlTableView"
	  value="#{myDataModelHandler.activeDataModel}" rendered="#{myDataModelHandler.selectedLetter != null}"
	  binding="#{myHtmlDataTableHandler.htmlDataTable}"
	  headerClass="tableViewHeader"
	  var="row">
	</t:dataTable>
	<t:panelGrid id="footerPager">
		 <t:dataScroller id="ds5"
				for="htmlTableView"
				fastStep="5"
				styleClass="scroller"
				paginator="true"
				paginatorMaxPages="5"
				paginatorTableClass="paginator"
				paginatorActiveColumnStyle="font-weight:bold;"
				rowsCountVar="rowsCount"
				renderFacetsIfSinglePage="true"
			>
			<f:facet name="previous">
				<t:graphicImage id="datascrollerPrevious" value="/images/left-one-step.gif" border="1" styleClass="dataScrollerImage"/>
			</f:facet>
			<f:facet name="next">
				 <t:graphicImage id="datascrollerNext" value="/images/right-one-step.gif" border="1" styleClass="dataScrollerImage"/>
			</f:facet>
			<f:facet name="first">
				<t:graphicImage id="datascrollerFirst"
				value="/images/left.gif" border="1" styleClass="dataScrollerImage"/>
			</f:facet>
			<f:facet name="last">
				 <t:graphicImage id="datascrollerLast"
				 value="/images/right.gif" border="1" styleClass="dataScrollerImage" />
			</f:facet>
		 </t:dataScroller>
	 </t:panelGrid>
</h:panelGrid>

This is the handler (backing bean) that holds the htmlDataTable:

public class MyHtmlDataTableHandler
{
    private HtmlDataTable htmlDataTable;
    public MyHtmlDataTableHandler()
    {

    }

    public HtmlDataTable getHtmlDataTable()
    {
        if(htmlDataTable == null)
        {
            log.debug("htmlDataTable == null");
            HtmlDataTable dataTable = new HtmlDataTable();
            dataTable.setRows(50);
            setHtmlDataTable(dataTable);
        }
        return htmlDataTable;
    }

    public void setHtmlDataTable(HtmlDataTable htmlDataTable)
    {
            this.htmlDataTable = htmlDataTable;
    }
}

You can see that I have modified the getter to create a new instance if the instance it holds is null. It’s important to set the number of rows here, or it will fail when trying to build the lazy data model.The dataModel we use for our dataTable is created and set before the RENDER_RESPONSE phase, as mentioned earlier this is to avoid that the data itself is retrieved multiple times. To be able to do this we need to use a phaseListener, so we implement it in our constructor of the dataModelHandler. Note that we have references here to the dataTableHandler as well as a util for helping us creating a paged data model. As you can see in the phaseListener beforePhase()-method, if no letter has been selected, we don’t create or set the model.

public class MyDataModelHandler
{
	private String selectedLetter;
	private PagedListDataModel activeDataModel;

	private MyHtmlDataTableHandler myHtmlDataTableHandler; //BEAN_REFERENCE SET IN FACES-CONFIG
	private PagedDataModelUtil pagedDataModelUtil; //BEAN_REFERENCE SET IN FACES-CONFIG

	public MyDataModelHandler()
	{
		LifecycleFactory lifecycleFactory = (LifecycleFactory) FactoryFinder.getFactory(FactoryFinder.LIFECYCLE_FACTORY);
		Lifecycle lifecycle = lifecycleFactory.getLifecycle(lifecycleFactory.DEFAULT_LIFECYCLE);

		lifecycle.addPhaseListener
		(
			new PhaseListener()
			{
				public void afterPhase(PhaseEvent arg0)
				{
				}
				public void beforePhase(PhaseEvent arg0)
				{
					if(selectedLetter != null)
					{
						setActiveDataModel(pagedDataModelUtil.createPagedDataModel(selectedLetter, getMyHtmlDataTableHandler().getHtmlDataTable()));
					}
				}	

				public PhaseId getPhaseId()
				{
					return PhaseId.RENDER_RESPONSE;
				}
			}
		);
	}

	public String getSelectedLetter()
	{
		return selectedLetter;
	}

	public void setSelectedLetter(String selectedLetter)
	{
		this.selectedLetter = selectedLetter;
	}

	public PagedListDataModel getActiveDataModel()
	{
		return activeDataModel;
	}

	public void setActiveDataModel(PagedListDataModel activeDataModel)
	{
		this.activeDataModel = activeDataModel;
	}

	public MyHtmlDataTableHandler getMyHtmlDataTableHandler()
	{
		return myHtmlDataTableHandler;
	}

	public void setMyHtmlDataTableHandler(MyHtmlDataTableHandler myHtmlDataTableHandler)
	{
		this.myHtmlDataTableHandler = myHtmlDataTableHandler;
	}

	public PagedDataModelUtil getPagedDataModelUtil()
	{
		return pagedDataModelUtil;
	}

	public void setPagedDataModelUtil(PagedDataModelUtil pagedDataModelUtil)
	{
		this.pagedDataModelUtil = pagedDataModelUtil;
	}
}

The pagedDataModel we create is an instance of PagedListDataModel which is exactly the same as in Cagatay Civici’s post:

public class PagedListDataModel extends DataModel
{
  private int rowIndex = -1;
  private int totalNumRows;
  private int pageSize;
  private List list;

  public PagedListDataModel(List list, int totalNumRows, int pageSize)
  {
    setWrappedData(list);
    this.totalNumRows = totalNumRows;
    this.pageSize = pageSize;
  }

  public boolean isRowAvailable()
  {
    if(list == null)
      return false;
    int rowIndex = getRowIndex();
    if(rowIndex >=0 && rowIndex < list.size())
      return true;
    else
      return false;
  }

  public int getRowCount()
  {
    return totalNumRows;
  }

  public Object getRowData()
  {
    if(list == null)
      return null;
    else if(!isRowAvailable())
      throw new IllegalArgumentException();
    else
	{
      int dataIndex = getRowIndex();
      return list.get(dataIndex);
    }
  }

  public int getRowIndex()
  {
    return (rowIndex % pageSize);
  }

  public void setRowIndex(int rowIndex)
  {
    this.rowIndex = rowIndex;
  }

  public Object getWrappedData()
  {
    return list;
  }

  public void setWrappedData(Object list)
  {
    this.list = (List) list;
  }

}

&#91;/sourcecode&#93;
As you see from the beforePhase()-method in the phaseListener I use a util to create the pagedDataModel. In my application I have a index on the back, but this can of course differ from your situation. I will therefore refer to the backend as "service", and whatever you use that holds and gets the data should not matter.
&#91;sourcecode language='java'&#93;
public class PagedDataModelUtil
{
	public PagedListDataModel createPagedDataModel(String selectedLetter, HtmlDataTable htmlDataTable)
	{
			int startPosition = htmlDataTable.getFirst();
			int dimensionToUse = htmlDataTable.getRows();
			Service service = new Service();
			int totalListSize = service.getTotalNumberOfHits(selectedLetter);

			List<Word> words = new ArrayList<Word>();
			words.addAll(service.getHits(selectedLetter, startPosition, dimensionToUse ));
			PagedListDataModel dataModel = new PagedListDataModel(words, totalListSize, dimensionToUse);
			return dataModel;
	}
}

So, we have now a table that retrieves only the data the user wants to see, and the data is not retrieved multiple times. Fine!But then, what about sorting?? Let’s say that one of the columns we have in our table is length, meaning the length of the word in number of characters. What if I want to sort by this?You could do this as well. I assume you have a header in your table, giving the labels for each column. You can create a commandLink on each label, and if you press one of them you set that value into a property, for example “sortedBy” in the dataModelHandler.First, the link:

 			<t:column>
				<f:facet name="header">
					<t:commandLink id="sortBySomethingLink" value="myColumnLink">
						<t:updateActionListener property="#{myDataModelHandler.sortedBy}" value="theFieldNameIWantToSortOn"></t:updateActionListener>

					</t:commandLink>
				</f:facet>
				<h:outputText value="#{someObject.someValue}" />

			</t:column>

We see that this link updates the property sortedBy in the MyDataModelHandler with the value specified. In the phaseListener, when we call the createPagedDataModel, we now need to pass this property.In other words we modify the createPagedDataModel() method to add support for sorting. We have a new parameter called sortedBy, and use StringUtils to test whether we should bring this one into play or not. If it is null or “” (blank), we don’t sort. We could also keep the sortedBy parameter as a property in our PagedDataModelUtil, in this way we can have an ascending / descending functionality. If the sortedBy parameters equals the sortedBy property, we flip the ascending boolean, if not we sort ascending by default:


private String sortedByLast = "";
private boolean ascending = true;

public PagedListDataModel createPagedDataModel(String selectedLetter, HtmlDataTable htmlDataTable, String sortedBy)
{
			int startPosition = htmlDataTable.getFirst();
			int dimensionToUse = htmlDataTable.getRows();
			int totalListSize = service.getTotalNumberOfHits(selectedLetter);

			List<Word> words = new ArrayList<Word>();
			if(StringUtils.isNotBlank(sortedBy))
			{
				if(sortedBy.equalsIgnoreCase(sortedByLast))
					ascending = !ascending;
				else
					ascending = true;

				words.addAll(service.getHitsSorted(selectedLetter, startPosition, dimensionToUse, sortedBy, ascending));
			}
			else
				words.addAll(service.getHits(selectedLetter, startPosition, dimensionToUse ));

			PagedListDataModel dataModel = new PagedListDataModel(words, totalListSize, dimensionToUse);
			return dataModel;
}

Of course, this puts the responsibility of sorting on the backend service. Somewhere we need to do the sorting. πŸ™‚ If you use a database and have a high number of rows/items, this might be a performance issue. In my application I use Lucene (index), and even with a very high number of rows I don’t have any problems with performance. But doing the sorting in the backend and getting the first 25 items e.g. is anyway faster than retrieving all the items and then do sorting.Ok, that was all this time!

Advertisements

18 Responses to “Lazy loading of tables in JSF – working with large amount of data”

  1. Kiran Patel Says:

    How can we handle SORTING along with this lazy loading with pagination?

    could you please provide some info with example on this?

  2. roneiv Says:

    Hi Kiran,

    If you read the last part of my post now, I’ve extended the part where I speak of sorting, hoping that the information I provide there gives you some ideas of how to solve your problem.

  3. steven Says:

    Hi , in a first time, i used your solution , but there is a big issue, because in fact, the method “beforePhase() is always called even if you are in other jsp, so it is too complicated., and you have to do some test(if), otherwise, you get in in the method constantly.

    sorry for my bad english, i’m french!

  4. roneiv Says:

    Hi Steven!

    I would have to agree that it is probably not in all cases this way of doing it is useful. The beforePhase is not linked to the view, but to the scope of your bean and the lifecycle of jsf. Therefore it executes every time jsf goes through the 6 phases.

    Still, in the application I use I have a “top level” jsp which renders header, footer, left frame etc, and then I use tags to dynamically render the page that the user selects. In this way it’s very easy for me to bring up a condition that would make sure that the phaseListener doesn’t execute my method unless I’m on that specific page.

    Perhaps you could think of a solution by using the “Referer” object in the request header? You can get it with something like:
    String referer = request.getHeader(“Referer”);

    You need to cut the host and contextRoot away from this string of course, but then you should have the name of the page/jsp that sent the request. Then adding an if based on this shouldn’t be too difficult:
    if(referer.equalsIgnoreCase(“thePageMyPhaseListenerShouldExecuteIn.jsp”))
    ….. do stuff

    This if means that if the request was sent from the page you want to use the phaseListener on, then you should perform the operations. This is a bit simplified of course, but it might give you a clue or point you to create something you can use at least. πŸ™‚

  5. steven Says:

    Ok i’m going to try it. May be it allows to me to delete all my test. Otherwise i found an other solution on this web site
    http://anydoby.com/jblog/article.htm?id=45
    Take a glance if you have time and tell me what do you think about that?

  6. roneiv Says:

    Hi again Steven!

    I’ve had a look at the example you pointed me to. I guess the biggest difference is that this example allows you to use the “sort-possibilities” included in the datatable and t:column. For my part I don’t use sorting this way, I have different “views” of the data in the table where not all of the views have a header with column names on top, so I implemented a t:jscookMenu (dropdowns) to select the sorted column. But if you use sorting with t:column or t:commandSortHeader this is a very nice solution.

    But except for that he solves the issue about retrieving data multiple times by keeping the last retrieved page in the datamodel. You can see here that if start and size differs, he fetches data, if not he just returns the page as it was:

    private DataPage fetchPageInternal(int start, int size) {
    if (lastStart != start || lastSize != size || page == null) {
    page = fetchPage(start, size);
    lastSize = size;
    lastStart = start;
    }
    return page;

    So if you have a problem using the phaselistener this could be another way of doing it that would also work well πŸ™‚ Note that you could do the changes to the PagedListDataModel only, you don’t necessarily have to use the SortablePagedDataModel if you don’t care about sorting this way. πŸ™‚

  7. steven Says:

    yes ,indeed, he keeps the last retrieved page in the datamodel. this an interesting solution because you don’t have to use phaselistener, and you can delete MyDataModelHandler.

  8. Hemant Says:

    Hi,
    It was indeed a helpful article, I was looking to somehow make the sorting applicable only to the visible rows on that page, the default behavior I suppose is that all the rows( say 50) would get sorted even if the rows per page has been chosen to 10. Is it possible to to it someway?

  9. roneiv Says:

    Hemant,

    Your request should be possible without doing much changes.

    When you get your list of items the first time (using service.getHits() ), keep this list in your class in some property.

    Then, if sortedBy is set, you don’t ask your service to give you new hits sorted by the given column, but you simply sort the list you already hold using a comparator. And then you use your sorted list in the constructor of the PagedListDataModel.

    If you’re not familiar with sorting by comparator, it’s like this:
    Collections.sort(yourList, yourComparator);

    YourComparator would be a class that implements the Comparator interface (java.util.Comparator).

    Hope this helps you! πŸ™‚

  10. Hemant Says:

    Hi roneiv,
    Thanks a lot for your prompt reply. I would try it out. Meanwhile I was looking for a property in the myfaces datatable component itself using which I could achieve it, the sorting feature provided by myfaces datatable sorts everything in the table in all the rows on all the pages. I was hoping that there might be some property which I can turn on/off for it.
    Thanks for the help.
    Hemant

  11. roneiv Says:

    Hemant,

    Sorry for late answer, I’ve been away for a couple of days. πŸ™‚ But as far as I know the sorting feature in myfaces datatable does not have a property where you can decide whether you want to sort the whole collection or only the page you are currently viewing.. πŸ™‚

    Bad luck!

  12. Xavier Says:

    Hi: I have a rather unusual requirement. I received from my client a SQL query that I have to use. This query can return anything from zero to something like 10000 records. The expected behaviour is the DataTable to show nothing more than 10 pages. So, if the number of records is 100, then 10 pages of 10 rows, if the number of records is 10000, then 10 pages of 1000 rows.

    I’m new with JSF (IceFaces), and at the moment, the piece of code for this requirement is weeks in the future, so my question is rather simple: can this be done using a custom DataModel like the one described, only with a different logic? Hope the answer is yes, so I can digg into the code in next days. Thanks in advance.

  13. roneiv Says:

    Xavier,
    Theoretically speaking, this should be possible with a custom DataModel like this. But limiting the number of pages to max 10 can cause problems still:

    – Performance: First doing a query to count the number of results, and then a second query to retrieve the first 1000 rows, it could cost you some time.

    – Memory load on your browser: Depending on how much information you want to present pr result/row, 1000 items could be a lot for the browser to handle. Not to mention that it doesn’t provide a good user experience if you have to scroll up and down 1000 rows to find the information you are searching for.

    I believe I would have reconsidered my max limit on the number of pages. And to support that high number of results I would have implemented this in some index structure liike Lucene instead of a DB πŸ™‚

    Regards,

    Eivind

  14. Xavier Says:

    Hi there again and thanks for the quick answer.

    I actually got into the example code of IceFaces and noted a personalized DataModel, so, as you say, doing a previous query to count the total number of records should do the trick.

    User experience is precisely that one, large scrolling, and they are confortable with it. They just want to find something in less than 10 clicks. They are used to use Ctrl-F plus key-word plus column sorting to narrow the number of rows to scan. Anyway I can’t filter the data coming from the datasource, so … πŸ™‚

    What worries me the most is performance. The large amount of rows won’t be the frequent case, but you never know. I’ll let you know (if you want) what do we do in the end with this req.

  15. aniket Says:

    Hi,
    very nice to see your work. i am trying to implement it and got the pagination to work. However i am facing some small problems –
    1. the beforeListener() method is getting called throught out the website, how do i restrict it to this specific page/class.
    2. on sorting i have to go back to the first page. currently i stay on the current page,probably because first is not set to 0. i have found a work around for this – in the actionlistener for sorting i make the htmldataTable to null, so that every time we sort it creates a new table.

    aniket.

  16. roneiv Says:

    Aniket,

    Sorry I am on vacation now but will respond to you when I am back πŸ™‚

  17. cagataycivici Says:

    Hi, I’ve a better solution now which is built-in to PrimeFaces Datatable. http://cagataycivici.wordpress.com/2009/09/28/lazy-loading-jsf-datatable/


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: