Working with External Data Sources in Silverstripe Part 2: Querying an API

In Part 1, we looked at how you might extract data from an XML feed that’s updated weekly and store it. In this part, we’ll study possible approaches to querying data in real time.


The problem

Our example API will be a property search. This API allows you to filter data by passing query string parameters, and returns results in JSON format. As an example, valid request URLs might look something like:

http://my-property-provider.com/api/search.json
http://my-property-provider.com/api/search.json?minBedrooms=3
http://my-property-provider.com/api/search.json?minPrice=300000&maxPrice=400000

You’re building an application that needs to query this API in a few different places. For example: a feed on the home page, a full-featured search form, and in the CMS. As a result you want to build a class, or set of classes, that you can re-use across your codebase to prevent code duplication. To keep things simple, we’ll only look at GET requests.

What?

The best solution I’ve found to this problem is to mimic the behaviour of ArrayList / DataList. This can be achieved by creating a class which implements most of the methods that ArrayList / DataList contain, and copying DataList’s “lazy loading” behaviour - where a query isn’t actually executed until you attempt to access the results.

Why?

This implementation has a number of benefits over other approaches. Firstly, the end result is an API that’s familiar to all SilverStripe developers: methods to filter, limit and sort results can be chained in the same way that DataList methods can - e.g. Property::get()->filter('Foo', 'Bar')->limit(5);.

The second benefit of this approach is that all the code used to query the API is nicely contained, rather than being dotted about your codebase in different Controllers.

Possibly the most important benefit of all, however, is that this approach will allow you to use your logic in a GridField.

Simply querying an API, putting the results into an ArrayList and passing that into a GridField is not a good idea: ArrayList filtering and limiting is all done in-memory by PHP. While this may work initially with a few results, you’ll quickly encounter performance issues (and probably memory limits) as the number of results begins to grow.

The approach of building your own class similar to ArrayList allows you to offload the work of filtering, limiting and sorting results to the “third-party” API.

How?

Picking up on the last benefit from above regarding displaying results in a GridField instance, the first thing we need to do is ensure that we’re building an API that’s fully compatible with GridField.

Thankfully, SilverStripe provides a number of interfaces which guarantee compatibility by setting out required methods. The interfaces that we’ll need to implement are as follows: SS_List (which allows our class to be treated as an iterable list container), SS_Filterable, SS_Sortable and SS_Limitable. While GridField only explicitly requires SS_List to be implemented, many of the components such as GridFieldFilterHeader require the other interfaces’ methods to be implemented as well.

Let’s take a look at some simplified code for our property search example:

<?php
class Property extends ArrayData 
{
  public static function get() 
  {
    return PropertyList::create();
  }
}

class PropertyList extends ViewableData implements SS_List, SS_Filterable, SS_Sortable, SS_Limitable 
{
  protected $filters = array();

  public function filter($field, $value) {
    $this->filters[$field] = $value;
    return $this;
  }

  // ... other methods for sorting, limit, offset
  public function getIterator() 
  {
    return new ArrayIterator($this->toArray());
  }

  public function toArray() 
  {
    $rows = $this->execute();
    
    $results = array();
    foreach ($rows as $row) 
    {
      $results[] = Property::create($row);
    }

    return $results;
  }

  public function execute() 
  {
    $service = RestfulService::create('http://my-property-provider.com/api/', 0);
    $service->setQueryString($this->filters);
    $response = $service->request('search.json');

    if($response->isError())
    {
      $results = array();
    } 
    else 
    {
      $results = json_decode($response->getBody(), true);
    }
    
    return $results;
  }
}

Walking through the above code step-by-step:

  • A developer calls Property::get()->filter('foo', 'bar');
  • PropertyList::filter() stores the filter values in the $filters class property
  • As soon as the developer tries to access the results (via ->first() or a foreach loop), PropertyList::getIterator() is called thanks to some PHP magic (explained below)
  • This then triggers PropertyList::toArray(), which calls PropertyList::execute() and sends the request to the API
  • Each result is then transformed into an instance of Property and returned in an array

How was PropertyList::getIterator() magically called?

The SS_List interface extends a number of built-in PHP interfaces, including IteratorAggregate. Whenever you do a foreach loop on an object instance, PHP will check if that object’s class implements Traversable. Because IteratorAggregate extends Traversable, and SS_List implements IteratorAggregate, SS_List passes this check.

PHP will then automatically call getIterator() on the object and will loop over the returned result. As you’re not allowed to just return an array from getIterator(), we return an instance of ArrayIterator (which essentially matches the behaviour of an array).

Put simply, by implementing SS_List, we tell PHP that it can transform a regular object into something that can be looped over by calling the getIterator() method.

Published on

18th January 2015
by Loz Calver

Filed Under

SilverStripe

How can we help?

Please send us some details of your requirements and we’ll be in touch.

Please note by submitting your details you are agreeing for Bigfork Ltd to store your data in order to process your enquiry and that you have read our Privacy Policy.