Working with External Data Sources in Silverstripe Part 1: Static Data

So, you’re used to creating DataObjects, querying databases and using SilverStripe’s ORM. But what happens when you need to handle and display large amounts of data from somewhere other than your database, such as data provided by a third-party API?

In this two-part series we’ll look at possible solutions for this scenario, and their benefits and drawbacks.

We’ll look at querying an API in real-time in part 2 of this series, but for now we’ll assume that the data you need to display is “fixed” - i.e. it doesn’t update frequently and you don’t need to do any complex querying. Let’s use the example of university courses provided in XML format. This data is updated once a week, and only needs to be displayed in a simple list view.

How do I fetch the data?

For our example scenario, the data feed is updated weekly, so we can update our website on a fixed schedule. The best way of doing this is a combination of a SilverStripe “dev task” and a Crontab to trigger the task at a set interval.

SilverStripe’s RestfulService API provides a simple way of using cURL to fetch data, here's a quick example on how this might be used to fetch data as part of a command-line accessible task:

<?php
// mysite/code/tasks/UpdateCoursesTask.php
class UpdateCoursesTask extends BuildTask 
{
	protected $title = "Update the course listings";
	protected $description = "Updates the list of courses from the XML feed.";
	protected $baseUrl = 'http://example.com/';
	protected $feed = 'courses.xml';
	protected $feedUser = 'username';
	protected $feedPass = 'p4ssw0rd';

	/**
	 * @param SS_HTTPRequest $request
	 */
	public function run($request) {
		// Construct the request
		$service = RestfulService::create($this->baseUrl, 0);
		// Set basic auth credentials
		$service->basicAuth($this->feedUser, $this->feedPass);
		// Make the request
		$response = $service->request($this->feed, 'GET');

		// Display details of errors when fetching the feed
		if($response->isError()) {
			exit('HTTP request failed: ' . $response->getBody());
		}

		try {
			$simpleXML = new SimpleXMLElement($response->getBody());
		} catch(Exception $e) {
			exit('Error parsing XML response: ' . $e->getMessage());
		}

		// Remove any existing courses - we're overwriting these
		Course::get()->removeAll();

		// Loop over list of courses
		foreach($simpleXML as $courseData) {
			// Cast the SimpleXMLElement to an array
			$courseData = (array) $courseData;

			// Write the course to the database
			$course = Course::create()->update($courseData);
			$course->write();
		}

		exit('Done!');
	}

}

Once you’ve performed a dev/build, you’ll be able to trigger your new task by either visiting http://mysite.com/dev/tasks/UpdateCoursesTask, or via the command line with sake dev/tasks/UpdateCoursesTask (see the SilverStripe command line documentation for more info).


Once you’ve retrieved the data you need from the feed, what do you do with it?

Solution: Dataobjects

The ideal solution for our “courses” example is a DataObject subclass. You just need to specify which fields are provided in the XML data feed, and when you fetch the data, simply write a DataObject to the database for each “row” you receive.

This is certainly the most intuitive solution for those used to working with SilverStripe – it requires minimal effort and is the easiest to understand for any future developers who may inherit your project.

Quick example

// Writing the data: assuming $courses is an array of course data
foreach($courses as $data) 
{
	$course = Course::create()->update($data); // Create a Course DataObject
	$course->write();                          // Write it to the database
}

// Returns a DataList...
$listOfCourses = Course::get();      
// ... which we can treat as an array
foreach($listOfCourses as $course) 
{ 
	echo $course->Title;
}

Solution: ArrayData or ViewableData

This solution is more suited to querying an API in real-time (as will be explained in part 2 of this article), where you’ll probably never need to store the data longer than the lifetime of the current request. An instance of ArrayData or ViewableData can be used in templates in the exact same way as a DataObject, but it is never written to the database.

Should you need to store the data (perhaps certain queries should be cached), you could always use the flexible SS_Cache API to facilitate this:

Quick example

// Writing the data: assuming $courses is an array of course data
$cache = SS_Cache::factory('CourseCache', 'Output', array('automatic_serialization' => true));
$courseList = ArrayList::create();
foreach($courses as $data) 
{
	$course = ArrayData::create($data); // Create an instance of ArrayData
	$courseList->push($course);         // Push it to our list of courses
}
$cache->save($courseList, 'courses');   // Save the data to our cache

// Loads the ArrayList from the cache...
$listOfCourses = $cache->load('courses'); 
// ... which we can treat as an array
foreach($listOfCourses as $course) 
{
	echo $course->Title;
}

Your ideas

These solutions are just two ideas I’ve used in the past on different projects, if you have any other solutions or tips I’d love to hear them and add them to the article. Please get in touch via email or Twitter.

Published on

15th October 2014
by Loz Calver

Filed Under

SilverStripe

How can we help?

Please send us some details of your requirements and we’ll be in touch.

Please note by submitting your details you are agreeing for Bigfork Ltd to store your data in order to process your enquiry and that you have read our Privacy Policy.