output stream » XML & XSL

Exporting bibliography from library catalog to XML

dtjohnso — Tue, 08 Feb 2011 18:58:58 +0000

As part of a ~~LibGuide~~ project that I was working on recently, I wanted to export the bibliographic data for rare Bibles in our Mack Library collection so that I could generate a list for public viewing. The rare Bibles in our collection are housed in two rooms, the Jerusalem Chamber and Special Collections, and because few people ever see the books in Special Collections I thought it would be nice to create a browsable list. Our current library catalog does not make it possible to browse the items in a particular room (or “location code” in library science terms).

The process I came up with involves three major components:

Finding and exporting the bibliographic records in Millennium using Create Lists
Filtering, finalizing, and structuring the data in Excel 2007
Converting the structured data into clean HTML with XSLT

I’ve posted my files in a github repo for others to adapt.

Step 1: Export records from Millennium

To create my list, I did an item record search based on location code, but stored the bib records in my list.

I then exported all the data to text. Because some of the variable-length fields included commas, I used tabs (“Control character” 9) as my field delimiter.

Step 2: Manipulation in Excel 2007

For my situation, data processing in Excel involved a number of small steps. Steps 2.2 and 2.3 may not be necessary for every use case.

2.1 Import data

A number of the titles in our collection are not in English, and so I had to specify that the imported data was Unicode (UTF-8). I found out that Excel 2003 does not support importing UTF-8, but Excel 2007 does.

On step 3 of the import wizard, I set the data format for all columns as “Text” rather than “General” to avoid any unwanted conversion of my bibliographic data.

2.2 Filter records (if necessary)

Because our Special Collections room includes many items besides the Bibles that I was interested in, I needed to filter my data. I did a quick filter for all items with Uniform Titles beginning with “Bible.” and used Select Visible Cells to copy and paste the data I wanted to a new worksheet to prepare for export.

2.3 Produce truncated bibID

The current version of the Innovative Web catalog at my institution does not respond correctly to a full 9 character bibID. Instead, I have to truncate the checkdigit at the end to get a valid link. Now, since that may not make much sense, let me just illustrate:

http://library.bju.edu/record=b1184983 VALID
http://library.bju.edu/record=b11849836 – INVALID (at least, for now)

The first example does not include the checkdigit “6″ at the end. Since the checkdigit is included in the data exported by Millennium, I created a new column at the end of my data and populated it with this Excel formula:

=LEFT(A2,8)

where “A2″ is the reference to the cell containing the untruncated bibID.

2.4 Match columns to XML Schema

In order to prepare my data for export, I need to match each column to an attribute in a custom XML Schema so I can export structured data that I can meaningfully interpret into HTML.

To do this, I’ll need to create my schema file, load into Excel, and match each element to the appropriate data columns.

My XSD is as follows, and can be easily adapted to another situation.

Once I have a valid schema, I load it into Excel through XML Source window on the Developer ribbon. To add a new Schema, click the “XML Maps” button and then “Add.” Then I drag each attribute in the XML tree visible in the XML Source pane and drop it on the appropriate column in my data until the result looks like this:

2.5 Export to XML

To generate XML from my formatted file, I click the “Export” button in the XML section of the Developer ribbon and save to file. Excel will follow my custom schema to produce structured data that I can use to generate HTML output.

Step 3: Generate HTML with XSLT

Now that my data is in XML, generating HTML is a simple matter of applying an XSL stylesheet. My stylesheet is as follows:

Producing valid RSS from my XSL Transform

dtjohnso — Thu, 11 Oct 2007 21:15:37 +0000

In a previous post, I showed the XSL transform I designed to transform a simple XML file I use into “(almost) valid RSS.” I called it (almost) valid because of one problem, which I address here.

The problem reviewed

My original XSL assigned an arbitrary ID to each entry in the news file based on its order in the list. What this meant was that the first item received the integer 1 for an ID, the second 2, the third 3, etc. Here it is:

<!--old method, produces invalid link & guid-->
<link><xsl:text>http://www.duncanandmeg.org/news.php#</xsl:text>
  <xsl:value-of select="position()" /></link>
<guid><xsl:text>http://www.duncanandmeg.org/news.php#</xsl:text>
  <xsl:value-of select="position()" /></guid>

This was fine for my original purposes, but since this hardly represented a unique & permanent identifier for each post, it caused some problems when echoed into the RSS and elements. The significance of the element in particular is eliminated with such an approach. Every time I added an item to the list (see the previous post for an explanation of my schema), it would be assigned the ID of 1, and the older posts would be assigned the ID numbers 2, 3, 4, 5 etc.

This meant that most RSS aggregators would not detect that a new post had appeared and would not update accordingly.

Choosing a unique ID schema

Since I could no longer use my simple numbering scheme based on position in the original XML file, I had to come up with some other identifier. I learned about the Tag URI algorithm, but decided that it was more complex than I really needed for this application, and the scheme of my original XML file didn’t lend itself to producing these anyway.

Since I don’t update the entries in this file frequently, I invented a simple method that produces relatively unique identifiers based on each entry’s data. All my new ID’s consist of two elements based on the data in my XML file:

The text from the entry’s title preceding the first space (” “) character
(If the entry consists of only one word, this element will not appear. This is a weakness that would be more important if updates were frequent)
The date of the entry encoded in the format mm-dd-yy

Once I settled on my scheme, producing it in XSL turned out to be fairly simple.

XSL transform

I’m including only the portion of the XSL tranform that deals with the entry and here. The rest of it is in the previous post.

<item>
  <title><xsl:value-of select="headline" /></title>

  <!--unique id created and stored in XSL variable-->
  <xsl:variable name="bookmark">

    <!--find word before first space-->
    <xsl:value-of select="substring-before (headline, ' ')" />
    <!--produce date in format mm-dd-yy-->
    <xsl:value-of select="translate (date, '/', '-')" />

  </xsl:variable>

  <!--variable placed in link and guid elements-->
  <link><xsl:text>http://www.duncanandmeg.org/news.php#</xsl:text>
    <xsl:value-of select="$bookmark" /></link>
  <guid><xsl:text>http://www.duncanandmeg.org/news.php#</xsl:text>
    <xsl:value-of select="$bookmark" /></guid>

  <!-- continue with transform, see original post -->

</item>

With a reasonably unique guid, most RSS readers properly update now when I post new items.

Rendering XML as valid RSS with XSLT

dtjohnso — Mon, 13 Aug 2007 21:30:02 +0000

I’ve been using a simple XML file to store updates to the main duncanandmeg.org website, and wanted to use XSL to transform that data into a valid RSS feed. This took a little more work than most of my XSL templates to date, so I thought I’d log the details here.

XML data

The following is the structure of my XML file. I’ve been storing dates in the file in short date format. The detail element is optional, and since is the only element where I embed raw HTML, I encapsulate the contents in CDATA elements.
Also, since I’m updating this file manually at this point, I add new entries to the top of the file rather than worry about using more automated date sorting.

<news>
  <entry>
    <headline>headline</headline>
     <date>mm/dd/yy</date>
    <detail><![CDATA[more details (if applicable)]]></detail>
  </entry>
</news>

XSL transform

Because RSS is a more complex standard than my simple XML file, the XSL transform is necessarily complex.

<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <xsl:output method="xml"/>

  <xsl:template match="/news">
  <rss version="2.0">
    <channel>
      <title>hard-coded title</title>
      <link>hard-coded url</link>
      <description>hard-coded description</description>
      <ttl>60</ttl>
      <xsl:apply-templates select="entry[position() < 6]" />
    </channel>
  </rss>
  </xsl:template>

  <xsl:template match="entry">
    <item>
      <title><xsl:value-of select="headline" /></title>
      <link><xsl:text>http://www.duncanandmeg.org/news.php#</xsl:text>
        <xsl:value-of select="position()" /></link>
      <guid><xsl:text>http://www.duncanandmeg.org/news.php#</xsl:text>
        <xsl:value-of select="position()" /></guid>
      <pubDate><xsl:value-of select="date" /></pubDate>
      <description>
        <xsl:if test="detail">
          <xsl:value-of disable-output-escaping="no" select="detail" />
        </xsl:if>
      </description>
    </item>
   </xsl:template>

</xsl:stylesheet>

This produces almost valid RSS feed for the top 5 items in the feed.

The generated RSS includes dates in short date format (mm/dd/yy), which violates the RSS standard which calls for the RFC-822 date-time format. Since all the implementations I could find on the web for doing date conversions in XSL were far too complex for my taste, I proceeded to do that with PHP when I actually transform the XML file into RSS.

PHP code

The following PHP represents a minimalist approach to executing the XSL transform and returning RSS as output. This script doesn’t handle caching, and there are issues with the and elements in the template for each .

The consequence of these weaknesses is that some feed readers don’t properly update when a new post is added to the news.xml file. This is something I’ll perhaps address in another post.

<?php
  //Optional. Sets default time zone to something other than GMT
  date_default_timezone_set('EST');

  //Load XML data and transform to RSS
  $xslt = new xsltprocessor;
  $xslt->importStyleSheet(DomDocument::load('rss.xsl'));
  $xmlResult = $xslt->transformToXML(DomDocument::load('news.xml'));

  //Load resulting data into the SimpleXML processor
  $xml=simplexml_load_string($xmlResult);

  //Loop through RSS data in SimpleXML
  //Replace each short date with an RFC-822 date-time
  $count = 0;
  foreach($xml->xpath('//pubDate') as $dates){
    $newDates = date(DATE_RFC822, strtotime($dates));
    $xml->channel->item[$count]->pubDate = $newDates;
    $count++;
  }

  //Optional. These lines insert RFC-822 values
  //for channel/lastBuildDate and channel/pubDate
  $xml->channel->addChild("lastBuildDate",$xml->channel->item[0]->pubDate);
  $xml->channel->addChild("pubDate",$xml->channel->item[0]->pubDate);

  //Set output content-type to XML
  header('Content-type: application/rss+xml; charset=utf-8');
  print $xml->asXML(); //Return as XML
?>