My colleague Jeff built the outline of this months ago, and I finally had the time to integrate all the pieces. Jeff did all the heavy lifting on isbn.nu for the past few years in terms of refactoring old code, turning into a maintainable and scalable beast, and answering my programming questions. He's helped me become an object-oriented programmer, and to understand how to build code that works.
In the last few weeks, I've built an API for isbn.nu, an interface that will allow an application programmer I'm working with to create an iPhone/iPod touch application that will work as a front-end to isbn.nu. That API sort of necessitated fixing authors into their unique identities and solving a number of other site problems that have been lagging.
I've got a bunch of other features and improvements coming at long last, but this unique author page/identity is one of the ones I've wanted to add for years.
]]>There are no 979 ISBN-13s out there yet (as far as I can tell), so it's a good time to make the transition.
I've written up a page that explains to my isbn.nu users precisely what's going on. It might be too technical/industry for them.
]]>First, the 10-digit ISBN (known now as ISBN-10) will no longer be the standard for use. R.R. Bowker, which coordinates ISBNs in the US, says that ISBN-10s can be phased out from use on books starting on January 1. It's not obligatory to get rid of them, but it will make less and less sense, because of point 2.
Second, the use of ISBN-13s that start with 979 will come into being. Up until Dec. 31, 2006, ISBN-10s have been freely convertible into EAN UPCs, or the global barcodes used to identify national origin and product stockkeeping unit (SKU) numbers. ISBN-10 can be converted by taking the digits 978 plus the first nine digits of the ISBN-10 and computing a new base-10 checksum for digit 13. (ISBNs use base 11, where the 10th digit can be zero through nine, or X representing 10 in base 11.)
With the introduction of 979s, not all ISBN-13s will be convertible into ISBN-10s; only 978-prepended ISBN-13s have a corresponding ISBN-10.
The reason for this is numberspace. They've run out of ISBNs in the current 10-digit space, and by adding another EAN prefix, they buy 10 to the 9th power potential new ISBNs. These are assigned in blocks to publishers, based on the publisher's scale and country of operation, and thus the numbers are used inefficiently. Certain ranges are reserved, as well.
I wrote about this at some length and less clarity two years ago. The deadline is finally upon us!
]]>The Chunky Problem relates to how people conceptualize information and then how that information is represented in databases and on Web sites, to name two concrete instantiations. Despite decades of analogy between the psychic construct that is the mind and card catalogs, computer filing systems, and so forth, it's clear from equally long stretches of research that people represent information in chunks that interrelate. Thus, even someone who can remember 1000 discrete ISBNs or phone numbers isn't representing that information with one neuron per digit. Rather, there's a holistic standing wave in which information flows as a basis of triggers and relationship that allows us to contain these facts.
You wouldn't know this to look at most Web sites that present information of any kind about books, media, or, really any data that's available in masses of larger than a few. Web sites typically represent data as stored in discrete, and offer very flat databases. A flat databases lacks relationships: each record is defined uniquely by one or more fields, and all of the information in the record is fielded. There's always an entry (even null) for each field. A relational database allows richer information by creating virtual objects: a list of attributes in one table might be assigned via another table to an object that resides in a third table which uniquely defines it.
For instance, a book might be a library edition paperback with an embossed cover. One table stores book attributes paperback, hardcover, turtlebound, etc. Another table stores a list of books by ISBN. The joining table then pairs the unique ISBN with the attributes to create an object.
XML is quite exciting to most people involved in structuring information because it allows the definition of an object-oriented structure and attributes that adhere to it without requiring the rigid formalized storage of a database. In an XML file, you can create objects explicitly, producing human-readable and machine-parseable results that represent data in a simpler and substantially more powerful method.
Now The Chunky Problem is that most media data is in very flat form. In fact, when I license data or review data for license for media like books and movies (in DVD and other form), I find that there is often neither no table structure and no normalization. Rather than have a table of actors with attributes (like a biographic sketch) and tie those by reference using a unique identifier to a table of SKUs (individual movie UPC codes, for instance), it's just a long list with repetitive information all organized uniquely by SKU.
Since information wants to be chunked, or so I think, I have spent nearly a decade working on systems--mentally and in actual code--that rework flat information into rich relationships that can be represented chunkily to an end user.
Thus, my long-standing example of The Wizard of Oz. There is a work of fiction called The Wizard of Oz, and it is instantiated as many, many things, which includes dozens or of ISBNs, dozens of books that precede the days of ISBNs and are out of print, collections in which it appears as an item in a larger compendium, essays about it, and parodies.
At the chunkiest level, you have the work, The Wizard of Oz. Move down a chunk and you're into categories of things: books, movies, parodies. Move into the books chunk, because you're thinking "I want to buy the book, Wizard of Oz," and you find the sea of types: paperback, hardcover, large print. Finally, we can delve into a specific SKU or ISBN.
There are ways to short circuit this, too. Imagine the chunky statement, "I want to buy a new copy of the latest edition of the book Wizard of Oz." Zoom down through and the user is on that precise book's offering page with details. Or, "find me a collection in print of The Wizard of Oz and Ozma of Oz." A system that's chunky knows about those two works, and can track those words as uniquely identified items across containers, which are instantiated editions. Thus, the system knows that ISBN Z is a container of unique work Wizard of Oz and unique work Ozma of Oz.
Chunkiness works as a way to share information across objects, too. A review of Wizard of Oz as a work should adhere at the chunkiest level. A review of a particular newly edited edition that might appear as several ISBNs is much less chunky. Chunkiness needs to adhere to objects at various scales.
There's a great little joke about a professor who brings a large glass jar into a class along with a bin of rocks, a bin of pebbles, and a container of sand. He pours rocks into the jar and asks the class if it's full. They agree it is. Then he shakes in pebbles. Is it full now? Yes. Then he pours in sand? Now? Yes. Then (in some versions) a couple of beers. It's apparently a metaphor for life. (The moral in many tellings: there's always room for a couple of beers.)
The Chunky Problem looks at this as the reverse: it used to be when you went to a bookstore online, the whole jar was full of sand. Over time, it became full of pebbles. Now it's a rocks, pebbles, and sand. I'd rather view it this way: the jar contains rocks. The rocks, when broken down, turn into pebbles. The pebbles, when pulverized, are sand. But unlike physical objects you can go from a jar to a grain of sand in one step, and the process is reversible at any time.
Most product sites would benefit from getting chunky.
]]>The problem with persistence relates to creating objects that represent a particular set of ideas that have a persistence as that object over time. Good example: The Köchel or K Numbers assigned to Mozart's works. Mozart himself didn't number or catalog his own oeuvre, but a 19th century intellectual did. The works are numbered by appearance and the numbers have been used widely to refer to specific works. The numbers don't change, even if additional works are discovered between existing numbers, to avoid destroying decades of research and publishing. The numbers are persistent object identifiers to which many attributes are attached.
In Bookland, our happy 978 and 979 UPC code prefix world, there's no such animal. I am finding there is no such animal among many realms of information in which there are discrete entities that have members but not a persistent and consistent method of numbering them.
In developing a new DVD and video price comparison service that I hope to launch later this year or in early 2006, I worked with a programmer to develop persistence in the kind of unique work objects that I create for isbn.nu every time the database is regenerated.
The idea is simple, when we finally figured it out: You have to build a structure in which the information present in the structure is authoritative when a record exists. New information is always considered less authoritative for specific values in an object, and reconciliation can happen or a queue of details to reconcile can be built based on the manner in which a particular new detail is out of sync with existing records.
Practically speaking, this means you prime the pump of a database on, say, movies with a good authoritative and deep source and then choose specific values that are unique to become authoritative, such as a director's name (even the spelling) or a movie production's title, like "Star Wars: Episode I." When new information arrives, it's checked by SKU against existing information. If the authoritative details vary they are either ignored or stuck into a queue for review--it's possible that the SKU had the wrong title, for instance. In either case, non-authoritative details, such as the movie's length or the DVD region encoding, are updated to whatever the new information provides.
This isn't the full depth of hygiene and reconciliation I'd like, but it means that once an object is created and numbered for a unique movie production such as "Pride and Prejudice" as produced in 2005, that object can retain the same ID number or other code permanently. All other details will accrue to or revolve around that initial unique movie production information.
I'll write more about this as the system rolls out, as it will for both the new movie site and for my existing isbn.nu book site.
Remarkable coincidence: This David Weinberger piece about the issues of metadata and book identification and sub-book (chapters, etc.) identification appears today in the Boston Globe.
]]>Okay, not everywhere. They were just among the topics of the top book searches at isbn.nu. I have a page that shows me and anyone else who cares the most recent 10 searches and (since I added this a few days ago) the top 10 searches since the database for popularity was reset.
This has helped me understand how the Internet works a little better, as I can explain the source of the popularity of some of these links.
How to Have Sex in the Woods. Many people search Yahoo asking "how to have sex"--there's nothing about the woods in there at all. The book price page on my site is the #11 answer to this question.
101 Ways to Promote Your Web Site. One spammer trick is to flood your site with referrals. They hope that you either review or publish your statistics, thus increasing their Google Whuffie. This book was highly "promoted" by get-rich-quick sites through referral spam. It doesn't have anything to do with the quality or nature of the book; I suspect it's just a random link choice.
Dollar Bill Origami. Sounds like a cool book, but folks arrive here because they're searching on dollar bill origami over at Google. I did not know so many people were interested in folding dollar bills.
Endangered Tigers. Google thinks I know something about this topic.
Rapid Application Development with Mozilla. A single page on RDF links to where to find prices for one book. And it provokes dozens of clickthroughs. Must be a popular page.
Quarkexpress 6: For Print and Web Design. The program name is misspelled in this iteration of the book title in my database. It should be QuarkXPress. That extra 'e' in the middle means that Google points folks to me whenever they mis-search on Quark's flagship program.
Lions: Life in the Pride and The African Lion: This one flummoxed me until I found that these two books and links to my site were used as part of an example in "Representing Classes As Property Values on the Semantic Web." I wrote the author telling her how amusing I thought this random traffic was! She was apologetic, and I said I did not mind: the more people that find my site, the better.
Finally, an old favorite, Susie Bright's book Mommy's Little Girl: On Sex, Motherhood, Porn, and Cherry Pie. She's a fantastic writer. But why me? (Or rather, why isbn.nu?) The answer is, unfortunately, horrible. Searches on "little girl porn" lead one to my price service's doorstep. I feel like putting up a custom page for those referrers--not for this book--"Y'oughta be ashamed of yourselves, pervs!"
]]>I'm starting to think more about how to build a Wiki-based repository of book details. There are a huge number of problems with such a project, but I do know how to seed the data. The Library of Congress's Cataloguing Distribution Service resells the vast data that the LOC has compiled on books, which include their holdings and general catalog information. The information is not copyrighted (per se) for use in the U.S. U.S. citizens have certain copyright interests when it comes to information created by the government. NASA, for instance, holds the copyright for images it takes, but releases them for use by U.S. citizens without advance or specific permission or licensing fees. Both the LOC, NASA, and other agencies can charge cost-recovery fees for distributing and maintaining data, which is why the CDS charges many tens of thousands of dollars for a complete set and subscription.
This information is available from other sources. Some libraries and other groups have purchased, say, Books All, the full set of book data, and could re-distribute it for free to a Wiki-bibliographic project. (Wikibib? Wikilibris?)
If the project were set up correctly, there would be an API that would allow publishers and other parties to import massive amounts of data in the right format into the project, too. So there would be many inputs that would have be monitored and contended with.
As I've written previously, corrections are a particularly troubling problem. Authoritativeness has to be an aspect of correcting information, but who decides who is authoritative? That's a problem that the Wikipedia contends with (some say) or thrives upon (others say).
]]>C. Max Magee explains that there will be some real transition issues for smaller booksellers who have to convert legacy systems. On the other hand, I've been told by a lot of the bookseller aggregator sites--ones like Alibris and ABEBooks who entirely or largely list books from independent booksellers--that most of the stores use one of a few inventory management packages. (Hey, so all the online booksellers have to move to using EANs primarily, too; some allow them now.)
The 978 prefix, which identifies a mythical country in the EAN worldspace called BookLand, allowed ISBNs to be freely converted into internationally compatible EAN systems. More confusingly, the US has been using a 12-digit UPC code in retailing which is transitioning to the full 13 digits for better globalization.
(Mark of the Beast theorists, start your biblical engines!)
It's an expansion of the namespace for ISBNs, too, because all existing ISBNs will be honored in the 978-prefix namespace forever. Publishers with existing ISBN inventories can continue to use them by converting them into the 978 system. New ISBNs will be assigned as a complete EAN number starting with 979 and will not be convertible back to the old 10-digit system. This opens 1,000,000,000 new ISBNs for assignment, just incidentally, since the two namespaces (978 and 979) will be independent.
Here's the U.S. ISBN authority's explanation.
Magee's site is worth reading for any inside-baseball bibliophile.
]]>So I ask: is anyone using Onix?
And Onix doesn't solve normalization or authority, just data fielding and formatting.
]]>I have to say that I'm pretty excited about the notion of building a Wiki sort of informational site. The Wikipedia itself is a shining example of how to manage such a project, and because facts about books aren't copyrighted, there's very little risk of having people contribute tainted data. In fact, depending on how well the project is structured, some online bookstores might provide a core of data to start with in the project in order to get the benefit of collaborative error fixing.
I know as an author it frustrates me that there's no good way for me to say to the world of bookselling, here's the definitive, authoritative, author-a-tative version of my book's details.
No.
Heaven forbid. Far too easy.
Instead, the vast majority of bibliographic data is entered by many hands in many places. The publisher may provide one form of database, or even a very clear XML dump, but probably in their own schema. As I talked about in my Scottish post just previously, unless you agree to normalization and authority, you're lost.
If Stephen King is listed as "King, Stephen (author)" in the data sent by one publisher and "Stephen E. King" by another, there has to be a mechanism by which you re-normalize or map all incoming autonomously normalized data into one standard set. This is the process of authority, generally speaking, but all data suppliers--authors, publishers, independent collectors of book data like Books In Print--must be consulting the same authority, or have a mapping from their normalization to an agreed-on normalization.
Thus, the new, bad information problem. If a name is misspelled and a new piece of data is collected that indicates the name is misspelled, how can a database be reconciled among those two pieces of data? Most booksellers and book information providers, and I speak from broad experience, produce flat outputs in which all new information is supposed to always be better than old information.
This produces the additional effect, obviously, that if you take information from multiple sources and they each produce feeds of corrections and updates, how can you even tell among two sources updated on the same date even at the same minute whether one is now authoritative and one is not?
You can't.
I support a wholesale revision in the method by which book information should be corrected, and am thinking about how to build a system to support my method of thinking. In my world, you have a definitive current snapshot of the most reliable reconciliation of all data--that's your live feed. But you also have a very deep mine of the change history with authoritativeness attached to it.
My proposed database structure would have a table for each field, and each table would have a datestamp, the informational value, and a score representing authority; a comment field would also be necessary for human review when needed.
The score would correspond to values like:
Physically examined the book
Author provided detail in email
Provided in updated XML feed from publisher
Manually renormalized
and so on.
The tricky part here is established what trumps what. For a page count field, you might have a simple hierarchy in which a physical examination by a trusted party of the actual book trumps everything else except a "physical examination was incorrect" override by a master authority.
It gets more complicated when you look into normalization, as always. Let's say you get data from sources A, B, and C. A tells you in an initial entry for a new book that the author's name is A. Alfred Smith. B tells you it's Alfred A. Smith. C tells you it's A.A. Smith. Don't laugh: I've seen the same title of a book--often with the image on the booksellers' sites--appear six different ways on eight stores.
How does poor Mr. Smith get his name correctly formed? One aspect might be to score the corrections from each source. If A has fewer patches applied to its data by a certain margin than B and C, A trumps. If B provides a new record later, then B's changes to the name might be ignored, unless the name goes from A. Alfred Smith to B. Benjamin Smythe. Notice the level of logic you need to compare normalizations: we'd want to score the differences between components of each kind of fielded data.
This is all to say that new, bad information is a substantial problem, and any method of trying to fix the accuracy, authoritativeness, and normalization of book--or any kind of fielded data--has to involve scoring, comparisons, and history.
]]>One of the normalizations was that whenever the word macintosh appeared anywhere in a book title, the elves or daemons in their clean-up routine changed it to MacIntosh. That's right: with hundreds of computer books including the computer model Macintosh (no capital letter i) in the title, Ingram was overriding this.
The Scottish Problem was a knotty one because of a related issue that I call the New, Bad Information Problem which I will spend some time on this blog discussing. If you fail to tag every field of information you receive from another party or enter with characteristics then when you receive new, bad information, it can easily overwrite your existing revised, good information.
Thus you need depth to every field, from page count to title to scholastic level. Otherwise, there's no way for anyone managing the information or collecting it to understand why a change was made, and whether that change should permanently overwrite any new, bad information. In fact, you can't even evaluate whether new information is bad or not without a history and a process of reconciliation.
I'll write more about this soon, including my ideas on how to use a revision history per field per record along with prioritization.
]]>I'm launching this site in the interests of starting conversations about the way in which book details -- author, title, subject, and even page count -- are collected, sold, disseminated, updated, broken, and misused.
My credentials? I worked for Amazon.com from 1996 to 1997 as catalog manager. In that capacity, I worked with data vendors, and developed or help developed in-house resources as well as Web site tools that would provide the best information about each book we listed. We wanted the list to be exhaustive, but also accurate. I developed an intimate knowledge with several data feeds from book distributors and the Library of Congress in the process.
Part of the outgrowth of my time at Amazon was the realization that the way in which most online bookstores and library catalogs dealt with searching was structured around the ISBN or a library cataloguing number, like the LCCN. Instead of dealing with books as works--that is a discrete idea not an instantiation as an edition--the searches always seemed to pull down every instance. If I search on Wizard of Oz, I'm generally thinking about the work "Wizard of Oz"--I need tools that turn my concept into a set which I can then refine down into the members of.
I used these concepts as part of my consulting for Powell's Books and Half.com after my non-compete with Amazon.com expired, although neither really implemented this part of my idea. I built isbn.nu partly as a programming experiment in 1999, and partly as a forum in which to try out my ideas on this topic.
Years after I left Amazon.com, they introduced a version of this kind of linkage, which has no particular name. Search on Wizard of Oz, and you're still presented with a long list, but the second item has a link that lets you see all 98 titles linked to a single authoritative Wizard of Oz. It's still not great: you can't view through editions. And if you click through to the first result in the list, you see a link that shows eight other editions in a better format. But what about the other 91 results?
ISBN.nu has had a kind of authority that I dub work authority for all of the books I list, which is nearly three million. Library scientists use the term authority to denote how to normalize multiple instances of, say, an author's name into a single definitive version. So if Jimmy Carter, James E. Carter, and James Earl Carter III are really the same person, the authority entry for this person maps those to, say, Jimmy Carter (which is President Carter's preference, I believe).
Authority doesn't mean that the information is correct. Rather, it means that you have authoritatively settled on a single form of a category of information that might be represented in several ways. It's a way to collapse lots of individual information that is fundamentally about the same thing into a single set of information that is mapped to the same thing. This is closer to how people conceive of what they want from a book search than any of the tools I've seen.
ISBN.nu's system isn't perfect. I haven't developed full author or title authority yet, so the same author may be listed in different ways and different authors with the same name are though to have written books they have not. I have worked with a programmer to fully normalize our data, which means to remove a lot of the detritus in information that comes into my system from a licensed source and file down the parts that aren't legitimate differences, like capitalization, extra spaces, and punctuation.
The next step beyond normalization will be full authority development, which I have in process.
In the coming entries, I hope to discuss a number of book information and meta-information problems I have encountered and coped with including The Scottish Problem, The New, Bad Information Problem, The Chunky Problem, and others with names that I hope you find similarly amusing.
I'll start talking about OCLC Research's xISBN project, and how I believe that with a little effort and some coordination, along with the tenets of Feist v. Rural, the Internet community could develop a WikiBook-a-pedia, or a compendium of updatable bibliographic information along with authority and chunky linkages that could be distributed freely.
Comments are welcome. I'm using TypeKey from Movable Type to prevent comment spam and other problems. I apologize for requiring a centralized, email-verified registration, but this seems like the ony solution to ensure that comments aren't turned into a horrible mush as happened on other sites I operate.
]]>