Standards » VoxPopuLII

Standardizing the World’s Legislative Information—One hackathon at a time

Annotation of legal texts, Cross-language legal information retrieval, digital law, Electronic government, elegislation, Legal metadata, Legal XML, Open Government Data, Semantic annotation of legal texts, Standards 8 Responses »

Sep 172012

As guest bloggers to this site, we have been asked to write about big ideas. We’ll get to those. But first, a note about hackathons.

Could legal hackathons be like this one day?

Hackathons used to be the exclusive domain of soda-and-coffee-guzzling, pizza-eating, all-night hacking, highly competitive computer programmers. The result of such a hackathon is often supposed to be a cool app (like the forerunner of Twitter) that is even cooler because it was built in the compressed schedule of the event. More recently, hackathons have been popping up in a variety of places, with some unexpected contexts and sponsors including the U.S. House of Representatives, NASA, Brooklyn Law School, New York City government, and others. These events serve as a way to prove (or build) the sponsor’s tech credentials and to cross-fertilize policy and technology expertise. There has been some handwringing and thoughtful commentary about the expansion of “civic” hackathons and what sustainable outcomes they produce.

As co-organizers, with Karen Suhaka, Greg Wilson, Charles Belle and others, of two legislative focused, hackathon-inspired events–the California Law Hackathon, and the International Legislation Un-hackathon–we can attest to their value in bringing engineers and lawyer and policy folks together. We can give some insights into the kinds of benefits these events have had in propelling efforts on legislative data standards, and some of the advances that have taken place in the development of these standards over the last year.

Big Idea: Legislative Data Standards

And now to the big idea: to represent all the world’s legislation in a standard structured data format. That’s actually two big ideas: (1) putting legislation into a structured data format, and (2) designing that format so that it is compatible with the wide variety of laws and legislative document types worldwide.

There are reasons for doing these things: First, introducing structured data to legislation can make it possible to search and analyze the law with greater precision and efficiency. And second, having a common standard can permit more comprehensive bill-tracking and comparison between jurisdictions.

California Bill with Metadata

It also can make it possible for legislatures with small (and shrinking) budgets to benefit from some of the same bill drafting software that is being developed for much larger jurisdictions. (Full disclosure: Xcential has developed such software for more than ten years, including the drafting platform used by the State of California.)

In the age of Google, these ideas may not seem so big; in fact, they are a subset of Google’s far-reaching mission. However, legislation is a corner of the world’s information that Google has not yet addressed in a systematic way. And as regular readers of this blog know, legislation presents its own hurdles, technical and bureaucratic (not necessarily in that order), that make this both an interesting and a challenging problem. One of the challenges is that the kind of people who generally work with data (we’ll call them engineers) and the kind of people who generally work with legislation (we’ll call them lawyers and policy folks) don’t often work on data and legislation together. One of us, a lawyer and policy type, has made this point graphically (and somewhat hyperbolically) in a Quora response to a question about whether version control software could be used for legislation. That question, and a subsequent discussion generated in response to a blogpost by software engineer Abe Voelker about version control for legislation, drew in many engineers and some lawyers and policy folks.

For software engineers who consider such things, it is very attractive to think about treating legal text as if it were software code; we could automatically highlight and cross-validate key terms, run test cases, automate redlining and version control, etc. It would be easy to see what the state of the law was at any particular point in time, and to trace the series of amendments that got us into the mess we’re in today. This desire is often expressed as “What if we had a Github for legislation?” On the other hand, people who work closely with legislation–researching it, drafting it or developing information systems to deal with it–tend to see the many places that the analogy between computer code and legal code break down. Legal texts have been shaped over hundreds of years by technologically conservative institutions, using print-based systems.

The full transformation of law to digital information is not going to happen overnight. While most law is already accessible in electronic format (often pdf), it is not encoded in a way that software engineers could start using their favorite text-munching tools. One of us, an engineer, has described this as the difference between computerization and automation. The move toward better digital tools for automating legislative drafting and research tasks will require more dialogue and working exchanges between engineers and the lawyers and policy folks.

That brings us back to hackathons.

What is a Legislative Hackathon?

Recognizing the need to bring lawyers and engineers together in order to implement our big idea(s), and appreciating the valuable bandwagon that hackathons have become, we decided to jump onboard. The first event we organized, the California Law Hackathon, was hosted just over a year ago, in September 2011, in Berkeley at the offices of Maplight, and in Denver by Karen Suhaka’s team at BillTrack50. The event focused on building web-based visualization tools to track the timeline of amendments to California legislation, and to link particular amendments, through their legislative sponsors, to particular donors or interest groups. We were joined remotely by a number of international participants, including John Sheridan, head of e-services for legislation.gov.uk, and a fellow guest contributor to this blog. As one participant noted, we learned a great deal at the event, including the limits placed on us by the existing data. Neither the legislative record, nor the donations databases are detailed enough to trace influence in politics in the way we hoped. This helped spark an interest in a more in-depth exploration of legislative data formats, and in particular how more and better metadata could be added to legislation.

That led to the International Legislation Un-hackathon, held simultaneously at UC Hastings, Stanford and Denver, with participants from the University of Bologna (Ravenna campus) and around the world. So assuming you can get engineers together with lawyers and policy folks, what do you do with them? We decided that we’d need a user-friendly tool that could be used to explore and add metadata to legislation from around the world. This could highlight a developing legislative XML standard, Akoma Ntoso (more about this standard soon), and give hands-on experience to lawyer and policy types kinds of text and analysis tools that engineers take for granted.

Hacking With A Legislative Editor

So one of us (the engineer, naturally) started building a web-based editor for legislation, while the other (the lawyer, naturally) started organizing the next hackathon. Of course, thought the lawyer, it would just

Legislative Editor at legalhacks.org

be a matter of time before all governments worldwide use such editors to draft their laws and regulations in a standard data format.

Advances in Legislative Data Standards Efforts

Akoma Ntoso

Akoma Ntoso (AkN) is a strong contender to be that format. Developed under the auspices of the UN Department of Economic and Social Affairs, AkN is an XML data structure that is meant to capture high-level forms and semantic ideas that are common to a broad variety of legal texts. OASIS, the folks who brought us the DocBook standards, among others, have convened a standards committee to create an official legislative data standard based on AkN. (More disclosure: the engineer is a member of this committee.) There’s just one problem. Few governments are using AkN to draft or store their legislation.

AkN itself is fast evolving, and with more exposure to legal data structures from different jurisdictions, the OASIS committee will be able to adapt AkN to better model those structures.

We saw the International Legislative Un-Hackathon as a venue to kick off this process. It was conceived with Charles Belle of UC Hastings, as part of the Legal Hacks initiative. The event was held simultaneously at UC Hastings, Stanford, in Denver. Jim Harper and Francis Avila of the Cato Institute came to the Hastings Event. We also had many international participants. Key among them were Professors Monica Palmirani and Fabio Vitali of the University of Bologna, the architects and primary evangelists of AkN. Over the course of the day, participants learned about AkN and, importantly, got a chance to try it out, marking up documents of their choosing with the web editor. In the process, as expected, we found bugs in the software and bugs in the standard. We found structures in U.S. legislation that didn’t fit well with the existing AkN element set. We saw places where there was confusion in applying AkN’s data structures to documents. All of this information was collected to incorporate in the development of both the editor and AkN, underscoring again the importance of getting more practical exposure for both.

University of Bologna Summer School–Ravenna

And we are working to expand the venues for this kind of practical exposure to develop the AkN standard. Every September, the University of Bologna hosts the LEX Summer School in Ravenna, Italy. For them, it’s an opportunity to introduce Akoma Ntoso to new groups of students from around Europe and around the world. For the students, it’s an opportunity to learn about the application of XML to legislation, see the success various groups are having around the world, and to meet interesting new people having a passion for legal informatics. One of us, the engineer, who was a student two years ago, was invited to return last year to present a success story, and this year is returning once more to deliver a class in how to build and use the HTML5-based editor for drafting legislation in XML. For us, this is an opportunity to expose the editor to the European legal traditions in order for us to better understand how our editor must evolve to fulfill our vision of a unified standard around the world with common, highly adaptable, tools.

Chile National Library of Congress Browser-based editor

Another step toward adoption of legislative data standards is a project by Chile’s National Library of Congress (BCN in Spanish) called the “History of the Law” (Historia de la Ley). This ambitious project aims to bring together machine learning, a legislative editor and other features to mark up Chile’s legislative record and other legislative documents. The BCN has chosen Xcential’s browser-based editor, working with the AkN standard, to conduct the mark-up and correction after documents are passed through an automated parser. As with the hackathon, but on a larger scale, we are learning from experience the modifications that are needed to AkN, to make it work with Chile’s live documents. Excitingly, each mismatch we find between AkN and actual legislation can be fed back into the OASIS committee process, to make AkN able to handle a wider variety of real-world use cases.

Other Efforts and the Future of Legislative Data Standards

We see these steps as just the beginning. European governments are also flirting with legislative standards, and Karen Suhaka’s group at BillTrack50 has converted all U.S. bills from all states into a single standard XML format showing that the technical hurdles can be overcome, and many of the practical benefits of doing so. In focusing on the projects (and hackathons) we are most closely involved with, we have certainly left out a lot of the initiatives that are advancing legislative data standards around the world. That’s what the comments are for. Let us know your experience with Akoma Ntosa as a legislative standard, and what you’re doing or interested in doing with AkN or other legislative data standards worldwide.

Grant Vergottini (the engineer) is a founder of Xcential. He is a leading authority on applications of XML data to legislation. Prior to founding Xcential, Grant was the Director of Applications at Chrystal Software, a company dedicated to XML design and reporting software. Before Chrystal, Grant led the redesign of Homestore.com, and founded Genedax Design Automation, which developed innovative team and data management applications for electronics design. Bringing data structures and automation tools to the legislative drafting process parallels the work that Grant did earlier in his career at Mentor Graphics and the Boeing Company, where he participated in the transformation from manual drafting to CAD software. Mr. Vergottini holds a Bachelor of Science in Electrical Engineering from Cleveland State University, where he graduated Summa Cum Laude.

Ari Hershowitz (the lawyer) is a consultant at Xcential, and founder of Tabulaw. Tabulaw develops software for lawyers, including a web-based legal research and writing platform. Prior to Tabulaw, Ari worked to protect wildlife and habitats from Chile to Mexico as Director of the BioGems project for Latin America at the Natural Resources Defense Council. Ari has a law degree from Georgetown University Law Center, a Masters in Computation and Neural Systems from Caltech, and a Bachelors in Molecular Biophysics & Biochemistry from Yale College.

VoxPopuLII is edited by Judith Pratt. Editors-in-Chief are Stephanie Davidson and Christine Kirchberger, to whom queries should be directed.

[Editor’s Note: For topic-related VoxPopuLII posts please see: Núria Casellas, Semantic Enhancement of legal information … Are we up for the challenge?; João Lima, et.al, LexML Brazil Project; and Rinke Hoekstra, The MetaLex Document Server

The Uniform Electronic Legal Material Act Is Ready for Legislative Action

authentication, Preservation of electronic legal information, Preservation of legal information, Public access to legal information, Standards 6 Responses »

Oct 152011

The Uniform Electronic Legal Material Act, referred to as UELMA, is ready for introduction into state legislatures. It has undergone its final proofing and formatting process by the National Conference of Commissioners of Uniform State Laws (NCCUSL, or ULC) and has been posted on NCCUSL’s archival Website at the University of Pennsylvania, and is soon to come to NCCUSL’s official site. The Act will be sent to the American Bar Association’s (ABA’s) House of Delegates for approval at the ABA Midyear Meeting in February, 2012 in New Orleans.

The UELMA addresses important issues in information management, providing sound guidance to states that are transitioning legal publications to digital formats. The Act is citizen-oriented, and leaves all issues concerning commercial publishing to state policy and contract law. Most importantly, the Act is outcomes-based, keeping it flexible in the face of changing technologies and evolving state practice. A brief account of UELMA’s development and its main provisions is included in this posting.

The UELMA was drafted in response to a request from the American Association of Law Libraries (AALL), following the AALL’s 2007 National Summit on Authentication of Digital Legal Information. The purpose of the Summit was to bring national attention to the issues surrounding the rapid rise in the number of states publishing primary legal information resources electronically and, in some cases, cancelling print resources and publishing legal information only in electronic format. Foremost among the issues were ensuring the trustworthiness of online legal resources and preserving the electronic publications to provide for continuing accessibility. The drafting of a uniform act on these topics was one of the top recommendations of the Summit’s attendees.

The ULC agreed to consider the development of a uniform law and appointed a Study Committee for that purpose. The Study Committee recommended that a law be developed and a Drafting Committee was charged with the task. After two years of consideration, including several face-to-face meetings, conference calls, and circulation of numerous drafts by email, the UELMA was read to and debated for the second time at the Annual Meeting of NCCUSL in July 2011. After more than six hours of floor consideration, the NCCUSL Committee of the Whole passed the draft act, sending it to a Vote of the States. UELMA passed its final hurdle with a positive Vote of the States, gaining approval by a vote of 45-0 (with 1 abstention and 7 jurisdictions not voting).

The UELMA, as it passed the Conference, requires a state that publishes official versions of its legal information in electronic format to do three things:

1. Authenticate the information, by providing a method to determine that the legal material is unaltered from the version published by the state officer or employee that publishes the material;
2. Preserve the information; and
3. Ensure public accessibility on a permanent basis.

At a minimum, legal material that is covered by the Act includes the most basic of state-level legal information resources, including the state constitution, session laws, codified laws or statutes, and state agency rules with the effect of law. In recognition of potential separation of powers issues, the UELMA does not automatically include judicial or executive materials, leaving it to the enacting state to decide whether and how to include those resources. States may choose to include court rules and decisions, state administrative agency decisions, executive official documents, or almost any other information resources they designate as legal material.

For each type of legal material, the state must name a state agency or official as the “official publisher.” The official publisher has the responsibility to authenticate, preserve, and provide access to the legal material. If legal material defined by the Act is published only electronically, that material must be designated “official” and meet the requirements of the Act. If there is a print version of the legal material, an official publisher may designate the online version “official,” but the requirements of the Act to authenticate, preserve, and provide access must be met for the electronic version.

The requirements of the Act are not ended if the official electronic legal material is superseded, overruled, or otherwise ceases to be current law. Legal material does not lose its value even if it is no longer in effect. Accordingly, once a source is designated as official, it continues to be covered by the provisions of the UELMA. Historical sources must be preserved and made available.

The Act does not affect any relationships between an official state publisher and a commercial publisher, leaving those relationships to contract law. Copyright laws are unaffected by the Act. The Act does not affect the rules of evidence; judges continue to make decisions about the admissibility of electronic evidence in their courtrooms.

The comments to the UELMA provide a great deal of background on the decisions and intent of the Drafting Committee. In many instances, the comments offer guidance to legislators who will be asked to consider the UELMA for passage. The comments are included with the Act on the University of Pennsylvania’s Biddle Law Library Website.

Some issues specific to one of the three parts of the Act (authentication, preservation, and public access) are as follows. More information on these points can be found in the comments to the Act.

Authentication (Sections 5 and 6) :

The Drafting Committee considered a wide range of approaches to authentication before settling on a policy of presenting a technology-neutral, outcomes-based document, leaving the choice of method used to authenticate legal material up to the states. This approach also leaves it to each state’s discretion to change methods, as necessary or desirable. What is required is that the official publisher provides a method for the user to determine that the electronic record is unaltered from the one published by the official publisher.

By the terms of the Act, the authenticated electronic legal material will receive a presumption of accuracy, the same presumption that is created by publication of legal material in print form. The burden of proving inaccuracy shifts to the party that disputes the accuracy of the electronic legal material. Electronic legal material from other states with substantially similar laws will receive the same presumption of accuracy.

Preservation (Section 7):

The Drafting Committee spent considerable time debating the preservation provisions. The biggest issues were finding a way to describe what legal material would be covered by a preservation requirement, and how legal material should be preserved.

The Drafting Committee decided that, ultimately, all legal material covered by the Act’s authentication provisions should also be subject to its preservation requirements. This was stated simply as requiring preservation of legal material “that is or was designated as official” under the Act. This language requires that states preserve superseded or amended legal material, which retains importance despite its no longer being currently effective. The comments to Section 7 make clear that the Drafting Committee intended the Act to cover not only the text of the law, but also the materials commonly published with the legal material. This would mean that the lists of legislators and state officials typically published with session laws would be preserved, as would proposed or final state constitutional amendments, legislative resolutions, and any other type of information published with a legal material source.

The Drafting Committee decided to use an outcomes-based approach for the preservation requirements, similar to its approach to authentication. The ultimate outcome of preservation is that legal material may be preserved in an electronic format, in print, or by whatever method the state may choose in the future; consistent with an outcomes-based approach, state policy and preference dictate the preservation method.

If legal material is preserved electronically, the UELMA requires that the integrity of the record be ensured, including through backup and disaster recovery preparations, and that the continuing usability of the legal material is ensured. Recent natural disasters in the U.S. have highlighted the importance of disaster recovery preparations. Further, information that is preserved in an unusable format is of no value. The comments make clear that migration to new formats or storage media will be required from time to time.

The comments also note that the Drafting Committee intended that legally significant formatting be preserved. The complexity of presentation of some legal materials — evident in indentations, italicization, and numbering of internal subdivisions, for example — may indicate or explain legislative or regulatory intent. Preservation should not change the meaning of the legal material, but rather should ensure that the legal material is capable of being authenticated.

The Act recognizes that states have decades, and in some instances centuries, of expertise in preserving print materials, and does not specify preservation requirements or outcomes if the state chooses to preserve legal material in print. Nor does the Act impose a duty on an enacting state to retrospectively convert its print material to an electronic format. If, however, the state chooses to digitize previously non-electronic legal material, and if that newly electronic legal material is designated as official, then the requirements of the Act must be met. Publication of legal material in an official electronic version subsequent to the adoption of the UELMA, even if the same legal material was published previously in print, triggers the requirements of the Act.

Permanent Access (Section 8):

Citizens must be informed as to government actions if they are to participate effectively in their government. Legal material is an essential information source for citizens to access to become informed. The UELMA recognizes this in requiring reasonable availability, on a permanent basis, of legal material, even that which is amended, repealed, or superseded.

The Drafting Committee debated conditions of access over several meetings, finally concluding that states already have long-term, relevant experience in making other materials available through archives, libraries, and state offices. The enacting state has discretion to decide where, when, and how to provide access, including whether to charge fees for access. Section 8’s requirement of permanent access does not require a state to provide unlimited access to its preserved legal information. This drafting decision is consistent with the rest of the UELMA, which defers to state policy and practice in its other provisions. Eventually, the Committee decided that the individual states could set their own requirements for access to legal material preserved under the Act, as long as the access is reasonable and in perpetuity. For this reason, the Act does not address whether states can charge fees for access to preserved electronic legal material.

The Standards section of the Act (Section 9) directs official publishers of electronic legal material to consider developing standards and best practices as they choose and to implement methods for the authentication, preservation, and permanent access of electronic records. The “Guiding Principles to Be Considered in Developing a Future Instrument,” the best practices document of the Hague Conference on Private International Law, were important guidelines that were repeatedly consulted in the drafting process.

Throughout its deliberations, the Drafting Committee was advised and informed by a large number of advisors and observers who came from federal and state governments, commercial legal publishers and software vendors, and a number of interested organizations. Two American Bar Association advisors brought knowledge of and experience with technologies to the drafting process. The observers were very helpful in assisting the Committee in its understanding of the possible impacts of proposed sections of the Act. In some instances, the observers were able to explain existing and emerging technologies that might be used to accomplish the Act’s specified outcomes. The Committee watched technology demonstrations and investigated various authentication processes already in effect. The drafting process was strengthened by the level of support and expertise the advisors and observers brought, but, in the end, the Act was entirely the Committee’s work.

By designating the Committee’s product a uniform law, the ULC recognized the importance of the topic and urged wide adoption of the Act. The final step in the UELMA’s development will be its introduction into state legislatures. Bill sponsors are being identified, and the ULC anticipates the UELMA will be introduced in at least 8 states in January 2012, with the possibility of introduction in as many as 12.

The ULC has appointed an Enactment Committee for the UELMA to assist the larger ULC Legislative Committee with its charge to “endeavor to secure the enactment of [uniform] legislation.” The Enactment Committee prepares “talking points” and summaries of the legislation, and works with individual legislatures, on occasion, to answer questions and further the introduction and approval of the Act. Volunteers from several interested associations are also preparing to work towards the Act’s approval. With strong support from the ULC and volunteers working on its behalf, by next summer the Uniform Act may itself become “legal material” in one or more states.

Barbara Bintliff is the Joseph C. Hutcheson Professor in Law at The University of Texas School of Law, and Director of Research at the School’s Tarlton Law Library and Jamail Center for Legal Research. She is The Reporter for The Uniform Electronic Legal Material Act.

VoxPopuLII is edited by Judith Pratt. Editor-in-Chief is Robert Richards, to whom queries should be directed. The statements above are not legal advice or legal representation. If you require legal advice, consult a lawyer. Find a lawyer in the Cornell LII Lawyer Directory.

The Impact of Metadata Standards on Traditional Legal Online Services in Germany

Legal metadata, Public access to legal information, Standards 2 Responses »

Jun 152010

It is common sense within the information industry that revenue will shift from print to online (see Ulrich Hermann, CEO of Wolters Kluwer Germany, in FAZ on April 7th 2010, p. 15).

But what is the impact of current technical trends in metadata standards on legal online services in Germany? Is there any impact? If you take the real market penetration of metadata standards into account, one could say that the concept of metadata standards has failed. At least, standards such as the “Saarbrücker Standard” (a standard, created in 2000, for court decisions in German jurisdictions) have never been used widely. Nevertheless we kicked off jurMeta — a proposed new metadata standard for German-language legal resources — at the EDV-Gerichtstag 2009, a major German conference on judicial information systems.

Market trends in Germany

In order to understand the impact of metadata standards, we need to consider such standards in the context of legal online services in Germany, and current trends.

What are the market trends in online services in Germany? The content coverage is rapidly growing, particularly from an end-user’s perspective. State authorities and publishing houses are publishing primary content free of charge for various reasons; law firms are running blogs with comments on the latest court cases (JuraBlogs); lawyers and clients are using services that publish advice, even when related to concrete disputes (frag-einen-anwalt.de). European legislation (in particular the Public Sector Information Directive) is encouraging member states and public sector bodies to take proactive measures to promote reuse of public sector information in order to exploit its business potential.

At the same time the technical infrastructure for sharing information is getting better. Interested persons do not need any technical skills to instantly set up a vertical search engine with Google custom search. Open source tools are even covering high end demands of private online publishers. There is an open source tool or project that addresses the private publisher’s needs in any aspect, e.g. performance (CouchDB), machine learning (Mahout), or relevance ranking (Open Relevance Project). And in most cases it is not only a solution but a high end solution that can be adopted free of (licensing) cost.

Of course social networks have to be named as a trend in law online. Services such as Marktplatz-Recht.de, JUSMEUM, and justanswer are creating communities of legal professionals. These services combine the benefits of online communities with expertise in the area of law and thus generate new revenue streams. Social networks organize value-adding-processes efficiently since they allow one to exploit cognition [i.e., the people having the most knowledge] at the source. (I once called this the paradigm of the “cheapest value adder”; see Gütezeichen als Qualitätsaussage im digitalen Informationsmarkt: dargestellt am Beispiel elektronischer Rechtsdatenbanken (2000), p. 25.) Chances are high that vertical social networks will repeat the success of their generic predecessors, at least at a smaller scale.

Do legal online services exhibit these features only in Germany? The answer is No. These market trends are most likely valid throughout the world. I will come to the specifics of legal online services in Germany shortly.

First let me sum up what I consider to be the major driver in legal online services for the next couple of years. This is what I would like to label “Digital Convergence”–not the above-mentioned individual trends in the areas of content, technical infrastructure, and users/community, but a synergetic combination of these three trends that will drive the future of legal online services.

What are the characteristics of legal online services in Germany in 2010?

First, the fact that Germany is a code-based country makes legal online services in Germany an ideal target for any kind of invention in the area of text retrieval. For example, at the federal level there are more than 5,000 pieces of legislation with more than 100 provisions each. If you think of one specific term in paragraph 4 of § 97 GWB as a very precise reference to a specific legal issue, consider that

§ 97 GWB is one out of more than 500,000 provisions,
these provisions follow a semantic structure, and
nearly all existing legal documents refer to specific provisions at the European, federal, state, or municipal level.

You thus get a sense of the challenge of legal online services in Germany. But you also get a sense of their potential. One could put it this way: The semantic web of legal content in Germany already exists; we “only” have to apply a common syntactical representation in order to create a very rich business resource by linking millions of individual and heterogeneous documents. The projects that share a common syntactical representation (e.g., jurMeta) will be rich sources for additional value-adding processes such as data mining.

Second, end-users of legal documents are by nature very conscious about retrieval quality. No matter whether they are aware of the parameters for retrieval quality such as scope, recall and precision, end users (e.g., lawyers) know that the one make-or-break case could exist; therefore the goal of any kind of information retrieval effort is to retrieve this one case but not others. To measure the relevance of retrieval results is very difficult, but determining relevance in law is far easier than in other domains. The importance of retrieval quality to end users of legal information systems is of course not specific to Germany, but is still an important driver for resource allocation in online retrieval projects. Therefore, legal research is the ideal test area for technical innovations in search and retrieval. Taking into account that lawyers are an attractive target group, investors will to a larger degree focus on legal research as a business opportunity. Such investment should speed progress.

Third, an important parameter for legal online services in Germany is the availability of primary legal content (legislation and case law). Public authorities in Germany claim copyright — at least to some extent — in official documents, and their efforts at publishing primary legal content online have been rudimentary. The existing offerings of legal content set up by state authorities are end-user oriented and very heterogeneous. The service providers — such as this firm — that technically publish the legal documents on behalf of the public authorities are private companies with business interests relating to legal information. Thus, allowing these companies to publish primary legal content on behalf of the public authorities is like letting the fox rule the henhouse. After all, official documents are the result of the tax-funded work of public authorities.

A modern publication infrastructure for primary legal content should functionally separate data collection from dissemination. Official documents should, if required, be anonymized, and stored on servers in a well-structured format for anybody to download either free-of-charge, or at cost of dissemination. This would allow non-commercial projects as well as commercial users to focus on value-adding processes, rather than crawling and re-engineering data that already exists as part of proprietary collections. In economic terms, this would lead to improved resource allocation, strengthen electronic media as tools for democratic processes, and support the goals of the Public Sector Information Directive.

Legal online research in Germany in 2020

In order to analyze the impact of metadata standards on legal publishers, I will be so bold as to predict how legal online services could look in 2020:

a) Public authorities will be obliged to publish any official document electronically in a well-structured format on servers accessible for anybody, for commercial or non-commercial use, at cost of dissemination.
b) There will be a wide range of legal online information services free of charge serving all sorts of information needs and target groups. These will range from easy-to-use systems for one-time users to expert systems for professionals.
c) The motivations for setting up legal online information services will vary to a larger extent than they do today. Successful online projects will have to support this growing range of motivations. (See Felix Zimmermann, JurMeta: A New Metadata Initiative for Legal Documents.) Since setting up legal online services will even be easier in 2020 than in 2010, more people will set up such services.
d) If one measures retrieval quality by recall and precision, I bet nearly all free services in 2020 will be better than the existing commercial legal online services in Germany in 2010. I invite anybody to come up with a proposal for a concrete experimental design with which to test this thesis, and I am curious to see the results of the experiment that proves this thesis wrong. But in any case:
e) The beneficiary of these trends will be the end user of legal information.

Impact on fee-based services and traditional publishing houses

Assuming that legal information services will radically improve in the future, is there a business opportunity for traditional publishing houses in legal online services at all? If the answer is yes, what will be the parameters for (business) success?

Applying a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis will identify the threats and opportunities for publishing houses. An important threat is the increasing speed of activity and change in the market. The technical lead-time of open-source-based start-ups might be too large for traditional publishers. An acquisition could help a traditional publisher catch up, but only if the acquirer is willing to adjust its processes and business principles. All traditional publishing houses I know claim to be customer oriented. I am convinced that market pressure (i.e., competition) will increase because of the numerous new legal online services being launched over the next couple of years. A substantial opportunity for traditional publishing houses lies in increasing the benefits that users derive from legal information, rather than trying to convert book selling to online media.

The biggest advantage of established legal publishing houses, as opposed to new start-ups, is their brand recognition, and the chance to position themselves as gatekeepers for high quality information. The corresponding threat is –- again –- based on new technologies: Services such as UserVoice allow new content providers to gain user feedback more effectively than through traditional market research, and at very low cost. Online social networks are analyzing the interactions among users, and are trying to exploit users’ behavior and ratings on the fly for business purposes. We will see whether traditional publishing houses will timely make use of such technologies, or whether such publishers will be able to catch up if they miss the chance to apply these new tools.

Conclusion

Assuming that, as mentioned above, a combination of trends will drive the future of legal online services, the publisher that identifies the synergies of the “digital convergence” and first applies them in customer products has the best chance to benefit from the current trends. Those that move too late might lose even more revenue than currently expected due to the change from print to online information.

So what about the impact of metadata standards such as jurMeta on legal online services in Germany? Metadata standards are an important component of the concept of the above-mentioned “digital convergence.” They are the syntactical bridge from the raw materials of German legal documents to the Semantic Web of legal information in Germany. Metadata standards for legal information services support decentralized value-adding processes, and thus are the key parameter for exploiting the synergies enabled by “digital convergence.” The metadata standards that best serve the various needs of the greatest number of legal online projects and services, will succeed.

Dr. Andreas Bock is CEO of kjur.de. After writing his Ph.D. thesis on the quality of digital information services (Gütezeichen als Qualitätsaussage im digitalen Informationsmarkt: dargestellt am Beispiel elektronischer Rechtsdatenbanken (2000)) at the Institute for Legal Informatics of the University of Hannover with Prof. Dr. Wolfgang Kilian, he worked for Westlaw (2000 to 2003) and LexisNexis (2003 to 2007), heading product development of their respective international legal online projects in Germany. Dr. Bock currently practices law at Kehr-Ritz. Together with Felix Zimmermann, he launched kjur.de, an online service for free legal content, in 2009.

VoxPopuLII is edited by Judith Pratt. Editor in chief is Robert Richards.

Environmentally-Friendly Citations

commercial systems, Legal citation, Legal citations, Legal descriptive metadata, Legal informatics, Legal knowledge representation, Legal metadata, legal research, Standards 9 Responses »

Mar 012010

Today in Canada, nearly three quarters of citations to recent case law use the neutral citation – an industry-independent, open identifier assigned by courts to their decisions. When we call something a “game-changer” most people assume that it was invented by Apple. Yet even though the neutral citation was not, it definitely is a game-changer in the legal publishing business. Here are some thoughts about why.

Cited and Citing Cases

Legal publishing would be much simpler if cases did not cite other cases or all sorts of other legal documents. However, in that event the law would be far less intelligible. The citations between legal documents help establish a coherent body of law. The interpretation of cases and statutes in their surrounding context of citing and cited legal documents is crucial in legal practice. It is often considered prudent to wait until the courts have their say on a freshly enacted statute before relying blindly on it. And no lawyer would bring up a “dynamite” case in court before carefully checking to see how this case has been treated by other case law.

Indeed, all legal publishers try in various ways to exploit the relations between legal documents in order to stand out in the eyes of their customers. Many features of the electronic publishing systems are based in some way on the relations between documents – hyperlinking, note-up, lists of cases and statutes considered, related cases, judicial history and treatment, search results ranking based on popularity, etc. The use of citation data has defined legal publishing for many years. Any major change in how things are cited in law will continue to have a great impact on legal publishing in the future.

Originally, citation data was neither intended nor designed to yield itself easily to computer processing, let alone free online publishing.

The Chinese Walls Around Print Report Series

The problem is well known. Print report citations were not designed to function outside the context of the report series they belong to. For example, “301 D.L.R. (4th) 513” does not mean anything to you if you don’t have the Dominion Law Reports nearby. Even if you were lucky enough to have the series in your firm’s library, you could not safely cite a case by, for example, 301 D.L.R. (4th) 513 and expect all your readers to understand what you are saying unless you assume that all your readers have the Dominion Law Reports in their libraries. This issue is amplified by the number of print law reports. In Canada, there are 70 major law report series according to the Queen’s University Law Library. (Although some from the list may have disappeared since the list was last updated, the number of report series remains large.)

To cope with this reality, the legal publishing world came up with citators, and in particular, their ability to offer and make use of parallel citations. Citators can tell you, among other things, what possible identifiers (citations) have been assigned to a particular case. For example, the following list – [2009] 1 S.C.R. 181; 301 D.L.R. (4th) 513; [2009] 2 W.W.R. 385; 183 C.R.R. (2d) 1; 320 Sask. R. 305 – means that the case Ravndahl v. Saskatchewan can possibly be identified by any the citations included in the list.

Commercial Electronic Databases Are Not Better

Nobody will dispute the fact that, for all practical purposes, electronic sources are the research tool. Printed reports will wither and gradually disappear. Those that remain because of their official status will be used, not for research, but only as the recognized source of citable law. Many if not most legal researchers will even affirm that this is already the de facto situation.

It is worthwhile to analyze what will happen in that new electronic context. Citations will be based on database identifiers. In Canada, such citations will take the following form: “[1998] O.J. No. 2515 (QL)”. In some ways, such a citation is leading to the same old problems discussed earlier in this post: those of proprietary citations. However, if the inconvenience of having to check in a specific book to know what was cited was annoying, citations consisting of commercial database identifiers create a much more serious problem. To get the cited material, whereas in the print environment the researchers had to take the time to visit the library, in the digital environment they must subscribe to the commercial database. In the era of the Internet, any type of proprietary citations could seriously threaten the legal information system.

The Free Law Publisher’s Initial Approach

One way of dealing with Chinese Walls is to live with them, and use one’s wits to figure out what is on the other side.

In 2004, CanLII was striving to be recognized as a legal information product that could successfully serve the everyday research needs of legal professionals. The idea emerged to develop a citator in order to improve hyperlinking — and a series of other cool features — using the relations between legal documents.

Building a citator is an expensive operation. Here is a brief outline of the manual and automated methodology mix that was employed to build Reflex, CanLII’s citator.

a law library

1. An editor keys in information about all cases published in a particular report series, for example, the Dominion Law Reports. Such information includes the case name, the docket number, the issuing court, the date of the decision, a very short excerpt from the case, and the report citation.

2. This operation results in many records like the following one:

Record 1
Case name: Ravndahl v. Saskatchewan
Docket: 32225
Date: 2009-01-29
Court: SCC
Excerpt: The appellant lost…
Citation: 301 D.L.R. (4th) 513

3. Another editor keys in the same information about all cases published in another report series, for example, the Western Weekly Reports, producing records like this one:

Record 2
Case name: Ravndahl v. Saskatchewan
Docket: 32225
Date: 2009-01-29
Court: SCC
Excerpt: The appellant lost…
Citation: [2009] 2 W.W.R. 385

301 D.L.R. (4th) 513

[2009] 2 W.W.R. 385

5. The operation is repeated for 35 report series on an ongoing basis.

Of course, in practice the exercise is much more complicated, as the software has to deal with various degrees of similarities of metadata; for example, almost identical case names (R. v. Smith and The Queen v. Smith). The program can also encounter other misfortunes, such as the absence of docket numbers or dates.

With this approach, CanLII was able to expand significantly the breadth of hyperlinking within legal documents, and all features based on the hypertext, such as sorting search results by number of citations, providing lists of related cases, and a few more. Because Reflex is able to resolve citations to cases that are unavailable on CanLII, the number of citations was used also as an indicator as to what are the most important cases missing from CanLII, or, in other words, where to start from if we want to scan cases from paper and publish them on CanLII.

As one of the founding members of the free-access-to-law movement, CanLII may have revolutionized the way Canadian law was made accessible, but the print was still a ubiquitous part of our publishing routine.

Another Way of Dealing With the Chinese Walls…

… is to simply destroy them. Before CanLII’s arrival on the legal information playground, LexUM, in collaboration with representatives from the judiciary, law librarians, court staff, IT consultants, and several forward-looking individuals from the commercial publishing circles, had set up the Canadian Citation Committee. The CCC is an ad hoc group formed to support the standardization efforts of the Judges Technology Advisory Committee (JTAC) of the Canadian Judicial Council (CJC).

The CCC designed and promotes several documentary standards, among them the neutral citation. The neutral citation was proposed as a unique, industry-independent identifier, assigned to a case by the court. It is formed in a simple way: by the year of the case, an acronym for the issuing court, and a serial number. For example, 2009 SCC 7 designates the case of the Supreme Court of Canada Ravndahl v. Saskatchewan, released in 2009.

Simple, open, Internet-friendly, environmentally-caring, promising: such is the neutral citation.

Who’s on Board?

Courts have gradually been adopting the neutral citation, beginning in 1999 and continuing to the present. The first adopter of the neutral citation was the Superior Courts of British Columbia. Today, all 50 Canadian courts follow the neutral citation standard. The last one to join — just this year, in fact — was the Ontario Superior Court of Justice – the toughest jurisdiction in the country (from a judicial administration and legal publishing point of view) because of the complexity of its judicial structure. As a result, all 50,000 cases issued annually by Canadian appellate, superior, and trial courts now bear neutral citations that have been assigned by the courts. To that number, we must add the decisions rendered by at least two dozen administrative tribunals which have also adopted the standard.

Probably a more important question than “Who’s on board?” is: Why are those institutions on board? Before embracing a change, people often need at least one ideological reason and at least one practical reason. On philosophical (and economic) grounds, it certainly made sense for court decisions to be freed from proprietary citation schemes. From a practical point of view, the most convincing argument was the convenience for the court to have a unique designation of its own decision at the very moment the reasons of the decision are issued.

Are Lawyers and Judges Following?

If you read carefully the first paragraph of this post, you know that the answer is yes. Lawyers and judges do cite cases using the neutral citation. They use neutral citations much more frequently than one may think.

Let’s bring in some data. On CanLII, case citations are hyperlinked if the citation comes from one of the 35 reports covered by CanLII’s citator or if the citation is a neutral citation. This allows for a citation resolution success rate of about 80%. This means that 80% of case citations on CanLII are hyperlinked. The rest, many of which are citations to proprietary commercial databases, are not.

In this context, it was tempting to verify the portion of the links attributable to the neutral citation. Or in other words, what is the percentage of case citations that contain the neutral citation – alone or among other parallel citations?

So we examined two sets of citations. The first one contained 40,000 citations of cases released in 2006, 2007 and 2008. The second one included 41,000 citations of cases released in 2007, 2008 and 2009.

The count showed the following. In data set 1 (citations pointing to cases released in 2006, 2007 and 2008), 85% of hyperlinked citations are, or contain, a neutral citation. In data set 2 (citations to cases released in 2007, 2008 and 2009), the neutral citation accounts for 91% of the links.

Data Set #1
40,000 citations
Citing cases released in 2008
Cited cases released in 2006, 2007, 2008
Links based on neutral citations: 85%
Share of all citations that are or contain neutral citations: 68%

Data Set #2
41,000 citations
Citing cases released in 2009
Cited cases released in 2007, 2008, 2009
Links based on neutral citations: 91%
Share of all citations that are or contain neutral citations: 73%

Needless to say, both the numbers and the progression look exciting. This, of course, is not the last reason we need before sending the print reports sailing into history.

It is just one more.

Ivan Mokanov is Deputy Director of LexUM. He oversees LexUM’s publishing and development activities and supervises various consulting and research projects in Canada and abroad. As a member of LexUM’s Executive Committee, he participates in LexUM’s administration and business development. Ivan is a graduate from Sofia University (B.C.L.) and the University of Montreal (LL.M.), and he is currently enrolled at HEC Montreal (M.B.A).

Preserving Born-Digital Legal Materials…Where to Start?

Digital law libraries, Law librarians and legal informatics, open source software, Standards 9 Responses »

Jan 102010

It’s tempting to begin any discussion of digital preservation and law libraries with a mind-blowing statistic. Something to drive home the fact that the clearly-defined world of information we’ve known since the invention of movable type has evolved into an ephemeral world of bits and bytes, that it’s expanding at a rate that makes it nearly impossible to contain, and that now is the time to invest in digital preservation efforts.

But, at this point, that’s an argument that you and I have already heard. As we begin the second decade of the 21st century, we know with certainty that the digital world is ubiquitous because we ourselves are part of it. Ours is a world where items posted on blogs are cited in landmark court decisions, a former governor and vice-presidential candidate posts her resignation speech and policy positions to Facebook, and a busy 21st-century president is attached at the thumb to his Blackberry.

We have experienced an exhilarating renaissance in information, which, as many have asserted for more than a decade, is threatening to become a digital dark age due to technology obsolescence and other factors. There is no denying the urgent need for libraries to take on the task of preserving our digital heritage. Law libraries specifically have a critically important role to play in this undertaking. Access to legal and law-related information is a core underpinning of our democratic society. Every law librarian knows this to be true. (I believe it’s what drew us to the profession in the first place.)

Frankly speaking, our current digital preservation strategies and systems are imperfect – and they most likely will never be perfected. That’s because digital preservation is a field that will be in a constant state of change and flux for as long as technology continues to progress. Yet, tremendous strides have been made over the past decade to stave off the dreaded digital dark age, and libraries today have a number of viable tools, services, and best practices at our disposal for the preservation of digital content.

Law libraries and the preservation of born-digital content

In 2008, Dana Neacsu, a law librarian at Columbia University Law School, and I decided to explore the extent to which law libraries were actively involved in the preservation of born-digital legal materials. So, we conducted a survey of digital preservation activity and attitudes among state and academic law libraries.

We found an interesting incongruity among our respondent population of library directors who represented 21 law libraries: less than 7 percent of the digital preservation projects being planned or underway at our respondents’ libraries involved the preservation of born-digital materials. The remaining 93 percent involved the preservation of digital files created through the digitization of print or tangible originals. Yet, by a margin of 2 to 1, our respondents expressed that they believed born-digital materials to be in more urgent need of preservation than print materials.

This finding raises an interesting question: If law librarians (at least those represented among our respondents) believe born-digital materials to be in more urgent need of preservation, why were the majority of digital preservation resources being invested in the preservation of files resulting from digitization projects?

I speculate that part of the problem is that we often don’t know where to start when it comes to preserving born-digital content. What needs to be preserved? What systems and formats should we use? How will we pay for it?

What needs to be preserved? A few thoughts…

Determining what needs to be preserved is not as complicated as it may seem. The mechanisms for content selection and collection development that are already in place at most law libraries lend themselves nicely to prioritizing materials for digital preservation, as I have learned through the Georgetown Law Library’s involvement in The Chesapeake Project Legal Information Archive. A collaborative effort between Georgetown and partners at the State Law Libraries of Maryland and Virginia, The Chesapeake Project was established to preserve born-digital legal information published online and available via open-access URLs (as opposed to within subscription databases).

So, how did we approach selection for the digital archive? Within a broad, shared project collection scope (limited to materials that were law- or policy-related, digitally born, and published to the “free Web” per our Collection Plan) each library simply established its own digital archive selection priorities, based on its unique institutional mandates and the research needs of its users. Libraries have historically developed their various print collections in a similar manner.

The Maryland State Library focused on collecting documents relating to public-policy and legal issues affecting Maryland citizens. The Virginia State Library collected the online publications of the Supreme Court of Virginia and other entities within Virginia’s judicial branch of government. As an academic library, the Georgetown Law Library developed topical and thematic collection priorities based on research and educational areas of interest at the Georgetown University Law Center. (Previously, online materials selected for the Georgetown Law Library’s collection had been printed from the Web on acid-free paper, bound, cataloged, and shelved. Digital preservation offered an attractive alternative to this system.)

To build our topical digital archive collections, the Georgetown Law Library assembled a team of staff subject specialists to select content (akin to our collection development selection committee), and, to make things as simple as possible, submissions were made and managed using a Delicious bookmark account, which allowed our busy subject specialists to submit online content for preservation with only a few clicks.

As a research library, we preserved information published to the free Web under a claim of fair use. Permission from copyright holders was sought only for items published either outside of the U.S. or by for-profit entities. Taking our cues from the Internet Archive, we determined to respect the robots.txt protocol in our Web harvesting activities and provide rights holders with instructions for requesting the removal of their content from the archive.

Fear of duplicating efforts

We have, on occasion, knowingly added digital materials to our archive collection that were already within the purview of other digital preservation programs. There is a fear of duplicating efforts when it comes to digital preservation, but there is also a strong argument to be made for multiple, geographically dispersed entities maintaining duplicate preserved copies of important digital resources.

This philosophy, especially as relates to duplicating the digital-preservation efforts of the Government Printing Office, is currently being echoed among several Federal Depository Libraries (and prominently by librarians who contribute to the Free Government Information blog) who are supporting the concept of digital deposit to maintain a truly distributed Federal Depository Library Program. Should there ever be a catastrophic failure at GPO, or even a temporary loss of access (such as that caused by the PURL server crash last August), user access to government documents would remain uninterrupted, thanks to this distributed preservation network. Currently there are 156 academic law libraries listed as selective depositories on the Federal Depository Library Directory; each of these would be candidates for digital deposit should the program come to fruition.

Libraries with perpetual access or post-cancellation access agreements with publishers may also find it worthwhile to invest in digital preservation activities that may be redundant. Some publishers offer easy post-cancellation access to purchased digital content via nonprofit initiatives such as Portico and LOCKSS, both of which function as digital preservation systems. Other publishers, however, may simply provide subscribers with a set of CDs or DVDs containing their purchased subscription content. In these cases, it is worthwhile to actively preserve these files within a locally managed digital archive to ensure long-term accessibility for library patrons, rather than relegating these valuable digital files, stored on an unstable optical medium, to languishing on a shelf.

Law reviews and legal scholarship

It has been suggested that academic law libraries take responsibility for the preservation of digital content cited within their institutions’ law reviews to ensure that future researchers will able to reference source materials even if they are no longer available at the cited URLs. While there aren’t specific figures relating to the problem of citation link rot in law reviews, research on Web citations appearing in scientific journals has shown that roughly 10 percent of these citations become inactive within 15 months of the citing article’s publication. When it comes to Web-published law and policy information, our own Chesapeake Project evaluation efforts have found that about 14 percent, or 1 out of every 7, Web-based items had disappeared from their original URLs within two years of being archived.

In the near future, we may find ourselves in the position of taking responsibility for the digital preservation of our law reviews themselves, given the call to action in the Durham Statement on Open Access to Legal Scholarship. After all, if law schools end print publication of journals and commit “to keep the electronic versions available in stable, open, digital formats” within open-access online repositories, there is an implicit mandate to ensure that those repositories offer digital preservation functionality, or that a separate dark digital preservation system be used in conjunction with the repository, to ensure long-term access to the digital journal content. (It is important to note that digital repository software and services do not necessarily feature standard digital preservation functionality.)

Speaking of digital repositories, the responsibility for establishing and maintaining institutional repositories most certainly falls to the law library, as does the responsibility for preserving the digital intellectual output of their law schools’ faculty, institutes, centers, and students (many of whom go on to impressive heights).

At the Georgetown Law Library, we’ve also taken on the task of preserving the intellectual output published to the Law Center’s Web sites.

The Preserv project has compiled an impressive bibliography on digital preservation aimed specifically at preservation services for institutional repositories (but also covering many of the larger issues in digital preservation), which is worth reviewing.

What systems and formats should we use?

Did I mention that our current digital preservation strategies and systems are imperfect? Well, it’s true. That’s the bad news. No matter which system or service you chose, you will surely encounter occasional glitches, endure system updates and migrations, and be forced to revise your processes and workflows from time to time. This is a fledgling, evolving field, and it’s up to us to grow and evolve along with it.

But, take heart! The good news is that there are standards and best practices established to guide us in developing strategies and selecting digital preservation systems, and we have multiple options to choose from. The key to embarking on a digital preservation project is to be versed in the language and standards of digital preservation, and to know what your options are.

The language and standards of digital preservation

I have heard a very convincing argument against standards in digital preservation: Because digital preservation is a new, evolving field, complying with rigid standards can be detrimental to systems that require a certain amount of adaptability in the face of emerging technological challenges. While I agree with this argument, I also believe that it is tremendously useful for those of us who are librarians, as opposed to programmers or IT specialists, to have standards as a starting point from which to identify and evaluate our options in digital preservation software and services.

There are a number of standards to be aware of in digital preservation. Chief among these is the Open Archival Information System (OAIS) Reference Model, which provides the central framework for most work in digital preservation. A basic question to ask when evaluating a digital preservation system or service is, “Does this system conform to the OAIS model?” If not, consider that a red flag.

The Trustworthy Repositories Audit & Certification Criteria and Checklist, or TRAC, is a digital repository evaluation tool currently being incorporated into an international standard for auditing and certifying digital archives. A small number of large repositories have undergone (or are undergoing) TRAC audits, including E-Depot at the Koninklijke Bibliotheek (National Library of the Netherlands), LOCKSS, Portico, and HathiTrust. This number can be expected to increase in the coming years.

The TRAC checklist is also a helpful resource to consult in conducting your own independent evaluations. Last year, for example, the libraries participating in The Chesapeake Project commissioned the Center for Research Libraries to conduct an assessment (as opposed to a formal audit) of our OCLC digital archive system based on TRAC criteria, which provided useful information to strengthen the project.

The PREMIS Data Dictionary provides a core set of preservation metadata elements to support the long-term preservation and future renderability of digital objects stored within a preservation system. The PREMIS working group has created resources and tools to support PREMIS implementation, available via the Library of Congress’s Web site. It is useful to consult the data dictionary when establishing local policy, and to ask about PREMIS compatibility when evaluating digital preservation options.

While we’re on the exciting topic of metadata, the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH, not to be confused with OAIS), is another protocol to watch for, especially if discovery and access are key components of your preservation initiative. OAI-PMH is a framework for sharing metadata between various “silos” of content. Essentially, the metadata of an OAI-PMH compliant system could be shared with and made discoverable via a single, federated search interface, allowing users to search the contents of multiple, distributed digital archives at the same time.

For an easy-to-read overview of digital preservation practices and standards, I recommend Priscilla Caplan’s The Preservation of Digital Materials, which appeared in the Feb./March 2008 issue of Library Technology Reports. There are also a few good online glossaries available to help decipher digital preservation jargon: the California Digital Library Glossary, the Internet Archives’ Glossary of Web Archiving Terms, and the Digital Preservation Coalition’s Definitions and Concepts.

Open source formats and software

Open source and open standard formats and software play a vital role in the lifecycle management of digital content. In the context of digital preservation, open-source formats, which make their source code and specifications freely available, facilitate the future development of tools that can assist in the migration of files to new formats as technology progresses and older formats become obsolete. PDF, for example, although developed originally as a proprietary format by Adobe Systems, became a published open standard in 2008, meaning that developers will have a foundation for making these files accessible in the future.

Other open source formats commonly used in digital preservation include the TIFF format for digital images, the ARC or WARC file for Web archiving, and the Extensible Markup Language (XML) text format for encoding data or document structure information. Microsoft formats, such as Word Documents, do not comply with open standards; the proprietary nature of these formats will inhibit future access to these documents when these formats become obsolete. The Library of Congress has a useful Web site devoted to digital formats and sustainability (including moving image and sound formats), which is worth reviewing.

Open source software is also looked upon favorably in digital preservation because, similar to open source formats, the software development and design process is made transparent, allowing current and future developers to develop new interfaces to or updates to the software over time.

Open source does not necessarily mean free-of-charge, and in fact, many service providers utilize open source software and open standards in developing fee-based or subscription digital preservation solutions.

Digital preservation solutions

There are many factors to consider in selecting a digital preservation solution. What is the nature of the content being preserved, and can the system accommodate it? Is preservation the sole purpose of the system — so that the system need include only a dark archive — or is a user access interface also necessary? How much does the system cost, and what are the expected ongoing maintenance costs, both in terms of budget and staff time? Is the system scalable, and can it accommodate a growing amount of content over time? This list could go on…

Keep in mind that no system will perfectly accommodate your needs. (Have I mentioned that digital preservation systems will always be imperfect?) And there is no use in waiting for the “perfect system” to be developed. We must use what’s available today. In selecting a system, consider its adherence to digital preservation standards, the stability of the institution or organization providing the solution, and the extent to which the digital preservation system has been accepted and adopted by institutions and user communities.

In a perfect world, perhaps every law library would implement a free, build-it-yourself, OAIS-compliant, open-source digital preservation solution with a large and supportive user community, such as DSpace or Fedora. These systems put full control in the hands of the libraries, which are the true custodians of the preserved digital content. But, in practice, our law libraries often do not have the staff and technological expertise to build and maintain an in-house digital preservation system.

As a result, several reputable library vendors and nonprofit organizations have developed fee-based digital preservation solutions, often built using open-source software. The Internet Archive offers the Archive-It service for the preservation of Web sites. The Stanford University-based LOCKSS program provides a decentralized preservation infrastructure for Web-based and other types of digital content, and the MetaArchive Cooperative provides a preservation repository service using the open-source LOCKSS software. The Ex Libris Digital Preservation System and the collaborative HathiTrust repository both support the preservation of digital objects.

For The Chesapeake Project, the Georgetown, Maryland State, and Virginia State Law Libraries use OCLC systems: the Digital Archive for preservation, coupled with a hosted instance of CONTENTdm as an access interface.

In our experience, working with a vendor that hosted our content at a secure offsite location and managed system updates and migrations allowed us to focus our energies on the administrative and organizational aspects of the project, rather than the ongoing management of the system itself. We were able to develop shared project documentation, including preferred file format and metadata policies, and conduct regular project evaluations. Moreover, because our project was collaborative, it worked to our advantage to enlist a third party to store all three libraries’ content, rather than place the burden of hosting the project’s content upon one single institution. In short, working with a vendor can actually benefit your project.

The ultimate question: How will we pay for it?

We still seem to be in the midst of a global economic recession that has impacted university and library budgets. Yet, despite budget stagnation, there has been a steady increase in the production of digital content.

Digital preservation can be expensive, and law library staff members with digital preservation expertise are few. The logical solution to these issues of budget and staff limitations is to seek out opportunities for collaboration, which would allow for the sharing of costs, resources, and expertise among participating institutions.

Collaborative opportunities exist with the Library of Congress, which has created a network of more than 130 preservation partners throughout the U.S., and the law library community is also in the process of establishing its own collaborative digital archive, the Legal Information Archive, to be offered through the Legal Information Preservation Alliance, or LIPA.

During the 2009 AALL annual meeting, LIPA’s executive director announced that The Chesapeake Project had become a LIPA-sanctioned project under the umbrella of the new Legal Information Archive. As a collaborative project with expenses shared by three law libraries, The Chesapeake Project’s costs are currently quite low compared to other annual library expenditures, such as those for subscription databases. These annual costs will decrease as more law libraries join this initiative.

I firmly believe that law libraries must invest in digital preservation if we are to remain relevant and true to our purpose in the 21st century. The core reason libraries exist is to build collections, to make those collections accessible, to assist patrons in using our collections, and to preserve our collections forever. No other institution has been created to take on this responsibility. Digital preservation represents an opportunity in the digital age for law libraries to reclaim their traditional roles as stewards of information, and to ensure that our digital legal heritage will be available to legal scholars and the public well into the future.

Sarah Rhodes is the digital collections librarian at the Georgetown Law Library in Washington, D.C., and a project coordinator for The Chesapeake Project Legal Information Archive, a digital preservation initiative of the Georgetown Law Library in collaboration with the State Law Libraries of Maryland and Virginia.

VoxPopuLII is edited by Judith Pratt. Editor in Chief is Rob Richards.

Suffusion theme by Sayontan Sinha

VoxPopuLII