skip navigation
search

Remember Me

Work we admire No Responses »
Jul 062016

One of the great things that happens at the LII is working with the amazing students who come to study at Cornell — and finding out about the projects they’ve been cooking up while we weren’t distracting them by dangling shiny pieces of law before their eyes. This spring, Karthik Venkataramaiah, Vishal Kumkar, Shivananda Pujeri, and Mihir Shah — who previously worked with us on regulatory definition extraction and entity linking — invited us to attend a presentation they were giving at a conference of the American Society for Engineering Education: they had developed an app to assist dementia patients in interacting with their families.

The Remember Me app does a number of useful things — reminds patients to prepare for appointments, take medications, and so forth. But the remarkable idea is the way it would help dementia patients interact with people in their lives.

Here’s how it works: the app is installed on both the phone of the dementia sufferer and their loved ones and caregivers. When one of the people whom the dementia patient knows comes into proximity to the patient, the app automatically reminds the patient who the person is and how they know them by flashing up pictures designed to place the person in familiar context and remind the patient of their connection. Given the way that memory is always keyed to specific contexts, this helps patients stay grounded in relating to people whom they love but which their disease may hinder their recollection of.

One notable feature of the app is that it was designed not for a class in app development but in cloud computing, which means that the app can be used by a large number of people. The nature of the app also presented additional requirements: the team noted that “as our project is related to health domain, we need to be more careful with respect to cloud data security.” Further, although the students were software engineers who were tasked with developing a scalable application, their app reflects a thoughtful approach to developing a user experience that can benefit people with memory and other cognitive impairments. Associate Director Sara Frug says “among the many teams of talented M.Eng. students with whom we have worked over the years, Karthik, Vishal, Shivananda, and Mihir have shown a rare combination of skill and sophistication in software engineering, product design, and project management. Their app is a remarkable achievement, and we are proud to have seen its earliest stages of development.”

The Remember Me app has been developed as a prototype, with its first launch scheduled for August.

 

Posted by sara at 1:25 pm

American Pi

Uncategorized No Responses »
Mar 142016

In 2009, the US House of Representatives agreed to a resolution designating March 14 as “Pi Day”.

So I thought I’d see whether I could rustle up a nice example of pi in the Code of Federal Regulations. I did manage to find the π I was looking for, but it turns out that pi also gives us some examples of why we need to disambiguate.

In the process I also found:

So the story of “pi” in the CFR is a story of disambiguation. When we read (or our software parses) the original text of a particular regulation, it’s reasonably straightforward to tell which of the many “pi”s it’s discussing. We have capitalization, punctuation, formatting, and structural and contextual clues. Once the search engines work their case-folding, acronym normalization, and other magic, we end up with an awful lot of “pi”s and very little context.

Postscript. The real pi was a bit of a letdown: “P = pi (3.14)” – well, close enough.

Posted by sara at 8:36 pm

Re-definition, Part 3: Where am I, again?

CFR, LII No Responses »
Feb 262016

In our last post, we mentioned that we gained a lot of efficiency by breaking up the CFR text into Sections and processing each section individually. The cost was losing direct access to the structural information in the full Title documents, which made us lose our bearings a bit. In the source data, each Section is nested within its containing structures or “ancestors” (usually Parts and Chapters). Standard XML tools (modern libraries all support XPath) make it trivial to discover a Section’s ancestry or find all of the other Sections that share an arbitrary ancestor.

Once we’d broken up the Titles into Sections, we needed to make sure the software could still accurately identify a containing structure and its descendent Sections.

The first idea was to put a compact notation for the section’s ancestry into each section document. Sylvia added a compact identifier as well as a supplementary “breadcrumb” element to each section. In theory, it would be possible to pull all sections with a particular ancestor and process only those. As it turned out, however, the students found it to be inefficient to keep opening all of the documents to see whether they had the ancestry in question.

So Sylvia constructed a master table of contents (call it the GPS?). The students’ software could now, using a single additional document, pull all sections belonging to any given ancestor. The purists in the audience will, of course, object that we’re caching the same metadata in multiple locations. They’re right. We sacrificed some elegance in the interest of expedience (we were able to deploy the definitions feature on 47 of 49 CFR titles after a semester); we’ll be reworking the software again this semester and will have an opportunity to consolidate if it makes sense.

Posted by sara at 2:32 pm

Re-definition, part 2: Some hills look steeper once you get close up

CFR, LII No Responses »
Jan 222016

Back in September when we started LII: Under the Hood, we waited until we had a bare-bones release of the eCFR text before we told you much about it. “What’s the big deal about going from one XML source to another?” asked, among other people, our director. As it turns out, it was a bit of a bigger deal than we’d hoped.

The good news about the new corpus was that it was much (, much, much, much) cleaner and easier to work with than the original book XML. Instead of being an XML representation of a printed volume (with all of the attendant printed-volume artifacts, such as information about the seal of the National Archives and Records Administration, OMB control numbers, the ISBN prefix, and the contact information for the U.S. Government Publishing Office), the eCFR XML contains just the marked-up text of the CFR and retains only minimal artifacts of its printed-volume origins (we’re looking at you, 26 CFR Part 1). The eCFR XML schema is simpler and far more usable (for example, it uses familiar HTML markup like “table” elements rather than proprietary typesetting-code-derived markup). The structure of the files is far more accurate as well, which is to say that the boundaries of the structural elements as marked-up match the actual structure of the documents.

The bad news was that the software we’d written to extract the CFR structure from the book XML couldn’t be simply ported over; it had to be rebuilt — both to handle the elements in the new schema and to deal with the remaining print-volume artifacts.

The further bad news — or perhaps we should say self-inflicted wound — was that in the process of rebuilding, we decided to change the way we were processing the text.

Here’s what we changed. In the past, we’d divided book-volume XML into CFR Parts; this time we decided instead to divide into Sections (the smallest units which occur throughout the CFR). The advantage, short-term and long-term, is that it makes it far easier to run text-enrichment processes in parallel (many sections can be marked at the same time). That is to say, if our software is linking each term to its definition, we no longer have to wait for all of the links to be added to Section 1.1 before we add links to Section 1.2. The disadvantage is that we have more metadata housekeeping to do to make sure that we’re capturing all of the sections and other granules that belong to a Part or Chapter. That is to say, when we’re working with a Section, we now need another way to know which Part it belongs to. And when we’re marking all instances of a term that has been defined with the scope of a Part, we need a way to be sure that we’ve captured all of the text that Part contains.

And as we learned from our students, metadata housekeeping entails a bit more of a learning curve than XML parsing.

So instead of porting software, we were (by which I mean, of course, Sylvia was) rebuilding it — with a new structure. Suddenly this looked like a much steeper hill to climb.

Next up: Climbing the hill.

Posted by sara at 12:47 pm

Re-definition, part 1

Uncategorized No Responses »
Jan 152016

“Lexicographer: A writer of dictionaries; a harmless drudge, that busies himself in tracing the original, and detailing the signification of words.” – Samuel Johnson

One of our ongoing projects here at LII is to include definitions in our electronic texts. We’ve had them in the UCC since the mid-1990’s; we put them in the CFR in its earlier version a few years ago. Now, as part of the process of rolling out the eCFR and migrating all our old features to it, we’re working on adding definitions to it as well. And just to make life more complicated, we decided at the same time to ask a team of intrepid M.Eng. students (more about them in a subsequent post) to work on the next generation of enhancements to the feature.

Adding definitions to text is a tricky process, so we’re going to milk it for several blog posts take a deliberate approach to spelling out how this works. For starters, let’s look today at why this is a difficult task — why it was hard before, and why it’s hard now.

What we aim for is simple: we want every section of the CFR to have all the key legal terms highlighted. Users could click on each defined term, and they would see how the CFR itself defines those terms. (As you probably know, regulators are famous for defining terms in what we might generously call idiosyncratic ways.) But since the CFR is about a gazillion pages long, there’s no way we can go through and do all those by hand. It needed to be automated.

Which instantly plunged us into the icy waters of natural language processing (or sort-of-natural language processing, a similar if somewhat understudied field).

Definitions in the CFR are not always conveniently labeled; they do not always say something like “we define the term “brontosaurus” to mean “a small plunger to be used on miniature toilets”. They are tricky to pick out — perhaps not tricky for a human reader (although they can be that, too) but certainly tricky for a Turing Machine, even one with its best tuxedo on and its shoes shined.

Consider the following passage:

“Inspected and passed” or “U.S. Inspected and Passed” or “U.S. Inspected and Passed by Department of Agriculture” (or any authorized abbreviation thereof). This term means that the product so identified has been inspected and passed under the regulations in this subchapter, and at the time it was inspected, passed, and identified, it was found to be not adulterated.

We need Tuxedoed Turing to figure out that the three terms in quotations are all equivalent, and all have the same definition. We need it to figure out if an “authorized abbreviation” is being used. We need it to figure out which subchapter this applies to, and not accidentally apply it too narrowly or too widely. And so forth.

And we need it to figure out all sorts of similar problems that other definitions might have: what term precisely is being defined, which words are the definition, what text the definition applies to.

And we made a lot of strides in that direction. The definitions ain’t perfect, but they’re there, in the CFR. Yay, us!

Except, uh, then we had to start all over again. For the eCFR. This time, with shiny new bumpers and a fresh paint job.

Tune in next week for the next part of the story.

Posted by sara at 3:05 pm

Resolutions, with a side of engine grease

Uncategorized No Responses »
Dec 312015

It will not have escaped the attention of our regular readers — both of them, as the old joke goes — that things have been quiet around here under the hood. It’s not that we haven’t been working; it’s that we’ve been too deep in elbow grease to type it up for you. We tried, but the third computer we ruined with dripping engine fluids caused a bit of a ruckus. So it’s been quiet.

But we just went at our hands with a brillo pad and fire hose, so we think it’s safe for the moment. And it’s New Year’s Eve, time for resolutions. So we are going to redouble our efforts to give the world a peek at what we’re doing, and what you’ve missed while we’ve been distracted by that dang fuel carburetor.

As a preview, and a public resolution to which we can point when our boss asks why we’re despoiling yet another keyboard, here are some of the things we’re going to be telling you about in the coming weeks.

  • Students’ work on definitions, entity linking, topic modeling, and a sub-site refresh
  • Citation resolution (turns out not every citation is to Marbury v Madison; who knew?)
  • Semantic web-based feature development

And that’s not all. For we remain hard at work, and new things are constantly in the works, and we’ll tell you about those as well. So watch this space!

But, uh, those are nice pants. You might want to slip on these coveralls first.

Posted by sara at 12:47 pm

Learning from the work of others

Uncategorized, Work we admire No Responses »
Nov 132015

Conferences give us a chance to see other’s work, share ideas, shift perspective, and re-energize our own work. In the last session of the last day of the 2015 Law Via the Internet conference in Sydney, we were treated to an extraordinary panel entitled “Language, translation, and comparative law: East Asian experience”. You might think this is veering a bit out of our lane, but the project report on the translation work from legal and computer science scholars and builders Yoshiharu Matsuura and Amy Shee had a lot in common with problems we’re contending with in our own shop.

Their description of the problem they were addressing:

“There is an implicit assumption that four jurisdictions in East Asia (China, Japan, Korea and Taiwan) share key concepts of law for various reasons. Some people might believe that these jurisdictions share similar legal culture. However, the reality is that accurate knowledge of the legal systems of four jurisdictions is not widely shared even in East Asia.”

Looking at the combination of the linguistic alignment problems and, as Professor Shee described it, “the other translation problem” (the team in Japan works in Ruby; the team in Taiwan works in PHP), we saw a lot of parallels with our own concerns about terminology conformance and bridging the scripting / scaling divide we run into (M.Eng. students we work with prefer Java, which is not our default).

Over the next few weeks, we’ll be posting more from the conference – more soon!

Posted by sara at 1:14 pm

Off to see the wizard...

Uncategorized No Responses »
Nov 062015

Not exactly, but we’re headed to the 2015 Law Via the Internet Conference hosted by AustLII in Sydney, Australia and we’re taking a week off from the blog.

Catch you next week!

Posted by sara at 1:05 pm

A day in the life: Traffic spike

Infrastructure No Responses »
Oct 302015

You were expecting only a couple of people.  And you have a couple bags of chips, some beer and a few soft drinks; that should be plenty.  Somewhere in the back there’s probably some pretzels and seltzer if things get low.  But then the door opens and people keep pouring in.  The living room fills, the kitchen, even the bedroom.  You squeeze through and look out the window, and the entire street is mobbed with people coming over.  No doubt about it: your supplies have crashed.

We know how you feel.

As our regular readers know, we’ve been working on building up the eCFR, so our systems mastermind, Nic, has been keeping a close eye on our load numbers.  But because we’re building things — hammers everywhere, and piles of plywood, and mind the cord please — when our numbers started to spike one evening a few weeks ago, Nic figured that it was some extra overhead from the feature deployment that was running.  Whatever it was, it was heavy: Nic said it was the only time he’d seen Nginx, the web server, run out of available worker processes.

But it wasn’t our the deployment.  Nic looked more closely at the server logs, and it was just an enormous spike in traffic.  Not enough to actually crash the site, but enough that not everyone was able to get the information they were looking for.  Nic kept digging.  It was puzzling: the huge wave of hits came at a very specific time — between 9:20 and 9:40 at night.  They were all from different locations, so it wasn’t a crawler run amok.  What was everyone looking for?  Did someone think we had early Star Wars tickets or something?

Well, the clue was in the specific information people were looking for.  Everyone, it seemed, was suddenly interested in the 14th amendment. Cross checking that against a news twitter stream, and there was the answer.  As it turns out the Republican Presidential debate was that night.  At some point shortly after 9:20, someone mentioned the 14th amendment.  And the next thing you know, we’re out of chips and pretzels.

But of course we at the LII, being hospitable folk, don’t like to run out of munchies.  So we’ve got a plan to make sure it doesn’t happen again.  Nic is gearing up to implement demand-driven autoscaling.  This will set up an automated monitor which keeps track of our load numbers, and then, if it detects a spike, adds a new machine and balances the workload between them.  Once the traffic slows down, the extra machine can quietly switch back off.

So next time the house fills up, we hope to have an extra tent ready to pitch itself automatically, and a whole tub of munchies in the garage.  Because rest assured, we’re here to welcome you, and no one likes a party without something to snack on.

Posted by sara at 3:38 pm

A Look at Indentation

CFR No Responses »
Oct 232015

Great was the rejoicing in the south tower of Myron Taylor Hall, headquarters of the LII, when we got notice of the bulk release of the Electronic Code of Federal Regulations (eCFR) in XML format.

What was not to like? The data was as up-to-date as the CFR could get, the XML was much cleaner than the book version, it had a friendly user guide etc., etc., etc..

It was also different enough from the book XML of the CFR, that we could not simply run it through our existing data enrichment process and serve it to the public as is. So, we curbed our enthusiasm long enough to put together a measured plan to re-do our code.

We have heard enough from you, our wonderful readers, that text indentation was one of the most valued features of our data presentation. Thus, it was the first feature we chose to implement.

All this verbosity is the set-up for a look at some of the messy sausage making details of adding indentation to the eCFR.

***

If you’re not familiar with XML (eXtensible Markup Language), it’s simply a way of marking up data with a predefined, consistent set of descriptive tags that are both easily human and machine readable. So, when we get XML data from the GPO, it looks something like this…

Snippet 1: XML from Title 1 of the CFR
==============================================
<?xml version=“1.0“ encoding=“UTF-8“ ?>
<DLPSTEXTCLASS>
<HEADER>
<FILEDESC>
<TITLESTMT>
<TITLE>
Title 1: General Provisions</TITLE>
<AUTHOR TYPE=“nameinv“>
</AUTHOR>
</TITLESTMT>
<PUBLICATIONSTMT>
<PUBLISHER>
</PUBLISHER>
<PUBPLACE>
</PUBPLACE>
<IDNO TYPE=“title“>
1</IDNO>
<DATE></DATE>
</PUBLICATIONSTMT>
<SERIESSTMT>
<TITLE>
</TITLE>
</SERIESSTMT>
</FILEDESC>
<PROFILEDESC>
<TEXTCLASS>
<KEYWORDS>
</KEYWORDS>
</TEXTCLASS>
</PROFILEDESC>
</HEADER>
<TEXT>
<BODY>
<ECFRBRWS>
<AMDDATE>Jan. 30, 2015</AMDDATE>
<DIV1 N=“1“ NODE=“1:1“ TYPE=“TITLE“>
<HEAD>Title 1 – General Provisions–Volume 1</HEAD>
<CFRTOC>
<PTHD>Part </PTHD>
<CHAPTI>
<SUBJECT><E T=“04“>chapter i</E> – Administrative Committee of the Federal Register </SUBJECT>
<PG>1</PG></CHAPTI>
<CHAPTI>
<SUBJECT><E T=“04“>chapter ii</E> – Office of the Federal Register </SUBJECT>
<PG>51</PG></CHAPTI>
<CHAPTI>
<SUBJECT><E T=“04“>chapter iii</E> – Administrative Conference of the United States </SUBJECT>
<PG>301</PG></CHAPTI>
<CHAPTI>
<SUBJECT><E T=“04“>chapter iv</E> – Miscellaneous Agencies </SUBJECT>
<PG>425
</PG></CHAPTI></CFRTOC>
<DIV3 N=“I“ NODE=“1:1.0.1“ TYPE=“CHAPTER“>
<HEAD> CHAPTER I – ADMINISTRATIVE COMMITTEE OF THE FEDERAL REGISTER</HEAD>
<DIV4 N=“A“ NODE=“1:1.0.1.1“ TYPE=“SUBCHAP“>
<HEAD>SUBCHAPTER A – GENERAL
</HEAD>
<DIV5 N=“1“ NODE=“1:1.0.1.1.1“ TYPE=“PART“>
<HEAD>PART 1 – DEFINITIONS </HEAD>
<AUTH>
<HED>Authority:</HED><PSPACE>44 U.S.C. 1506; sec. 6, E.O. 10530, 19 FR 2709; 3 CFR, 1954-1958 Comp., p.189.
</PSPACE></AUTH>
<DIV8 N=“§ 1.1“ NODE=“1:1.0.1.1.1.0.1.1“ TYPE=“SECTION“>
<HEAD>§ 1.1 Definitions.</HEAD>
<P>As used in this chapter, unless the context requires otherwise – </P>
<P><I>Administrative Committee</I> means the Administrative Committee of the Federal Register established under section 1506 of title 44, United States Code; </P>
<P><I>Agency</I> means each authority, whether or not within or subject to review by another agency, of the United States, other than the Congress, the courts, the District of Columbia, the Commonwealth of Puerto Rico, and the territories and possessions of the United States; </P>
<P><I>Document</I> includes any Presidential proclamation or Executive order, and any rule, regulation, order, certificate, code of fair competition, license, notice, or similar instrument issued, prescribed, or promulgated by an agency; </P>
<P><I>Document having general applicability and legal effect</I> means any document issued under proper authority prescribing a penalty or course of conduct, conferring a right, privilege, authority, or immunity, or imposing an obligation, and relevant or applicable to the general public, members of a class, or persons in a locality, as distinguished from named individuals or organizations; and </P>
<P><I>Filing</I> means making a document available for public inspection at the Office of the Federal Register during official business hours. A document is filed only after it has been received, processed and assigned a publication date according to the schedule in part 17 of this chapter.</P>
<P><I>Regulation</I> and <I>rule</I> have the same meaning. </P>
<CITA TYPE=“N“>[37 FR 23603, Nov. 4, 1972, as amended at 50 FR 12466, Mar. 28, 1985]
</CITA>
</DIV8>
</DIV5>
…
…
…
</DIV1>
</ECFRBRWS>
</BODY>
</TEXT>
</DLPSTEXTCLASS>

The text of the regulations are enclosed within tags that provide some context for what you’re looking at, have meaning for how it should be displayed or provide additional metadata that may be useful to the enrichment process.

As a first step, we consulted the user guide to see if there was any information on how to indent the text. There was something! On page 13, was this snippet of XML (Figure 1) with the enumeration indicators highlighted. The next page had a suggestion for how that could be displayed (Figure 2).

Figure 1: 5 CFR 151.101 in XML format

Figure 2: Presentation suggested by the Government Print Office for 5 CFR 151.101

Obvious to us and as indicated by the user guide itself, there was no way to achieve this display given just the information from the markup. A good place to look for extra information was within the CFR itself.

We found what we were looking for in Title 1, Section 21.11, which is about how the CFR enumerators are organized, or more accurately, are supposed to be organized. Of particular interest was the hierarchy of paragraphs given by subsection 21.11(h):

(h) Paragraphs, which are designated as follows:
level 1(a), (b), (c), etc.
level 2(1), (2), (3), etc.
level 3(i), (ii), (iii), etc.
level 4(A), (B), (C), etc.
level 5(1), (2), (3), etc.
level 6(i), (ii), (iii), etc.

In our first iteration of indentation, we added attributes to each paragraph defining a depth of indentation corresponding to the 6 levels above. Section 151.101 of Title 5, the example in the user guide pages above, looked lovely. But, (you knew it would not be that simple, right?) this implementation worked fine for only about 60% of the random selection of sections we tested it on.

Where the algorithm did not work, the main reason for failure was the presence of multiple enumerators within a single paragraph. In other words, each enumerator should have its own paragraph but not all paragraphs were marked as such.

Snippet 2: XML from 9 CFR 2.1
==============================================
<p>(a)(1) Any person operating or intending to operate as a dealer, exhibitor, or operator of an auction sale, except persons who are exempted from the licensing requirements under paragraph (a)(3) of this section, must have a valid license. A person must be 18 years of age or older to obtain a license. A person seeking a license shall apply on a form which will be furnished by the AC Regional Director in the State in which that person operates or intends to operate. The applicant shall provide the information requested on the application form, including a valid mailing address through which the licensee or applicant can be reached at all times, and a valid premises address where animals, animal facilities, equipment, and records may be inspected for compliance. The applicant shall file the completed application form with the AC Regional Director. </p>

In the snippet above, we have the case where there are 2 enumerators at the beginning of the paragraph. Since our algorithm assumed one enumerator per paragraph, it would only find (a) but not (1). We fixed that in the second iteration.

In our third iteration, we went after more embedded enumerators (see snippet 3 below) by creating a category for these previously untagged enumerators. We named them, nested paragraphs, and tagged them as such.

Snippet 3: XML from 8 CFR 103.3
==============================================
<p>(a) <i>Denials and appeals</i> – (1) <i>General</i> – (i) <i>Denial of application or petition.</i> When a Service officer denies an application or petition filed under § 103.2 of this part, the officer shall explain in writing the specific reasons for denial. If Form I-292 (a denial form including notification of the right of appeal) is used to notify the applicant or petitioner, the duplicate of Form I-292 constitutes the denial order.</p>

In the last snippet, the paragraph has 3 enumerators, (a), (1), and (i). We’ve developed a library of patterns that our algorithm uses to find them all. In title 26 alone, we find and tag 13,563 nested paragraphs!

So, we now have a pretty nice indentation feature, that while not completely finished, is already an improvement over what we were able to do before for the CFR. See 8 CFR 103.3 (a)(1)(iii)(A) and its corresponding eCFR version for an example of this.

We’re putting it on the back burner for now but there is more to come for indentation. For instance, we know from extensive study of the markup that there are actually 8 levels of nesting to be had, not 6. And, we have to provide special handling for sections that do not follow the numbering scheme in 1 CFR 21.11.

We’re grateful for our beta testers and readers. If you come across places where our current indentation scheme does not work, please let us know. In the interim, we’ll be devoting some brain cycles to adding cross references and other links to the eCFR.

Posted by sylvia at 1:48 pm
Older Entries Newer Entries