{"id":42,"date":"2015-10-23T13:48:15","date_gmt":"2015-10-23T17:48:15","guid":{"rendered":"http:\/\/blog.law.cornell.edu\/tech\/?p=42"},"modified":"2015-10-23T15:49:07","modified_gmt":"2015-10-23T19:49:07","slug":"a-look-at-indentation","status":"publish","type":"post","link":"https:\/\/blog.law.cornell.edu\/tech\/2015\/10\/23\/a-look-at-indentation\/","title":{"rendered":"A Look at Indentation"},"content":{"rendered":"

Great was the rejoicing in the south tower of Myron Taylor Hall, headquarters of the LII, when we got notice of the bulk release of the Electronic Code of Federal Regulations (eCFR) in XML format.<\/em><\/p>\n

What was not to like? The data was as up-to-date as the CFR could get, the XML was much cleaner than the book version, it had a friendly user guide etc., etc., etc..<\/em><\/p>\n

It was also different enough from the book XML of the CFR, that we could not simply run it through our existing data enrichment process and serve it to the public as is. So, we curbed our enthusiasm long enough to put together a measured plan to re-do our code.<\/em><\/p>\n

We have heard enough from you, our wonderful readers, that text indentation was one of the most valued features of our data presentation. Thus, it was the first feature we chose to implement.<\/em><\/p>\n

All this verbosity is the set-up for a look at some of the messy sausage making details of adding indentation to the eCFR.<\/em><\/p>\n

***<\/code><\/p>\n

If you’re not familiar with XML (eXtensible Markup Language), it’s simply a way of marking up data with a predefined, consistent set of descriptive tags that are both easily human and machine readable. So, when we get XML data from the GPO, it looks something like this…<\/p>\n

Snippet 1: XML from Title 1 of the CFR<\/span><\/div>\n
==============================================<\/span><\/div>\n
<\/div>\n
<?<\/span>xml<\/span> version<\/span>=“<\/span>1.0“<\/span><\/span> encoding<\/span>=“<\/span>UTF-8“<\/span><\/span> ?><\/span><\/span><\/span><\/div>\n
<<\/span>DLPSTEXTCLASS<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HEADER<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>FILEDESC<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>TITLESTMT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>TITLE<\/span>><\/span><\/span><\/span><\/div>\n
Title 1: General Provisions<\/<\/span>TITLE<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>AUTHOR<\/span> TYPE<\/span>=“<\/span>nameinv“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>AUTHOR<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>TITLESTMT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PUBLICATIONSTMT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PUBLISHER<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>PUBLISHER<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PUBPLACE<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>PUBPLACE<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>IDNO<\/span> TYPE<\/span>=“<\/span>title“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
1<\/<\/span>IDNO<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>DATE<\/span>><<\/span><\/span>\/DATE<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>PUBLICATIONSTMT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>SERIESSTMT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>TITLE<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>TITLE<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>SERIESSTMT<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>FILEDESC<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PROFILEDESC<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>TEXTCLASS<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>KEYWORDS<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>KEYWORDS<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>TEXTCLASS<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>PROFILEDESC<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>HEADER<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>TEXT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>BODY<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>ECFRBRWS<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>AMDDATE<\/span>><\/span><\/span>Jan. 30, 2015<\/<\/span>AMDDATE<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>DIV1<\/span> N<\/span>=“<\/span>1“<\/span><\/span> NODE<\/span>=“<\/span>1:1“<\/span><\/span> TYPE<\/span>=“<\/span>TITLE“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HEAD<\/span>><\/span><\/span>Title 1 – General Provisions–Volume 1<\/<\/span>HEAD<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>CFRTOC<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PTHD<\/span>><\/span><\/span>Part <\/<\/span>PTHD<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>SUBJECT<\/span>><<\/span>E<\/span> T<\/span>=“<\/span>04“<\/span><\/span>><\/span><\/span>chapter i<\/<\/span>E<\/span>><\/span><\/span> – Administrative Committee of the Federal Register <\/<\/span>SUBJECT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PG<\/span>><\/span><\/span>1<\/<\/span>PG<\/span>><\/<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>SUBJECT<\/span>><<\/span>E<\/span> T<\/span>=“<\/span>04“<\/span><\/span>><\/span><\/span>chapter ii<\/<\/span>E<\/span>><\/span><\/span> – Office of the Federal Register <\/<\/span>SUBJECT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PG<\/span>><\/span><\/span>51<\/<\/span>PG<\/span>><\/<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>SUBJECT<\/span>><<\/span>E<\/span> T<\/span>=“<\/span>04“<\/span><\/span>><\/span><\/span>chapter iii<\/<\/span>E<\/span>><\/span><\/span> – Administrative Conference of the United States <\/<\/span>SUBJECT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PG<\/span>><\/span><\/span>301<\/<\/span>PG<\/span>><\/<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>CHAPTI<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>SUBJECT<\/span>><<\/span>E<\/span> T<\/span>=“<\/span>04“<\/span><\/span>><\/span><\/span>chapter iv<\/<\/span>E<\/span>><\/span><\/span> – Miscellaneous Agencies <\/<\/span>SUBJECT<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>PG<\/span>><\/span><\/span>425<\/span><\/div>\n
<\/<\/span>PG<\/span>><\/<\/span>CHAPTI<\/span>><\/<\/span>CFRTOC<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>DIV3<\/span> N<\/span>=“<\/span>I“<\/span><\/span> NODE<\/span>=“<\/span>1:1.0.1“<\/span><\/span> TYPE<\/span>=“<\/span>CHAPTER“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HEAD<\/span>><\/span><\/span> CHAPTER I – ADMINISTRATIVE COMMITTEE OF THE FEDERAL REGISTER<\/<\/span>HEAD<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>DIV4<\/span> N<\/span>=“<\/span>A“<\/span><\/span> NODE<\/span>=“<\/span>1:1.0.1.1“<\/span><\/span> TYPE<\/span>=“<\/span>SUBCHAP“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HEAD<\/span>><\/span><\/span>SUBCHAPTER A – GENERAL<\/span><\/div>\n
<\/<\/span>HEAD<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>DIV5<\/span> N<\/span>=“<\/span>1“<\/span><\/span> NODE<\/span>=“<\/span>1:1.0.1.1.1“<\/span><\/span> TYPE<\/span>=“<\/span>PART“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HEAD<\/span>><\/span><\/span>PART 1 – DEFINITIONS <\/<\/span>HEAD<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>AUTH<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HED<\/span>><\/span><\/span>Authority:<\/<\/span>HED<\/span>><<\/span>PSPACE<\/span>><\/span><\/span>44 U.S.C. 1506; sec. 6, E.O. 10530, 19 FR 2709; 3 CFR, 1954-1958 Comp., p.189.<\/span><\/div>\n
<\/<\/span>PSPACE<\/span>><\/<\/span>AUTH<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>DIV8<\/span> N<\/span>=“<\/span>\u00a7 1.1“<\/span><\/span> NODE<\/span>=“<\/span>1:1.0.1.1.1.0.1.1“<\/span><\/span> TYPE<\/span>=“<\/span>SECTION“<\/span><\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>HEAD<\/span>><\/span><\/span>\u00a7 1.1 Definitions.<\/<\/span>HEAD<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><\/span><\/span>As used in this chapter, unless the context requires otherwise – <\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><<\/span>I<\/span>><\/span><\/span>Administrative Committee<\/<\/span>I<\/span>><\/span><\/span> means the Administrative Committee of the Federal Register established under section 1506 of title 44, United States Code; <\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><<\/span>I<\/span>><\/span><\/span>Agency<\/<\/span>I<\/span>><\/span><\/span> means each authority, whether or not within or subject to review by another agency, of the United States, other than the Congress, the courts, the District of Columbia, the Commonwealth of Puerto Rico, and the territories and possessions of the United States; <\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><<\/span>I<\/span>><\/span><\/span>Document<\/<\/span>I<\/span>><\/span><\/span> includes any Presidential proclamation or Executive order, and any rule, regulation, order, certificate, code of fair competition, license, notice, or similar instrument issued, prescribed, or promulgated by an agency; <\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><<\/span>I<\/span>><\/span><\/span>Document having general applicability and legal effect<\/<\/span>I<\/span>><\/span><\/span> means any document issued under proper authority prescribing a penalty or course of conduct, conferring a right, privilege, authority, or immunity, or imposing an obligation, and relevant or applicable to the general public, members of a class, or persons in a locality, as distinguished from named individuals or organizations; and <\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><<\/span>I<\/span>><\/span><\/span>Filing<\/<\/span>I<\/span>><\/span><\/span> means making a document available for public inspection at the Office of the Federal Register during official business hours. A document is filed only after it has been received, processed and assigned a publication date according to the schedule in part 17 of this chapter.<\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>P<\/span>><<\/span>I<\/span>><\/span><\/span>Regulation<\/<\/span>I<\/span>><\/span><\/span> and <<\/span>I<\/span>><\/span><\/span>rule<\/<\/span>I<\/span>><\/span><\/span> have the same meaning. <\/<\/span>P<\/span>><\/span><\/span><\/span><\/div>\n
<<\/span>CITA<\/span> TYPE<\/span>=“<\/span>N“<\/span><\/span>><\/span><\/span>[37 FR 23603, Nov. 4, 1972, as amended at 50 FR 12466, Mar. 28, 1985]<\/span><\/div>\n
<\/<\/span>CITA<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>DIV8<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>DIV5<\/span>><\/span><\/span><\/span><\/div>\n
…<\/span><\/div>\n
…<\/span><\/div>\n
…<\/span><\/div>\n
<\/<\/span>DIV1<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>ECFRBRWS<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>BODY<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>TEXT<\/span>><\/span><\/span><\/span><\/div>\n
<\/<\/span>DLPSTEXTCLASS<\/span>><\/span><\/span><\/span><\/div>\n

The text of the regulations are enclosed within tags that provide some context for what you’re looking at, have meaning for how it should be displayed or provide additional metadata that may be useful to the enrichment process.<\/p>\n

As a first step, we consulted the user guide to see if there was any information on how to indent the text. There was something! On page 13, was this snippet of XML (Figure 1) with the enumeration indicators highlighted. The next page had a suggestion for how that could be displayed (Figure 2).<\/p>\n

\"Figure<\/a><\/p>\n

\"Figure<\/a><\/p>\n

Obvious to us and as indicated by the user guide itself, there was no way to achieve this display given just the information from the markup. A good place to look for extra information was within the CFR itself.<\/p>\n

We found what we were looking for in Title 1, Section 21.11<\/a>, which is about how the CFR enumerators are organized, or more accurately, are supposed to be organized. Of particular interest was the hierarchy of paragraphs given by subsection 21.11(h):<\/p>\n

(h) Paragraphs, which are designated as follows:<\/span><\/span><\/div>\n
level 1(a), (b), (c), etc.<\/span><\/span><\/div>\n
level 2(1), (2), (3), etc.<\/span><\/span><\/div>\n
level 3(i), (ii), (iii), etc.<\/span><\/span><\/div>\n
level 4(A), (B), (C), etc.<\/span><\/span><\/div>\n
level 5(1), (2), (3), etc.<\/span><\/span><\/div>\n
level 6(i), (ii), (iii), etc.<\/span><\/span><\/div>\n

In our first iteration of indentation, we added attributes to each paragraph defining a depth of indentation corresponding to the 6 levels above. Section 151.101 of Title 5<\/a>, the example in the user guide pages above, looked lovely. But, (you knew it would not be that simple, right?) this implementation worked fine for only about 60% of the random selection of sections we tested it on.<\/p>\n

Where the algorithm did not work, the main reason for failure was the presence of multiple enumerators within a single paragraph. In other words, each enumerator should have its own paragraph but not all paragraphs were marked as such.<\/p>\n

Snippet 2: XML from 9 CFR 2.1<\/span><\/div>\n
==============================================<\/span><\/div>\n
<\/div>\n
<<\/span>p<\/span>><\/span><\/span>(a)(1) Any person operating or intending to operate as a dealer, exhibitor, or operator of an auction sale, except persons who are exempted from the licensing requirements under paragraph (a)(3) of this section, must have a valid license. A person must be 18 years of age or older to obtain a license. A person seeking a license shall apply on a form which will be furnished by the AC Regional Director in the State in which that person operates or intends to operate. The applicant shall provide the information requested on the application form, including a valid mailing address through which the licensee or applicant can be reached at all times, and a valid premises address where animals, animal facilities, equipment, and records may be inspected for compliance. The applicant shall file the completed application form with the AC Regional Director. <\/<\/span>p<\/span>><\/span><\/span><\/span><\/div>\n

In the snippet above, we have the case where there are 2 enumerators at the beginning of the paragraph. Since our algorithm assumed one enumerator per paragraph, it would only find (a) but not (1). We fixed that in the second iteration.<\/p>\n

In our third iteration, we went after more embedded enumerators (see snippet 3 below) by creating a category for these previously untagged enumerators. We named them, nested paragraphs<\/em>, and tagged them as such.<\/p>\n

Snippet 3: XML from 8 CFR 103.3<\/span><\/div>\n
==============================================<\/span><\/div>\n
<\/div>\n
<<\/span>p<\/span>><\/span><\/span>(a) <<\/span>i<\/span>><\/span><\/span>Denials and appeals<\/<\/span>i<\/span>><\/span><\/span> – (1) <<\/span>i<\/span>><\/span><\/span>General<\/<\/span>i<\/span>><\/span><\/span> – (i) <<\/span>i<\/span>><\/span><\/span>Denial of application or petition.<\/<\/span>i<\/span>><\/span><\/span> When a Service officer denies an application or petition filed under \u00a7 103.2 of this part, the officer shall explain in writing the specific reasons for denial. If Form I-292 (a denial form including notification of the right of appeal) is used to notify the applicant or petitioner, the duplicate of Form I-292 constitutes the denial order.<\/<\/span>p<\/span>><\/span><\/span><\/span><\/div>\n

In the last snippet, the paragraph has 3 enumerators, (a), (1), and (i). We’ve developed a library of patterns that our algorithm uses to find them all. In title 26 alone, we find and tag 13,563 nested paragraphs!<\/p>\n

So, we now have a pretty nice indentation feature, that while not completely finished, is already an improvement over what we were able to do before for the CFR. See 8 CFR 103.3 (a)(1)(iii)(A)<\/a> and its corresponding eCFR version<\/a> for an example of this.<\/p>\n

We’re putting it on the back burner for now but there is more to come for indentation. For instance, we know from extensive study of the markup that there are actually 8 levels of nesting to be had, not 6. And, we have to provide special handling for sections that do not follow the numbering scheme in 1 CFR 21.11<\/a>.<\/p>\n

We’re grateful for our beta testers and readers. If you come across places where our current indentation scheme does not work, please let us know. In the interim, we’ll be devoting some brain cycles to adding cross references and other links to the eCFR.<\/p>\n","protected":false},"excerpt":{"rendered":"

Great was the rejoicing in the south tower of Myron Taylor Hall, headquarters of the LII, when we got notice of the bulk release of the Electronic Code of Federal Regulations (eCFR) in XML format. What was not to like? The data was as up-to-date as the CFR could get, the XML was much cleaner […]<\/a><\/p>\n","protected":false},"author":128,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[322],"tags":[],"class_list":["post-42","post","type-post","status-publish","format-standard","hentry","category-cfr"],"_links":{"self":[{"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/posts\/42","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/users\/128"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/comments?post=42"}],"version-history":[{"count":12,"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/posts\/42\/revisions"}],"predecessor-version":[{"id":63,"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/posts\/42\/revisions\/63"}],"wp:attachment":[{"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/media?parent=42"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/categories?post=42"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.law.cornell.edu\/tech\/wp-json\/wp\/v2\/tags?post=42"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}