Abstract

The XML specification defines a format for structured data (XML documents) and a grammar-based constraint language for these (DTD). In SGML-based systems, DTDs were often very complex and feature-rich constructs, which controlled a lot of the processing of SGML documents. XML greatly simplified DTDs, and de-facto usage of DTDs today simplified them even more. In many systems today, DTDs are not used at all or generated from sample documents. In this lecture, it is argued that DTDs (or schemas, to be more general) should be taken seriously in any non-trivial XML application, because they are a representation of the underlying (and often underspecified) data model of the application.

XML Validation

• XML knows two states of documents, well-formed and valid
• well-formed documents satisfy all basic constraints of the XML specification
  • they can be parsed according to the XML grammar
  • they satisfy the additional constraints (e.g., start and end tags match)
  • together, this means they can be translated into a tree
• valid documents have been validated against a DTD
  • a document must be well-formed before it can be validated
  • all elements and attributes have been defined
  • elements and attributes are used according to their definition

Validation and Applications

Non-XML, Well-Formed, and Valid

[ Could not find listing file "src/non.xml" ]

<address>
 <name short="iSchool">School of Information</name>
 <voice>(510) 642-1464</voice>
 <fax>(510) 642-1464</fax>
 <website>http://ischool.berkeley.edu/</website>
 <postal>...</postal>
</address>

<address>
 <name short="iSchool">School of Information</name>
 <phone type="voice">(510) 642-1464</phone>
 <phone type="fax">(510) 642-1464</phone>
 <website>http://ischool.berkeley.edu/</website>
 <postal>...</postal>
</address>

DTD Example

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE address SYSTEM "address.dtd">

<!ELEMENT address (name, phone*, website*, postal?)>
<!ELEMENT name (#PCDATA)>
<!ATTLIST name
 short CDATA #REQUIRED
>
<!ELEMENT phone (#PCDATA)>
<!ATTLIST phone
 type ( voice | fax ) #REQUIRED
>
<!ELEMENT postal (#PCDATA)>
<!ELEMENT website (#PCDATA)>

• The DTD defines constraints on element and attribute usage
• The DTD does only in part constrain textual contents

XML Schema Languages

• DTDs are part of XML itself
  • XML specifies the document format and one schema language
  • DTD support is provided by most XML processors (validating processors)
• Other schema languages are available
  • XML Schema as the W3C's recommendation
  • Schematron as a rule-based alternative
  • various other research projects and products
• Choosing appropriate schema language(s) is important
  • we look at DTDs because they are part of XML itself
  • we look at XML Schema because it is widely used
  • we look at Schematron because it is simple and powerful
  • you may even invent your own schema language (a.k.a. meta-programming)

XML is SGML light

• XML is a subset of SGML
  • XML documents have been greatly simplified
  • XML DTDs have retained more of SGML's peculiarities
• DTD design should be left to XML experts
  • simple DTDs (for prototypes) are easy to define (or generate)
  • serious DTDs for complex data models are hard to define
• XML is a useful tool for experiments and prototypes
  • basic knowledge of DTDs is required
  • serious XML schemas often use XML Schema anyway

Connecting Documents and DTDs

• A DTD is a schema for a set of documents
  • there may be just one document for a DTD, there may be billions (HTML)
  • in most cases, DTDs are managed as a separate resource
• The Document Type Declaration contains or points to markup declarations that provide a grammar for a class of documents
  • the part which is contained is called Internal Subset
  • the part which is pointed to is called External Subset
  • internal and external subset together are the Document Type Definition (DTD)
• External subsets are identified by Public and System Identifiers
  • public identifiers use a special notation
  • system identifiers are URIs (relative or absolute)
  • applications use (i.e., know or retrieve) the DTD for validation

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE address SYSTEM "address.dtd">

DTDs are not XML Documents

• DTDs use a special syntax
  • somewhat ironic when everything else is XMLized
  • DTDs cannot be processed with standard XML tools
  • more compact than XML syntax
• Definition of elements and attribute lists
  • elements are defined by the content they allow
  • attribute lists are sets of allowed attributes on elements

<!ELEMENT address (name, phone*, website*, postal?)>
<!ELEMENT name (#PCDATA)>
<!ATTLIST name
 short CDATA #REQUIRED
>
<!ELEMENT phone (#PCDATA)>
<!ATTLIST phone
 type ( voice | fax ) #REQUIRED
>
<!ELEMENT postal (#PCDATA)>
<!ELEMENT website (#PCDATA)>

Syntax Rules

• There is no container containing the whole DTD
  • <!ELEMENT xml EMPTY> thus is a complete DTD
• Definitions (officially called declarations) use <!... > syntax
  • ELEMENT is used to define an element
  • ATTLIST is used to define an attribute list
  • ENTITY is used to define an entity
• The document element is not marked explicitly
  • but it must be declared in the document type declaration
  • this means the document element is defined by the document, not by the DTD

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE address SYSTEM "address.dtd">
<address>

Element Only Content

• Element content is defined by a grammar for the children
  • sequences are indicated with a comma: ,
  • choices are indicated with a vertical bar: |
  • optional parts are indicated with a question mark: ?
  • repeatable parts are indicated with a plus: +
  • optional and repeatable parts are indicated with a asterisk: *
  • parentheses can be used for grouping and nesting

<!ELEMENT table
     (caption?, (col*|colgroup*), thead?, tfoot?, (tbody+|tr+))>
<!ELEMENT caption  %Inline;>
<!ELEMENT thead    (tr)+>
<!ELEMENT tfoot    (tr)+>
<!ELEMENT tbody    (tr)+>
<!ELEMENT colgroup (col)*>
<!ELEMENT col      EMPTY>
<!ELEMENT tr       (th|td)+>
<!ELEMENT th       %Flow;>
<!ELEMENT td       %Flow;>

Mixed Content

• Mixed Content allows text content and elements to be mixed
  • Whitespace characters are allowed in Element Only Content (this must not be declared)
  • for non-whitespace characters, character data must be allowed explicitly
• The allowed child elements may be constrained, but not their order or their number of occurrences
• Mixed Content always is defined as <!ELEMENT x (#PCDATA | a | b | ...)* >

<!ELEMENT address (#PCDATA | %inline; | %misc.inline; | p)*>

• character only content is a special case of mixed content
  • the element may only contain characters (no other elements)
  • the repetition is not necessary because there is no choice

<!ELEMENT style (#PCDATA)>

Empty Content

• Empty elements can be useful
  • they may contain all information in attributes
  • their presence may carry semantics without the need for additional information

<!ELEMENT img EMPTY>
<!ATTLIST img
  %attrs;
  src         %URI;          #REQUIRED
  alt         %Text;         #REQUIRED
  name        NMTOKEN        #IMPLIED
  longdesc    %URI;          #IMPLIED
  height      %Length;       #IMPLIED
  width       %Length;       #IMPLIED
  usemap      %URI;          #IMPLIED
  ismap       (ismap)        #IMPLIED
  align       %ImgAlign;     #IMPLIED
  border      %Length;       #IMPLIED
  hspace      %Pixels;       #IMPLIED
  vspace      %Pixels;       #IMPLIED
  >

Attributes belong to Elements

• Attributes are specified in an element's Attribute List
  • an element definition may have any number of attributes associated with it
  • attributes may occur at most once on an element
• Attributes definitions have a name, a type, and a default declaration
  • the attribute appears according to the default declaration
  • if the attribute is present, its value must conform to the type

<!ELEMENT param EMPTY>
<!ATTLIST param
  id          ID             #IMPLIED
  name        CDATA          #REQUIRED
  value       CDATA          #IMPLIED
  valuetype   (data|ref|object) "data"
  type        %ContentType;  #IMPLIED
  >

Attribute Types

• Attribute values can be constrained (which is not possible for element content)
  • CDATA means any character string (but no markup)
  • enumerated types list allowed values: (data|ref|object) (list of XML names)
  • ID for identifying elements (part of ID/IDREF)
  • IDREF for referencing identified elements (part of ID/IDREF)
• Application-oriented attribute types are often simulated
  • using Parameter Entities, modeling information can be preserved

  accept-charset %Charsets;  #IMPLIED

<!ENTITY % Charsets "CDATA">
    <!-- a space separated list of character encodings, as per [RFC2045] -->

• The default declaration specifies the attribute's presence
  • #REQUIRED means the attribute has to be specified (on every element)
  • #IMPLIED marks an optional attribute (the parser may imply a value)
  • "..." specifies a default value (and the attribute is optional)

References in Documents

• Without Validation, there are no IDs
  • ID is an attribute type declared in the DTD
  • xml:id is an attempt to support schema-independent IDs
• IDs are used to assign identities to elements
  • the XML processor reports duplicate IDs as errors (part of validation)
• IDREFs are used to reference existing IDs
  • the XML processor reports references to non-existing IDs as errors (part of validation)
• IDs must be XML Names (in particular, they may not start with a number)

ID/IDREF in a Document

<document>
 <section id="sgml" author="dret">
  <title>Standard Generalized Markup Language (SGML)</title>
  <p>SGML is an ISO standard ...</p>
  <section id="sgml-syntax" author="bob">
   <title>SGML Syntax</title>
   <p>SGML uses markup, which is ...</p>
  </section>
 </section>
 <section id="xml" author="bob">
  <title>Extensible Markup Language (XML)</title>
  <p>XML is based on SGML (Section <ref name="sgml"/>) ...</p>
  <p type="example">XML can be used ...</p>
  <section id="xml-syntax" author="dret">
   <title>XML Syntax</title>
   <p>Section <ref name="sgml-syntax"/> describes ...</p>

<!ELEMENT section ( title, p+, section* ) >
<!ATTLIST section
 id ID #REQUIRED
 author CDATA #REQUIRED >
<!ELEMENT title ( #PCDATA )>
<!ELEMENT p ( #PCDATA | ref )*>
<!ATTLIST p
 type CDATA #IMPLIED >
<!ELEMENT ref EMPTY >
<!ATTLIST ref
 name IDREF #REQUIRED >

References within the Tree

Formatting Example

XSLidy can generate links to sections such as the section about ID/IDREF, this link is then translated into the appropriate HTML code, meaning a link with the target being a fragment identifier to the slide number.

<p>XSLidy can generate links to sections such as the section about <link href="ididref"/>, this link is then translated into the appropriate HTML code, meaning a link with the target being a fragment identifier to the slide number.</p>

After running XSLidy, the following HTML is generated:

<p>XSLidy can generate links to sections such as the section about <a href="#(23)">ID/IDREF</a>, this link is then translated into the appropriate HTML code, meaning a link with the target being a fragment identifier to the slide number.</p>

ID/IDREF Semantics

• Rooted in the document world
  • all parts are assembled before processing
  • names are symbolic and assigned as required
  • mixed syntax and semantics
• Good idea, but many shortcomings
  • constraints apply to one document only
  • IDs and IDREFs are global instead of scoped
  • identifiers should be allowed to use any type
  • identifier processing should be type-specific (2 ≟ 02)
• Applications must know how to process ID/IDREF
  • for HTML export, links can be generated
  • for databases, keys should be used

General Entities

• XML's core concept of physical data structures
  • an entity is a named unit of data which can be referenced
  • within documents, it is referenced by the markup &entity-name;
• Entities can be used to name and reuse document content

<!ENTITY aacute "&#225;"> <!-- latin small letter a with acute,
                                  U+00E1 ISOlat1 -->
<!ENTITY acirc  "&#226;"> <!-- latin small letter a with circumflex,
                                  U+00E2 ISOlat1 -->
<!ENTITY atilde "&#227;"> <!-- latin small letter a with tilde,
                                  U+00E3 ISOlat1 -->
<!ENTITY auml   "&#228;"> <!-- latin small letter a with diaeresis,
                                  U+00E4 ISOlat1 -->

• Character References look like entities: ☺ or ☺ = ☺
  • they can be used to represent any Unicode character, they are processed as single characters

Parameter Entities

• Parameter entities are parsed entities for use within the DTD
  • a parameter entity must be specifically declared as such
  • within DTDs, it is referenced by the markup %entity-name;
  • outside of DTDs, parameter entities cannot be used
• As general entities, parameter entities are meant for reuse
  • in a DTD, reuse is mostly about reusing structures
  • parameter entities are DTDs duct tape, not elegant, but effective

XHTML Parameter Entities (Attributes)

<!ELEMENT p %Inline;>
<!ATTLIST p
  %attrs;
  %TextAlign;
  >

<!ENTITY % attrs "%coreattrs; %i18n; %events;">

<!ENTITY % coreattrs
 "id          ID             #IMPLIED
  class       CDATA          #IMPLIED
  style       %StyleSheet;   #IMPLIED
  title       %Text;         #IMPLIED"
  >

<!ENTITY % i18n
 "lang        %LanguageCode; #IMPLIED
  xml:lang    %LanguageCode; #IMPLIED
  dir         (ltr|rtl)      #IMPLIED"
  >

<!ENTITY % LanguageCode "NMTOKEN">
    <!-- a language code, as per [RFC3066] -->

<!ENTITY % TextAlign "align (left|center|right|justify) #IMPLIED">

XHTML Parameter Entities (Content)

<!ELEMENT p %Inline;>
<!ATTLIST p
  %attrs;
  %TextAlign;
  >

<!ENTITY % Inline "(#PCDATA | %inline; | %misc.inline;)*">

<!ENTITY % inline "a | %special; | %fontstyle; | %phrase; | %inline.forms;">

<!ENTITY % special
   "%special.basic; | %special.extra;">

<!ENTITY % special.basic
 "br | span | bdo">

<!ENTITY % special.extra
   "object | applet | img | map | iframe">

<!ENTITY % misc.inline "ins | del | script">

Additional Mechanisms

• DTDs have more advanced mechanisms
  • used in few applications, mostly by SGML veterans
  • should not be used in new projects
• Conditional Sections for configurable DTDs
  • parts of a DTD can be enclosed in special constructs
  • based on parameter entity setting, these parts can be switched on or off
• External Entities for referencing external resources
  • parsed entities contain content parsed by the XML processor
  • inclusion should be done with XInclude
  • unparsed entities contain non-XML content (e.g., images or plain text)
  • referring to non-XML content is handled on the application level

DTD for XML Schemas

• XML documents are processed by applications
• Applications have assumptions about XML documents
• DTDs allow to formalize some of these constraints
• Part of the constraint checking must still be programmed

Modeling DTDs

• Data models can be mapped to many different DTDs
• What is a good DTD? What is a bad DTD?
• How does the DTD affect further processing?