Abstract

For Web-based publishing, one of the core components is how to make published information available, which is the task of a Web Server (technically speaking, an HTTP Server). While there are many different Web server implementations, this lecture uses the most popular Web server software, the Apache HTTP Server, as an example for the important aspects of Web server configuration.

Web Basics

• The Web transfers URI-identified XML-encoded resources using HTTP
• Identification: URI
  • resources should be identified by URIs to make them accessible
  • using URIs as identification makes it possible to base resource access on HTTP
• Communications: HTTP
  • HTTP is an access protocol providing non-trivial resource access
  • HTTP-based infrastructures support caching and redirection
• Representation: HTML/XML (RSS, Atom, XLink)
  • HTTP allows content negotiation and thus various views of resources
  • ideally, the resources provide access to other resources (with URI/HTTP)

Server Statistics

HTTP Service

• HTTP provides basic URI-based services
  • handling URI redirections and URI rewriting
  • content negotiation based on HTTP headers and server configuration
  • but finally, some resource representation (i.e., URI-based content) has to be returned
• HTTP relies on some content provider for delivering the representation
  • in simple cases, the file system is the content provider
  • HTTP servers may do a bit of content provision themselves with Server-Side Includes (SSI)
  • there can be arbitrarily complex logic behind the process of providing content

Apache Request Processing

Filters

• Apache provides a number of standard filters
  • mod_include implements Server-Side Includes (SSI)
  • mod_ssl implements SSL encryption (HTTPS)
  • mod_deflate implements compression/decompression on the fly
  • mod_charset_lite transcodes between different character sets
  • mod_ext_filter runs an external program as a filter
• Additional filters can be installed as third-party components
  • HTML and XML processing and rewriting
  • XSLT transforms and XIncludes
  • file upload handling and decoding of HTML Forms
  • image processing
  • text search-and-replace editing

Web Pages

• Files (i.e., resource representations) can have a number of properties
  • the language of their content
  • the media type they are using for their content
  • for text-based media types, the character set and character encoding
• Different representations can be provided in different ways
  • they might be provided as static files, produced manually
  • they might be generated on demand
  • for language variants, manual production is more likely
• File systems are simple (and bad) Content Management Systems (CMS)
  • navigation information can be inferred from the available content
  • access to content can be based on workflows and authentication
  • document production can be automated (your daily news digest)

Where to Look

• Apache is configured using text-based files
  • this is the most reliable way across various platforms
  • some GUIs are available but are usually not a good idea
• Apache comes with an out-of-the-box configuration
  • httpd.conf located in some installation-dependent directory
  • your mileage may vary: Mac OS X's Web sharing has its own default Apache setup
• Configuration files contain configuration directives
  • directives may apply to the entire server, particular directories, files, hosts, or URIs
  • on a per-directory basis, .htaccess files can overwrite configuration directives
  • .htaccess files are searched through the full URI hierarchy

Limiting Clients

• Access control sometimes can be based on simple rules
  • for intranet applications, access control can be based on an IP address range
  • sometimes other criteria may be important (such as the user agent)

  Order deny,allow
  Deny from all
  Allow from berkeley.edu

• Module mod_setenvif allows to use arbitrary environment variables

  SetEnvIf User-Agent BadBot GoAway=1
  Order allow,deny
  Allow from all
  Deny from env=GoAway

• Module mod_rewrite allows the use or arbitrary criteria

  RewriteEngine On
  RewriteCond %{TIME_HOUR} > 20 [OR]
  RewriteCond %{TIME_HOUR} < 07
  RewriteRule ^/fridge - [F]

Managing Authentication

• Access control sometimes may be too limiting
  • authentication allows access if proper credentials can be supplied
  • authentication usually is based on user/password information
• Apache has a number of ways how to handle authentication information
  • mod_authn_file for storing user names and passwords in text files
  • mod_authn_dbm for storing user names and password in dbm files
  • Apache-internal authentication is inconvenient to manage (replicated information)
  • mod_authnz_ldap for requesting authentication from an LDAP directory
  • mod_authn_dbd for requesting authentication from a SQL database

Requesting Authorization

• Servers often have password-protected areas
  • <Directory /usr/local/apache/apache/htdocs/secret> in httpd.conf
  • placing an .htaccess file in the directory (AllowOverride AuthConfig required)

  AuthType Basic
  AuthName "Restricted Files"
  AuthBasicProvider file
  AuthUserFile /usr/local/apache/passwd/passwords
  Require user dret

• Groups can be managed in a group file

  AuthType Basic
  AuthName "By Invitation Only"
  AuthBasicProvider file
  AuthUserFile /usr/local/apache/passwd/passwords
  AuthGroupFile /usr/local/apache/passwd/groups
  Require group invitedPeople

• Often it is sufficient to recognize a user as a well-known user

   Require valid-user

Server-Side Processing

• A request results in a file that has to be served
  • this may happen after URI redirection, rewriting, or content negotiation
  • the file is read from the file system and included in the HTTP response
  • response headers are set as configured for the server
• The file is copied directly to the HTTP response
  • the fastest way from the disk to the network
  • anything else would affect highly loaded Web server negatively
• Documents may contain code which should be interpreted
  • this could be very simple information like insert the file's modification date
  • as a consequence, the complete file must be parsed before it can be sent

SSI Configuration

• The Web server must allow SSI to be used
  • the SSI module mod_include must be compiled into the server
  • Options Includes must be set to enable SSI processing
• The Web server must know which files it should parse
  • parsing all files would slow down processing unnecessarily
  • Apache supports two different ways of marking a file as SSI
  1. the file extension can be used (files must be named as set)

     AddType text/html .shtml
     AddHandler server-parsed .shtml

  2. the file mode (only available in Unix file systems) can be used (files must be chmod +x)

     XBitHack on

• The fist way is cleaner and more portable (across OS)
  • any decent Web server should not expose file extensions anyway

Basic SSI

• SSI directives are mostly used for simple tasks
• The server replaces the directive with the result
  • the syntax uses HTML comment syntax

    <!--#element attribute=value attribute=value … -->

• Some directives return frequently used values
  • these values depend on the individual request

    <!--#echo var="DATE_LOCAL" -->

    <!--#echo var="LAST_MODIFIED" -->

    <!--#config timefmt="%A %B %d, %Y" -->

• Directives may include other files
  • depending on these files, SSI may be nested

    <!--#include virtual="footer.shtml" -->

Processing Problems

• SSI directives are executed
• Whenever things are executed, problems may occur
  • less likely for including date and time information
  • more likely for including other files
• SSI error handling is very simple
  • standard error text: [an error occurred while processing this directive]

    <!--#config errmsg="[ It appears that you don't know how to use SSI ]" -->

    SSIErrorMsg "[ It appears that you don't know how to use SSI ]"

  • for production systems, it makes more sense to not display error messages

    SSIErrorMsg "<!-- Error -->"

Advanced SSI

• SSI pages can use conditional code
  • the variables are environment variables or set by Apache's SetEnvIf

    <!--#if expr="test_condition" -->
    <!--#elif expr="test_condition" -->
    <!--#else -->
    <!--#endif -->

• Directives may invoke external programs
  • the call mechanism is through the Common Gateway Interface (CGI)

    <!--#include virtual="/cgi-bin/counter.pl" -->

  • this may introduce security issues (everybody can invoke the program)
• Directives may invoke OS commands
  • this works in the same way as the OS shell

    <!--#exec cmd="dir" -->

  • this introduces major security issues (everybody can invoke the command)
  • IncludesNOEXEC enables SSI but disables exec

URI vs. MIME

• URIs do not have any media type semantics
  • File extensions are convenient conventions for file names
  • if possible, file extensions should be avoided in URIs
  • Browsers act based on the media type in the HTTP response
• Apache uses a list of recognized media types
  • very similar to Unix File Type Handling with mime.types

  text/calendar
  text/css   css
  text/directory
  text/enriched
  text/html   html htm

  text/calendar   ics ifb
  text/css   css
  text/directory
  text/enriched
  text/html   html htm

Type Maps

• URIs are names for resources
  • HTTP always transfers resource representations and not resources themselves
  • resources are the abstract concept and may have more than one representation
• Type maps provide an explicit list of available representations
  • AddHandler type-map .var instructs the server to look for type maps
  • based on the request and the type map the server chooses the best representation

URI: foo

URI: foo.en.html
Content-type: text/html
Content-language: en

URI: foo.fr.de.html
Content-type: text/html;charset=iso-8859-2
Content-language: fr, de

Emulating Type Maps

• Managing type maps can become a cumbersome process
  • for Content Management Systems (CMS) the type maps can be generated based on CMS content
  • for manually maintained Web servers there is a more convenient way
• MultiViews enables the server to emulate type maps
  1. /dir/foo is requested and does not exist and MultiViews is enabled
  2. the server looks for /dir/foo.* and emulates a type map
  3. media types and encodings are set based on the server configuration
  4. based on the emulated type map and the request the server chooses the best match

Negotiation Algorithm

• Apache uses fours parameters for content negotiation
  • media type based on the HTTP Accept header field
  • language based on the HTTP Accept-Language header field
  • entity encoding based on the HTTP Accept-Encoding header field
  • character set and encoding based on the HTTP Accept-Charset header field
• Apache has a built-in algorithm for determining the best variant
  1. assign qualities to all variants based on the accept headers (for all dimensions)
  2. based on internal weighting, the various dimensions are used to find the best representation
  3. send the best representation and indicate the dimensions in the Vary header
  4. if nothing was selected, send a 406 No acceptable representation status code

Controlling Crawlers

• Crawlers are important for making a Web site accessible
  • certain areas of a Web site may not be interesting for crawlers
  • the robots exclusion protocol provides a simple way to steer crawlers

# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

User-agent: webcrawler
Disallow:

User-agent: lycra
Disallow: /

User-agent: *
Disallow: /tmp
Disallow: /logs

Steering Crawlers

• Crawling a web site is a tough job and generates a lot of traffic
  • one of the biggest problems is that crawlers know little about the Web site
  • using site-specific knowledge Web servers can help crawlers
  • how much control can crawlers allows without being vulnerable for abuse?
• Most search engines offer some control over their crawlers
  • Google's XML Sitemap format
  • Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)
  • content syndication formats (RSS or Atom) requested by feed readers

Your Customer Touchpoint

• Server setup is important for being a good Web citizen
• Server setup integrates with URI design and REST protocols
• Generating and analyzing statistics is an important tool