Libwww

Other names: CERN Common WWW Library, 1 ; 2 CERN World-Wide Web Library of Common Code, 3 W3C Reference Library, 4 W3C Sample Code Library, 5 W3C Protocol Library 6 Original authors: Tim Berners-Lee , 6 Jean-François Groff , 7 ; 6 Henrik Frystyk Nielsen 6 ; 8 Developer: José Kahan Initial release: 1.0, November 1992; 33 years ago (1992-11) 9 Stable release: 5.4.2 10 / 24 June 2017; 8 years ago (24 June 2017)

Repository : github .com /w3c /libwww

Written in: C Operating system : FreeBSD , 11 Solaris , 11 Linux , 11 Mac OS X , 11 Microsoft Windows 11 Type : Library for web browsers , servers , and other protocols License : W3C Software Notice and License Website: www .w3 .org /Library

Libwww, a name as uninspired as it is functional, stands as a foundational software library from the nascent days of the World Wide Web . It wasn’t just an early library; it was the early library, providing the essential, core functionalities that allowed the web to even begin its slow, inevitable sprawl across our collective consciousness. This venerable piece of code implemented critical technologies like HTML and HTTP , alongside a host of other protocols that, frankly, most people have long forgotten.

Tim Berners-Lee , laboring away at the European Organization for Nuclear Research (CERN )—a place more accustomed to smashing atoms than connecting documents—unveiled libwww (then known by the equally thrilling moniker, the Common Library) in the latter half of 1992. It essentially distilled the reusable essence from the very first browsers ever conceived: WorldWideWeb (yes, that was its name, before they realized spaces were useful) and the rather primitive Line Mode Browser . One might say it was the primordial soup from which the digital world eventually crawled.

For a time, libwww held significant sway, becoming a crucial dependency for browsers that, for a brief, shining moment, were considered popular, such as Mosaic 12 . But, as is the way of all things, especially in technology, interest proved fleeting. By 1997, the collective gaze had shifted, and the World Wide Web Consortium (W3C), having inherited the mantle from CERN , began to visibly reduce its commitment to the project 13 . Its purpose, once grand and encompassing, was then rather unceremoniously redefined to serve as “a testbed for protocol experiments” 6 . In this diminished capacity, it continued to be maintained, primarily for the benefit of the W3C’s own web standards -promoting browser, Amaya 14 . Unsurprisingly, active development of libwww finally sputtered to a halt in the year 2000 9 ; 15 . A fitting end for something that had seen the future and found it merely… adequate.

History

The genesis of libwww can be traced back to 1991 and 1992, when Tim Berners-Lee and a rather industrious student at CERN named Jean-François Groff embarked on a mission. Their goal was to refactor and rewrite various integral components of the original WorldWideWeb browser. This initial browser, a marvel of its time, was built specifically for the NeXTstep operating system . Their efforts were aimed at transforming this specialized code into portable C code, a strategic move intended to broadly showcase the revolutionary potential of the burgeoning World Wide Web 16 . Without this foresight, the web might have remained a niche curiosity, confined to the elegant, but ultimately exclusive, black boxes of NeXT.

In its infancy, this collection of code was simply known as the Common Library and was not, for the record, available as a distinct, standalone product 1 . Before it made its grand debut to the wider world, the Common Library was actually integrated into the CERN program library (CERNLIB), a repository of scientific software 17 . Showing early signs of its cross-platform ambitions, the library was successfully ported to DECnet in July 1992 18 . It wasn’t until May 1993, within the pages of the World Wide Web Newsletter—a publication that likely held more gravitas then than it would today—that Berners-Lee officially announced the rebranding of the Common Library to the more streamlined “libwww.” More significantly, he declared it licensed under the public domain , a decision explicitly made to foster and accelerate the development of web browsers 19 .

This choice of licensing was not without its internal debate. Berners-Lee had initially weighed the option of releasing the software under the GNU General Public License (GPL). However, he ultimately decided against it, driven by pragmatic concerns that the GPL’s restrictive nature might deter large, influential corporations such as IBM from adopting and utilizing the library, thereby hindering the web’s widespread acceptance 16 ; 20 . This rapid, almost frantic, early development cycle of the library, while a testament to its creators’ drive, did present challenges. Robert Cailliau , for instance, encountered significant hurdles when attempting to integrate libwww into his MacWWW browser, a testament to the chaotic beauty of pioneering software 12 .

From February 1994, through an impressive stretch until July 1999, encompassing versions 2.17 to 5.2.8, the stewardship of libwww fell to Henrik Frystyk Nielsen . Nielsen’s journey with the project began as a graduate student at CERN and continued as he transitioned to the newly formed World Wide Web Consortium (W3C) 6 ; 8 ; 21 . A pivotal moment arrived on March 21, 1995, with the release of version 3.0, when CERN officially transferred the responsibility for libwww to the W3C, marking a symbolic shift in the web’s governance 9 . From 1995 onward, the Line Mode Browser , a relic of an even earlier era, ceased to be released as a standalone entity, instead becoming an integrated component of the libwww package 22 .

By March 2, 1997, Nielsen, perhaps sensing the shifting tides or simply the sheer exhaustion of maintaining such a beast, announced that Libwww 5.1 was anticipated to be its final release 13 . Later that year, on December 24, 1997—a rather bleak Christmas Eve for web developers—Nielsen issued an unsuccessful plea for an external party, outside the W3C’s immediate purview, to assume the ongoing maintenance of the library 23 . The web was evolving, and libwww, despite its foundational importance, was beginning to feel like yesterday’s news.

Nielsen eventually departed the W3C in July 1999 21 . Following his departure, the project, by then a shadow of its former self, was headed by José Kahan, who, for a time, was the sole W3C employee still involved with the project 21 ; 14 .

The official pronouncements of its demise came in waves, as if the W3C couldn’t quite bring itself to pull the plug decisively. On September 2, 2003, the W3C (re-)stated that development had indeed ceased, citing the rather predictable and eternally valid excuse of a “lack of resources” 24 . A few months later, on January 29, 2004, the W3C reiterated its stance, confirming that it would not continue development and, with a hopeful but ultimately futile gesture, sought maintainers from the broader open source community 25 .

The first (and, tellingly, only) “community supported maintenance release” finally materialized in 2005, a rather significant three-year gap after its official abandonment 15 . After a further, truly impressive lapse of 12 years, a security patch was grudgingly released in 2017 26 . One might wonder if it was out of genuine concern or merely a collective sigh that someone had to do it.

Features

In 2003, José Kahan, perhaps attempting to polish a somewhat tarnished legacy, boldly asserted that “libwww is the only library that has a full implementation of the HTTP specification, including caching and pipelining” 24 . While such claims are often subject to historical revisionism, it speaks to the comprehensive nature of the library in its prime. It was, for a time, the definitive toolkit for building web interactions.

Libwww, in its robust heyday, supported a wide array of protocols , forming the backbone of early internet communication:

• file : Allowing access to local files, a fundamental capability for any system that interacts with a user’s machine 27 .
• FTP : The venerable File Transfer Protocol , essential for moving files across networks long before HTTP became the dominant force 27 .
• Gopher : A hierarchical document retrieval system that predated and briefly competed with the World Wide Web itself. Libwww’s support ensured that early web clients could still navigate the vast, text-based Gopher-space 27 .
• HTTP 1.1: The cornerstone of the web. Libwww’s implementation included sophisticated features like a Persistent Cache Manager for improved performance and pipelining for more efficient request handling, which were advanced capabilities at the time 27 ; 9 .
• NNTP : The Network News Transfer Protocol , which powered Usenet newsgroups, a vibrant (and often chaotic) forum for online discussion and information exchange in the pre-web era 27 .
• Telnet : A simple, text-based network protocol for remote command-line access, representing a very basic form of internet interaction 27 .
• WAIS : Wide Area Information Servers , another early distributed information retrieval system, offering full-text search capabilities across databases 27 .

Beyond protocol support, libwww offered a suite of other features, demonstrating its ambition to be a comprehensive web toolkit:

• TLS and SSL Integration: The library could leverage OpenSSL to provide secure communication channels, an increasingly vital component as the web began to handle sensitive information. Because, apparently, even in the early days, some people cared about privacy 28 .
• Compression Support: It incorporated gzip compression and decompression capabilities through the widely used zlib library, optimizing data transfer and reducing bandwidth consumption, a critical concern in an era of dial-up connections 29 .
• Parsing and Styling: A robust set of parser s for HTML 30 ; 31 , RDF 32 , SGML 33 , and XML 32 was included, alongside a style sheet manager 34 . This was crucial for interpreting and rendering the complex structures of web content, even if “complex” back then often meant a bold tag and a horizontal rule.
• Database Integration: It featured an integration with a SQL database, specifically utilizing the MySQL server. This capability was particularly useful for applications requiring persistent data storage, such as early web crawlers that needed to index vast amounts of information 35 .

Furthermore, libwww was designed with extensibility in mind, supporting plug-ins 36 . Because nothing says “future-proof” like the ability to bolt on more functionality later, until the whole thing collapses under its own weight.

Applications using libwww

Libwww, despite its eventual fading relevance, was a workhorse in its time, powering a surprisingly diverse range of applications. It was employed in projects of varying scales and complexities, from the foundational tools of the web to more specialized utilities. Its modular design allowed it to be integrated into everything from full-fledged web browsers and editors to automated Internet bots and batch processing tools. The pluggable modules inherent in libwww extended its capabilities significantly, adding crucial support for HTTP/1.1 features such as robust caching, efficient pipelining, the POST method for submitting data, Digest Authentication for secure access, and deflate compression, all essential for a dynamic and interactive web 36 .

The World Wide Web Consortium (W3C), in its capacity as a standards body, developed the Arena web browser . This wasn’t merely a browser for general use; it was conceived as a critical testbed and validation tool. Its purpose was to rigorously test emerging web technologies, including early iterations of HTML3 , the nascent Cascading Style Sheets (CSS), the then-new Portable Network Graphics (PNG) image format, and, naturally, libwww itself 37 . Arena later passed this torch, and its role was assumed by Amaya , another W3C project that combined browsing with editing capabilities 38 .

A survey conducted in September 2003, long after libwww’s peak, still managed to identify at least 19 distinct applications that were actively utilizing the library 25 . A testament to its lingering utility, or perhaps just to the sheer inertia of existing codebases.

Notable applications that relied on libwww included:

• Agora : An early web browser 39 .
• Arena: The W3C’s experimental browser and testbed 40 ; 41 .
• Amaya : The W3C’s browser and editor, which continued to use libwww for its underlying network functionality 40 ; 41 ; 42 .
• Cello : One of the very first graphical web browsers for Microsoft Windows 43 .
• CERN httpd server: The original web server, developed by CERN 41 .
• Cygwin : A compatibility layer for running Unix-like software on Windows, which included libwww in its early distributions 44 ; 45 .
• Distributed Oceanographic Data Systems with the OPeNDAP : Utilized libwww for data access and transfer in scientific applications 46 ; 47 .
• GRIF Symposia: An early HTML editor that demonstrated the library’s utility beyond mere browsing 48 ; 49 .
• Lynx : The text-based web browser, a testament to libwww’s versatility across different interface paradigms 9 ; 41 .
• MacWWW : The first graphical web browser for Macintosh computers 50 .
• Mosaic : A groundbreaking graphical web browser that popularized the web 9 ; 12 ; 41 .
• Robot Operating System (ROS): A flexible framework for writing robot software, which has, at various points, utilized libwww for network interactions 51 .
• TkWeb: Another early web browser 52 .
• tkWWW : A graphical web browser built using the Tk toolkit 53 .
• WorldWideWeb (later Nexus): The very first web browser and editor, from which libwww’s code was initially derived 54 ; 55 .

Several applications were also integrated directly into the libwww package itself, serving as examples and utilities:

• Command Line Tool: A practical application designed to illustrate how developers could leverage libwww to construct simple batch-mode tools for interacting with the Web 36 ; 56 . Because sometimes, you just need to fetch something without all the graphical fuss.
• Line Mode Browser : A minimalist, text-only web browser, often lauded for its accessibility and simplicity, which was bundled as part of the library 57 .
• Webbot: A straightforward application that demonstrated the use of libwww for building web robots or crawlers, fundamental tools for indexing and automating web tasks 36 ; 58 .
• Mini Server: A small, illustrative application showcasing how libwww could be used to implement a basic web server or a proxy, highlighting its dual utility in both client and server-side contexts 36 .

Criticism

Even a foundational library like libwww, important as it was, did not escape scrutiny. The developers behind libcurl , a more modern and widely used client-side URL transfer library, have openly voiced criticisms regarding libwww’s shortcomings. They specifically pointed out that libwww was “not as portable ” as desired, implying that its codebase presented challenges when adapting it to various computing environments and operating systems 59 . This lack of portability is a significant drawback for any library aiming for widespread adoption, as the digital landscape is notoriously fragmented.

Furthermore, a critical flaw highlighted was its lack of thread-safety 59 . In an era where concurrent processing and multi-threaded applications are the norm, a library that cannot safely be used by multiple threads simultaneously becomes a liability, prone to data corruption and crashes. It’s like building a house with perfectly good bricks, but forgetting to put mortar between them.

Another point of contention raised by the libcurl team was libwww’s deficiency in supporting “several HTTP authentication types” 59 . As the web evolved, robust and varied authentication mechanisms became crucial for securing access to resources. A library that lagged in this area would inevitably fall short in meeting the security demands of complex web applications.

It’s also worth noting, as some developers have pointed out, that neither libcurl nor libwww are universally suitable. For certain highly constrained projects, or those demanding extreme minimalism, both libraries can be considered too “heavyweight.” There are niche applications where even the leanest of these comprehensive libraries is simply too much, requiring developers to craft their own, bare-bones HTTP clients 60 . A reminder that sometimes, the simplest tool is the best, even if it means reinventing the wheel with fewer spokes.

See also

• Free and open-source software portal
• Library for WWW in Perl
• cURL