5. The Semantic Web: A Vision and the Stack It Built

About 27 minutes spoken, 3,913 words. Plain text for a reader app: 05-semantic-web-vision-and-stack.txt

This is the fifth of ten talks about ontologies. Last time we saw the engineering sense of the word emerge from a military-funded programme about reuse, and we met Cyc, the field’s most expensive experiment. Today, the moment the discipline was handed the whole web, and the standards that came out of it. By the end you should understand why the technology has the shape it has.

One caveat first, said out loud rather than buried. Reading the whole stack as a set of answers to one magazine article is my reconstruction. The article does not claim it, and no source I am working from claims it. One major piece, the query language, arrived five years after the article was published, which on its own tells you the story is tidier than the history. Take the reconstruction as a way of remembering the design, not as a chain of cause and effect.

The article appeared in Scientific American in May two thousand and one. The authors were Tim Berners-Lee, who invented the World Wide Web, along with James Hendler and Ora Lassila. It is called “The Semantic Web,” and it opens with a scenario I am going to read you a piece of, because everything follows from it.

The entertainment system is belting out the Beatles when the phone rings. Pete answers, and his phone turns the sound down by sending a message to all the other local devices that have a volume control. His sister Lucy is on the line from the doctor’s office. Their mother needs to see a specialist and then has to have a series of physical therapy sessions, twice a week or so. Lucy says she will have her agent set up the appointments. Pete agrees to share the driving. At the doctor’s office, Lucy instructs her Semantic Web agent through her handheld browser. The agent retrieves information about the prescribed treatment from the doctor’s agent, looks up several lists of providers, checks which of them are in-plan for their mother’s insurance and within twenty miles of her home with a rating of excellent or very good on trusted rating services, and then starts matching available appointment times, supplied by the agents of individual providers through their own websites, against Pete’s and Lucy’s schedules.

Hold that scenario in your head, because I want to inventory what it demands. The inventory is mine rather than the article’s, but I do not think any of it is a stretch.

First, Lucy’s agent talks to the doctor’s system, to several clinics, to rating services, to an insurer’s list, and to two personal calendars. No service was built to talk to any other. So requirement one: data from independent sources must combine without anyone having agreed in advance to combine it.

Second, the agent has to know that the doctor’s “physical therapy” and the clinic’s “physiotherapy” are the same treatment. Requirement two: shared meaning across independent vocabularies.

Third, the agent has to work out that a provider rated “excellent” satisfies a request for “excellent or very good,” which nobody stated. Requirement three: make inferences about information that was left out.

Fourth, and I want to stress this one, the appointment times are supplied by the agents of individual providers through their own websites. Not through a booking platform. Not through an aggregator. Requirement four: decentralisation. That is the article’s own emphasis and not my gloss. Decentralisation is the point of the exercise.

Four requirements. Now watch the standards line up against them.

Requirement one is combining data from strangers. The answer is RDF, the Resource Description Framework, a standard from the W three C, the World Wide Web Consortium, which is the body that publishes almost everything in this lecture. RDF was designed as a metadata data model, and Sinclair Target’s history calls it the grammar in which Semantic Web pages expressed information. What it says is: express everything as three-part statements. Subject, predicate, object. Anna works at the clinic. The clinic provides physical therapy. That is the entire data model. Formally it is a labelled directed multigraph made of subject, predicate and object statements. The version I am describing, RDF one point one, was published in two thousand and fourteen, and in practice a graph like that lives in a triplestore.

Why triples rather than tables? This argument is mine and not the specification’s, so weigh it accordingly. If I hand you two relational databases about the same domain, merging them is a project. You reconcile schemas, resolve key collisions, decide what to do about columns that exist on one side only. If I hand you two sets of triples, you concatenate the files. The union of two graphs is a graph. There is no schema negotiation because there is no schema to negotiate.

RDF also carries the identity half of requirement two. Subjects and predicates are identified by IRIs, Internationalized Resource Identifiers. You will hear the older term URI, Uniform Resource Identifier, at least as often, and the two thousand and one article uses that one throughout. Either way, the idea is a globally unique name, so that when two publishers who have never met both use the same identifier for physical therapy, their statements join up the moment the graphs merge. Objects can also be plain literals, or blank nodes, which are anonymous and not directly identifiable from the statement they sit in.

There is a limit on that trick that undercuts the optimism, and it is stated in the specification rather than by critics. Producers and consumers must agree on what an identifier means, because that agreement is not inherent in RDF. A shared name is not a shared meaning.

Now let me clear up the thing that has confused people for twenty years. RDF is an abstract graph model. It is not a file format. Do not confuse the abstract model with any one serialisation of it. The same graph can be written as Turtle, TriG, N-Triples, N-Quads, JSON-LD, or RDF in XML, the Extensible Markup Language, among others. Turtle, the Terse RDF Triple Language, is the compact one, intended to be more readable and more editable by hand than the XML form, and it is what you would write yourself. JSON-LD, JavaScript Object Notation for Linked Data, encodes linked data in JSON while keeping the shape of traditional JSON, so linked-data meaning can be introduced by modifying documents a web developer already has. Turtle cannot express named graphs on its own, and TriG extends it for exactly that. The XML form is the one OWL requires conformant tools to be able to exchange, which is why it will not die. These are spellings. The graph is the model.

Two of those spellings deserve a moment more, because they are aimed at different people. Turtle’s syntax resembles the query language we will come to, and its job is to abbreviate the identifier-heavy form of a triple: you declare prefixes once, then share a subject or a predicate across several statements instead of repeating it. JSON-LD comes from the other direction. The document carries a context, and that context maps ordinary JSON property names and types onto concepts in an ontology, using the RDF model underneath. The context can sit inside the document or be referenced from outside it, which is how you give linked-data meaning to traditional JSON that nobody wants to rewrite. Two reserved keys supply identity and type, both IRI-based. Values can be coerced to a stated datatype or tagged with a language. And because the identifiers resolve, a client can follow them to discover further RDF. An RDF processor can interpret the mapped properties when it understands the vocabulary, which is the shared-meaning caveat again in different clothes. Where that ended up is worth naming: schema dot org and search-engine work, biomedical informatics, provenance, Activity Streams and ActivityPub, and Thing Descriptions in the Internet of Things.

Requirement two, shared meaning, needs more than identifiers. It needs a way to say what a term means, and that gives us two layers.

The first is RDFS, RDF Schema, and it is deliberately basic. Classes and instances. Transitive class and property hierarchies, subclass and subproperty. Global domains and ranges for properties. Labels and comments for humans. That is close to the whole language, and applications use the domain and range information for inference, for editing suggestions, and for error discovery. If your ontology is a controlled vocabulary with a hierarchy, RDFS does the job.

Two traps come with that simplicity, and both are in the specification’s own list. The first: a domain or a range is global to the property, not local to a class. Say that the property “treats” has a range of patients and that holds everywhere the property is used; RDFS gives you no way to say that inside one class it means something narrower. The second follows from the first. Declaring a domain twice does not narrow anything. It widens it, because multiple declarations mean the subject belongs to all of the declared classes at once. RDFS also does not prescribe what an application ought to do with any of this, and the specification is explicit that where RDF Schema and the RDF Semantics document disagree, RDF Semantics is the authority. That last point is small but it shows you how the stack is assembled. The layers are documents, documents can conflict, and somebody had to write down which one wins.

The second is OWL, the Web Ontology Language, and it is where the ambition lives. OWL two adds keys, property chains, richer datatypes, qualified cardinalities, further property characteristics, and better annotations. Cardinality: this flight has exactly one assigned aircraft. Property chains: the manager of my manager is my second-line manager. OWL two also has profiles, called EL, QL and RL, which restrict what you are allowed to say in order to buy something specific back. Reasoning over very large ontologies. Querying against relational data. Rule-based processing of RDF graphs. Each profile trades expressive power for a computational or implementation benefit, and lecture six explains that trade properly.

While we are here, OWL has a feature that confuses everyone, and it is easier to hear now than to discover later. OWL has two semantics, because an ontology can be viewed either as an abstract structural object or as an RDF graph. The Direct Semantics supports OWL two DL, that is OWL two Description Logic, and description-logic reasoning. The RDF-Based Semantics applies to any OWL two ontology taken as an RDF graph. The description-logic route is the one that buys well-behaved reasoning, and it charges for it in syntactic conditions. For example, a transitive property is not allowed to appear in a number restriction. So OWL is not one language with a quirk. As I read it, it is two views of the same documents, and the seam is visible in the specification. Worth knowing, because when a perfectly reasonable-looking piece of RDF makes your reasoner refuse it, you are standing on that seam.

Two footnotes on OWL, both of which save time later. The first is syntax. I said the XML form is the mandatory one for interchange, and it is, but it is not the only one on offer. OWL has its own XML syntax, a Functional Syntax, the Manchester Syntax, and Turtle, and they exist because processing and reading are different jobs. Different file, same ontology. The second footnote is a caveat on the RL profile. Rule-based reasoning in RL, run over an arbitrary RDF graph, is sound but may be incomplete unless the ontology satisfies RL’s structural definition and the query conditions that come with it. You can trust what it tells you. You cannot assume it told you everything. And the OWL document I have been taking the shape of the language from is itself an overview, informative rather than the normative definition, which is spread across several core specifications.

Requirement three, inferring what was never stated, is what OWL is for and what a reasoner does. A reasoner computes class consistency, subsumption, and instance retrieval. If the ontology says every physiotherapist is a healthcare provider, and the data says Anna is a physiotherapist, then Anna is a healthcare provider, and nobody had to write it down. Target’s history puts RDF Schema and OWL in exactly that role: they are how you make inferences about omitted information.

Then there is the piece the article did not plan for, which is querying. It turns out that once you have a large graph, most of what people want is to ask it questions rather than derive theorems. SPARQL is the answer, and the acronym stands for SPARQL Protocol and RDF Query Language, which contains itself, and tells you something about the era. In its version one point one it is the W three C’s query language for RDF graphs. Triple patterns are RDF triples with variables in them, and a basic graph pattern matches when you can substitute terms for those variables and get an equivalent subgraph. On top of that you get optional matches, alternatives, negation, and property paths, which express traversal compactly, including paths of arbitrary length. Queries run against a dataset with one default graph and any number of named graphs. And there are four result forms: SELECT returns variable bindings, CONSTRUCT builds a new graph from a template, ASK reports whether a match exists at all, and DESCRIBE returns a description graph whose shape the service decides, not your query. It works over data stored as RDF or merely exposed as RDF through middleware.

Three things the neat summary leaves out. First, shaping the answers. Beyond matching you get assignment, inline tables of values, aggregation, grouping, subqueries, expressions, and the sequence modifiers: ordering, deduplication, offset and limit. Ordering matters more than it sounds, because a pattern match initially produces its solutions in no specific order at all. Deterministic presentation is something you ask for. So a query that came back in a stable order all through testing and shuffles itself in production is doing exactly what the specification says it may do.

Second, federation. Querying across several endpoints, which is what the decentralised story actually requires, is not in the query language recommendation. It is defined in a separate SPARQL one point one recommendation of its own. The boundary is drawn explicitly, and I will only note where it falls: the central act of the two thousand and one scenario, an agent asking many independent providers at once, sits outside the document that defines how you ask.

Third, and this genuinely surprises people, a query language has an attack surface. A query that names the graphs it wants may dereference those identifiers, which means it spends network, disk and processor time on your behalf at addresses you did not choose. The specification warns about denial of service, about access to local files, about attacks that use the query service to reach through a firewall, about risks specific to whatever extensions an engine has added, and about identifiers containing Unicode characters that look like other characters. Every one of those follows from the design choice we were admiring earlier. The identifiers are global, and global means the thing at the far end belongs to somebody else.

And there is a layer that arrived later than the rest and that, in my experience, matters more than its position in the stack suggests. Validation. SHACL, the Shapes Constraint Language, and its cousin ShEx, Shape Expressions. SHACL describes RDF graphs by constraining their content, their structure and their meaning. Node shapes constrain nodes. Property shapes constrain the values you reach along a path. Constraints state things like datatype, minimum count, length, ranges, patterns, and logical combinations. Targets say where a shape applies, and targeting a class also targets the members of its subclasses. You run it with a data graph and a shapes graph, and what comes back is an RDF validation report, with severities: Violation, Warning, Info.

Now, why is that a separate technology? Why can OWL not do it? Because OWL and SHACL answer opposite questions, and this is one of the most useful distinctions in the field.

OWL says: here are my premises, tell me what follows. If the ontology says every order has a customer, and I show you an order with no customer, OWL concludes that there exists a customer I have not told you about yet. It does not complain. It infers.

SHACL says: here are my expectations, tell me what is broken. Same situation, and the report names a violation: this node has zero values for the customer property, and the shape requires at least one.

The reason OWL behaves that way is stated plainly by Horrocks, Patel-Schneider and van Harmelen. OWL, they write, “adopts the standard logical model of an open world assumption: a statement cannot be assumed true on the basis of a failure to prove it. Clearly, on the huge and only partially knowably World Wide Web this is the correct assumption.” That is the whole explanation. OWL was built for the web in the scenario we opened with, where the fact you are missing is probably sitting on somebody else’s server. SHACL was built for a graph you control, where a missing fact is a bug.

One honest complication before we move on. I have just given you a slogan, and the specification is less tidy. SHACL is not only constraints. Its built-ins can be extended through SPARQL or through JavaScript, and it carries SHACL Rules, a mechanism for inferring new statements. So the validation language infers too. My reading is that this does not collapse the distinction, since inferring under a closed set of shapes you wrote yourself is a different act from inferring under an open world. But if somebody tells you the whole difference between OWL and SHACL is that one infers and the other checks, they have given you the slogan and not the specification.

And one trap, because it is easy to get wrong. A property shape can carry its own target, saying where it applies. Include that same property shape inside a node shape and its own target is ignored, and the node shape decides. Placement changes targeting behaviour, and nothing warns you. The shapes simply stop firing where you expected them to. The other half of that lesson is what the output is for: the report carries severities and messages so that a person can be told what failed and what to do about it. A validation report is addressed to a human, not just to a build script.

So: RDF for merging, global identifiers for identity, RDFS and OWL for meaning, reasoners for inference, SPARQL for questions, SHACL for validation. Six pieces, and five of them line up against the four requirements. The sixth, SPARQL, does not. It postdates the article by five years and answers a need the scenario never mentions. I point that out because the tidy version of this story gets told a great deal, including by me, ten minutes ago.

Now, what happened to the vision. Lecture eight does the criticism properly, so this is only the shape of the decline, and I am taking it from Sinclair Target’s retrospective of two thousand eighteen, which sets it out in four phases.

The first, from two thousand and one to two thousand and five, was the golden age of Semantic Web activity, and the W three C issued the first version of the RDF standard in two thousand and four. The second phase shifted from setting standards and building toy examples to creating and popularising large RDF datasets, which is linked open data. The third adapted the standards to the actual practices and preferences of web developers, and that is where JSON-LD and schema dot org came from. The fourth is the W three C’s own retrenchment, folding the work under the heading of Data Activity. Somewhere along the way Berners-Lee began referring to the Semantic Web as Web three point oh.

Target’s diagnosis of the sequencing is the part I would keep, and it generalises well outside this field: standards before applications. A committee that specifies before anybody ships produces artifacts nobody adopts.

The other thing that happened is that the agent layer centralised, into Google, into Yelp, into Siri, so that a provider advertises to the platform rather than from its own site. That is the exact inversion of the scenario we started with. In the article the clinic publishes and the agent finds it. In practice the clinic files its details with a platform and hopes.

Look back at requirement four. Decentralisation. That is the requirement that died, and my own view is that it died to economics rather than to any technical defect, because the other three were satisfied by standards that work.

There is also an objection aimed at the vision itself rather than at what became of it. Lecture eight is built on that objection, so today I only put it on the table. The vision assumed that publishers would annotate their pages voluntarily, accurately, and at scale, with no direct incentive to do any of it. Target’s summary of the counter-argument is that most web users were likely to provide either no metadata at all, or else lots of misleading metadata meant to draw clicks. That single assumption, that people will describe their own data honestly and for nothing, is the point on which most later criticism turns. Notice, and the emphasis here is mine, that it is not a criticism of RDF, or of OWL, or of any specification we have looked at today.

One more observation about where the work went, and this one comes from the source rather than from me. The two thousand and one plan was to make the data structured enough that a simple agent could act on it. The current plan is to make the agent capable enough to act on unstructured data. Same scenario, same errand at the doctor’s office. The layer doing the work moved.

But the machinery survived, and the retrospectives themselves name where: schema dot org, knowledge graphs, Wikidata, DBpedia, and the biomedical ontologies. Wikidata is the clearest case. It is a free knowledge base that humans and machines both read and edit, it exports in standard formats, it has a public SPARQL service, and its WikiProjects are where the modelling and quality decisions actually get made. What every survivor has in common is that it is a governed, funded, bounded domain where somebody has an incentive to curate. The open web is none of those things. So when you next evaluate an ontology proposal, ask which of those two situations it resembles.

So, the claim to keep.

Every awkward feature of these standards is an answer to a requirement in the two thousand and one article, with the honest caveat that the mapping is a reconstruction and that the query language does not fit it. Triples because data from strangers must merge. Global identifiers because merging needs shared identity. OWL because agents must infer what nobody stated. SHACL because inference is not validation. And most of those requirements assumed an open web of strangers, which is the assumption that did not hold.

Next time, we open the box and find out how the reasoning actually works, and meet the one assumption that breaks everybody’s intuition the first time.