Data privacy and accountability

My statement for todays’ panel on privacy. For today’s panel, I want to talk about data privacy in the context of the notion of accountability.

Imagine you browse the web, looking for shoes. For the weeks to follow, whenever you visit a web page, adverts of shoes will be presented to you.

Have you ever asked yourself why these adverts are shown to you, who has information about you, what information they have about you, and how did they decide to serve this advert to you?

A system able to answer such why/who/what/how questions is accountable. Being accountable means being able to provide explanations or justifications for decisions and actions.

To be able to provide accountability, there is a need to be able to trace flows of data (traceability), tracing data across systems enables explanations to be provided about the transformations, operations, and decisions made about such data.  Several names are available for such notion, traceability or provenance.  Provenance of a decision helps explaining factors that affected the decision, data involved in it, etc. The word is common for food: provenance of food is a sign of its quality; likewise, provenance of a piece of art enables its authenticity to be asserted. Over the last 15 years, I have been leading research activities around provenance of data, and led a standardisation activity for provenance on the web.

The European GDPR General data protection regulation coming in 2018 has a component dubbed the “right to explanation”. There are still some uncertainty about what it entails both legally and technically.

What has it got to do with privacy? Privacy and accountability have an interesting relation that I want to discuss.

Consider expense claims, a topic well understood by this audience. Imagine that Alice and Bob have a business meeting conducted over a meal. Bob has to make his expense claim public.  This may indirectly make the presence of Alice at the restaurant’s location public. Alice’s privacy is in tension with Bob’s accountability/transparency requirement.

So, there is a tension between privacy and accountability. 100% private doesn’t give you accountability, 100% accountable doesn’t give you privacy.

Privacy is important, so is accountability! These are values that we want to promote Technically, legally and as a society, we are still learning to understand these values and how they should be protected.






PROV-Template: A Quick Start

The aim of this blog post is to provide simple guidelines to generate provenance using the PROV-Template approach.

A quick reminder:  a provenance template is a PROV document, describing the provenance that it is intended to be generated. A provenance template includes  some variables that are placeholders for values. So,  a provenance  template can be seen as a declarative specification of the provenance intended to be generated by an application.   A set of bindings contains associations between variables and values. The PROV-template  expansion algorithm, when provided with a template and a set of bindings, generates a provenance document, in which variables have been replaced by values.

Therefore, three steps are involved in this methodology.

  1. Design a “provenance template” describing the structure of the provenance intended to be generated.
  2. Instrument the application, log values, and create “binding files” from these values.
  3. Produce provenance by expanding the template using binding files.

We consider a simple computation, which we would like to describe with provenance.   The computation consisted of 3 calls of binary functions: the functions were composed in such a way that the results of two calls were used by the third one. To simplify, we assume that the operations were arithmetic +, -, and *, and the values flowing and out of these operations were integers. Note my use of past tense: the aim of provenance is to describe past computation, as opposed to a future, hypothetical computation (or workflow).


As we have 3 binary functions, we design a template describing the invocation of a binary function.  It consists of an activity (denoted by variable operation), two used entities (denoted by variables consumed1 and consumed2), a generated entity (denoted by variable produced), and an agent (denoted by variable agent) responsible for the activity.  Graphically, the template can be represented as follows.

Template for the invocation of a binary function

Using the PROV-N notation, the template is expressed as follows.  We see that variables are declared in the namespace with prefix var. Each entity and activity is associated with a type, expressed by a variable, which can also be instantiated.

 prefix tmpl <>
 prefix var <>
 prefix vargen <>

 bundle vargen:b
  activity(var:operation, [ prov:type='var:operation_type' ] )
  used(var:operation, var:consumed1, - )
  used(var:operation, var:consumed2, - )  
  wasGeneratedBy(var:produced, var:operation, - )
  wasDerivedFrom(var:produced, var:consumed1)
  wasDerivedFrom(var:produced, var:consumed2)

To be able to generate provenance, one needs to define so-called “bindings files”, associating variables with values. The structure of bindings file is fairly straightforward: with the most recent version of the ProvToolbox, a bindings file can be expressed as a simple JSON structure. Such JSON structures are very easy to generate programmatically from multiple programming languages. However, in this blog post, we do not want to actually program anything in order to generate provenance.

Therefore, we are going to assume that the application already logs values of interest. We are further going to assume that the data can be easily converted to a tabular format, and specifically, that a CSV (comma separated values) representation can be constructed from those logs. The structure that we expect is illustrated in the following figure. In the first line of the file, we find variable names (exactly those found in the template) acting as column headers. In the second line, we find the type of the values found in the table.

Application log as a CSV file. First line contains variable names whereas second line contains the type of their values. Subsequent lines are the actual values.

Concretely, the CSV file uses commas as separator. The third, fourth, and five lines contain deftail of the invocations of the plus, times, and subtraction functions.

operation, operation_type, consumed1, consumed_value1, consumed2, consumed_value2, produced, produced_value, agent
ex:op1, ex:plus, ex:e1, 10, ex:e2, 11, ex:e3, 21, ex:Luc
ex:op2, ex:times, ex:e4,  5, ex:e5,  7, ex:e6, 35, ex:Luc
ex:op3, ex:subtraction, ex:e3, 21, ex:e6, 35, ex:e7, -14, ex:Luc

Each line can automatically be converted to a JSON file. For instance, the third line containing the details of the addition operation can be converted to the following JSON structure, which is essentially a dictionary associating each variable with its corresponding value, with an explicit representation of the typing information where appropriate.

   {"operation": [{"@id": "ex:op1"}],
    "operation_type": [{"@id": "ex:plus"}],
    "consumed1": [{"@id": "ex:e1"}],
    "consumed_value1": [ {"@value": "10", "@type": "xsd:int"}],
    "consumed2": [{"@id": "ex:e2"}],
    "consumed_value2": [{"@value": "11", "@type": "xsd:int"}],
    "produced": [{"@id": "ex:e3"}],
    "produced_value": [ {"@value": "21", "@type": "xsd:int"} ],
    "agent": [{"@id": "ex:Luc"}]},
 "context": {"ex": ""}

We do not need to create this JSON structure ourselves. Instead, we provide an awk script that converts a given line into a bindings file.

function ltrim(s) { sub(/^[ \t\r\n]+/, "", s); return s }
function rtrim(s) { sub(/[ \t\r\n]+$/, "", s); return s }
function trim(s)  { return rtrim(ltrim(s)); }

NR==1 {                                # Process header
    for (i=1;i<=NF;i++)                
        head[i] = trim($i)                  
NR==2 {                                # Process types
    for (i=1;i<=NF;i++)                
        type[i] = trim($i)             
    for (i=1;i<=NF;i++) {              # For each field
	if (first) {
	} else {
	    printf ","
	if (type[i]=="prov:QUALIFIED_NAME") {
	    printf "\"%s\": [{\"@id\": \"%s\"}]",  trim(head[i]), trim($i)
	} else if (type[i]=="xsd:string") {
	    printf "\"%s\": [ \"%s\" ]",  trim(head[i]), trim($i)
	} else  {
	    printf "\"%s\": [ {\"@value\": \"%s\", \"@type\": \"%s\"} ]",trim(head[i]), trim($i), trim(type[i])
    printf "\n"                        
    printf("\"context\": {\"ex\": \"\"}\n")

To facilitate the processing, we even provide a Makefile with a target do.csv that processes a line (variable LINE) of the csv file to generate a bindings file. It is then used by the utility provconvert to expand the template file. The target workflow hard-codes the presence of three lines in the CSV, the generation of a bindings file for each line, and the expansion of the template with these bindings. All files are then merged in a single provenance file using the -merge option of provconvert.


	cat bindings.csv | awk -v line=$(LINE) -f src/main/resources/awk/tobindings.awk  > target/bindings$(LINE).json
	provconvert -bindver 3 -infile template_block.provn -bindings target/bindings$(LINE).json -outfile target/block$(LINE).provn

	$(MAKE) LINE=3 do.csv
	$(MAKE) LINE=4 do.csv
	$(MAKE) LINE=5 do.csv
	printf "file, target/block3.provn, provn\nfile, target/block4.provn, provn\nfile, target/block5.provn, provn\n" | provconvert -merge - -flatten -outfile target/wfl.svg

The resulting provenance is displayed in the following figure.

Expanded provenance showing three activities, consumed and generated entities, and an agent.


Concluding  Remarks

Given a log file in CSV format, we have shown it is becoming easy to generate PROV-compliant provenance without having to write a single line of code: an awk script converts CSV data to JSON, used to expand a template expressed in a PROV-compliant format.

For the provenance to be meaningful, the application must be instrumented to log the relevant values. For instance, each entity/agent/activity is expected to have been given a unique identifier.

The template design phase is also critical. In our design, we decided that one template would describe the invocation of a single function. The same template was reused for all function calls. Alternatives are possible: multiple activities could be described in a single template, alternatively different types of activities could be described in different templates. I will come back to this issue in another blog post in a few weeks.

Principles for Algorithmic Transparency and Accountability: A Provenance Perspective

A few days ago, the ACM U.S. Public Policy Council (USACM) released a statement and a list of seven principles aimed at addressing potential harmful bias of algorithmic solutions. This effort was initiated by the USACM’s Algorithmic Accountability Working Group.  Algorithmic solutions are now widely deployed to make decisions that affect our lives, e.g., recommendations for movies, targeted ads on the web, autonomous vehicles, suggested contacts or reading in social networks, etc.  We have all come across systems making decisions that are targeted to us individually, and I am sure that many of us have wondered  how a given recommendation was made to us, on the basis of which information and what kind of profile. Typically, no explanation is made available to us!  Nor there is any means to track the origin of such decisions!

Interestingly, emerging regulatory frameworks, such as the EU General Data Protection Regulation, are introducing the “right to explanations” (see in particular related to Article 22 on Automated individual decision-making, including profiling. So, the regulatory framework is evolving, even though there is still no consensus on how to actually achieve this in practice.

Furthermore, algorithmic bias is a phenomenon that has been observed in various contexts (see for instance two recent articles of the  New-York Times and the Guardian). Given their pervasive nature, ACM U.S. Public Policy Council acknowledges that it is imperative to address “challenges associated with the design and technical aspects of algorithms and preventing bias from the onset”.   On this basis, they propose 7 principles, compatible with their code of ethics.

As a provenance researcher, I have always regarded the need to log flows of information and activities, and ascribe responsibility for these as crucial steps to making systems accountable. This view was echoed by Danny Weitzner and team in their seminal paper on Information Accountability.  I was therefore delighted to see that “Data provenance” was listed as an explicit principle of the USACM list of seven principles. So, instead of paraphrasing them, I take the liberty of copying them below.


Figure 1: ACM US Public Policy Council list of seven principles for Algorithmic Transparency and Accountability



However, I feel that provenance, as understand it, encompasses several of these principles, something that I propose to investigate in the rest of this post.  To illustrate this, I propose Figure 2, a block diagram outlining the high-level architecture of a transparent and accountable system.  At the heart of such a system, we find its Business Logic which provides its primary functionality (e.g. Recommendations, Analytics, etc).  In provenance-aware systems, applications log their activities and data flows, out of which a semantic representation is constructed, which I refer to as provenance. PROV is a standardised representation for provenance, which was recently published by the World Wide Web consortium and seeing strong adoption in various walks of life.  In this context, provenance is defined as “a record that describes the people, institutions, entities, and activities involved in producing, influencing, or delivering a piece of data or a thing”.

There is no point constructing such a semantic representation, if it is not being exploited. Various capabilities can be built on top of such a provenance repository, including query interfaces, audit functionality, explanation service, redress mechanism and validation, which we discuss now in light of the seven principles.


The Role of Provenance in the Architecture of an Accountable System

Figure 2: The Role of Provenance in the Architecture of an Accountable System


The first principle (Awareness) identifies a variety of stakeholders: Owners, Designers and Builders, Users, but the second principle also mentions the role of Regulators, and we believe that potential third-party Auditors are also relevant in that context.  While technology makes progress with algorithmic solutions, society is much slower to react, and there is indeed work required to increase awareness, and establish what the user rights are, and what the obligations on owners should be, whether by means of regulations or self-regulations. The SmartSociety project recently published a Social Charter for Smart Platforms, which is an illustration of what rights and obligations can be in “smart” platforms. 

The second principle (Access and Redress) recommends mechanisms by which systems can be questioned and redress enabled for individuals.This principle points to the ability to query the system and its past actions, which is a typical provenance-based functionality. For those seeking redress, there is a need to be able to refer to an event that resulted in an unsatisfactory outcome; PROV-based provenance mandates that all outcome, data and activity instances are uniquely identified.  Furthermore, we are of the view that such a redress mechanism, including reached resolutions, should be inspectable in a similar fashion; thus, provenance of redress requests and resolutions should also be inspectable.

The third principle (Accountability) is concerned with holding institutions responsible for the decisions made by their algorithmic systems.   For this, one needs a non-repudiable account of what has happened, and suitable attribution of decisions to system components, their owners, and those legally responsible for the system’s actions. Again, such an account is exactly what PROV offers: therefore we see the third principle being implemented technically with queries over provenance representation, and socially with suitable regulatory and enforcement mechanisms.

The fourth principle (Explanation) requires explanations to be produced about the unfolding of activities and decisions.  There is emerging evidence that provenance can serve as a form of computer-based narrative, out of which textual explanations can be composed and presented to users.  We recently conducted some user studies about the perceived legibility of natural language explanations by casual users.  We also used a similar technique in order to provide explanations about user ratings in a Ride Share application.

The fifth principle (Data Provenance) is explicitly focusing on training data used to train so-called “machine-learning” algorithms. We believe that it is not just training data that is relevant, but any external data, the business logic and designers may rely upon. It is expected that public scrutiny of such data offers opportunity to correct potential bias, and in general, any concern that may affect decisions. To operationalize this principle, one needs to have access to a description of the data (potentially, the data itself), but also how it is used in training algorithms, and how this potentially affects decisions. PROV-based Provenance, queries and explanations are required here to allow such scrutiny. Some of our recent work focused on analytics techniques to assess the quality of data, using provenance information; such a mechanism becomes useful to ensure some form of quality control in systems.

The sixth principle (Auditability) demands models, algorithms, data, and decision to be recorded, so that they can be audited. All these can easily be described in PROV, by means of “PROV entities“, which can be used or generated by “PROV activities“, under the supervision of responsible agents. Specific auditing functions (aimed at various stakeholders) can query the provenance to expose individual entities, but also their aggregate characteristics, over longer periods of time. Techniques that we have developed, such as provenance summarisation, become really critical in this context, since they enable us to investigate aggregate behaviour of applications, instead of individual circumstances.

The seventh principle (Validation and Testing)  recommends regular validation of models and testing for harmful outcomes.  This suggests that processing over provenance, checking whether  some expected criteria has been met or not, can be implemented by policy-based approaches over provenance, detecting whether  past executions comply with expectations, described as policies. We have applied this technique to decide whether processing was performed in compliance with usage policies. If this is good practice to undertake validation and testing, therefore, it also becomes a necessity to document such a practice, to be able demonstrate that such validation and testing takes place.

So overall, the provenance research community has been investigating issues around capturing, storing, representing, querying and exploiting provenance information, all of them having a critical role in the principles of Algorithmic Transparency and Accountability.  There is still much to research however, including critical issues around (1) agreed domain-specific extensions of PROV to support transparency and accountability; (2) better integration of the software engineering methodologies with provenance; (3) enforceable compliance with architecture; (4) non repudiation of provenance; (5) querying and auditing facility; (6) compliance checks over provenance; (7) user-friendly explanation of complex algorithmic decisions; (8) scalability of all the above issues.

In the spirit of Principle 1,  I hope this blog post contributes to raising awareness of these issues. Feedback and comments welcome!

What is in ProvToolbox 0.7.3?

Today, I released ProvToolbox 0.7.3. The  principal changes in this new version of ProvToolbox are concerned with prov-template, the templating system for provenance. The new release also contains few minor bug fixes and changes.

1. Template System

A reminder: a PROV-template is a PROV document, in which some variables are placeholders for values. A PROV-template is a declarative specification of the provenance intended to be generated by an application.   A set of bindings contains associations between variables and values. The PROV-template  expansion algorithm, when provided with a template and a set of bindings, generates a provenance document, in which all variables have been replaced by values.

PROV-template is a new approach to creating a provenance-enabled application. Templates are designed and embedded in the application’s code, the application logs values (in the form of bindings), and provenance is automatically generated by template expansion.

A tutorial for templates is available on this blog:

In ProvToolbox 0.7.3, we have adopted a more compact and user-friendly representation for sets of bindings. Instead of representing them as PROV, we can now represent them as JSON.  At the same time, we also handle variables in a more uniform manner, allowing variables occurring in mandatory position, to be also used in attribution position. I won’t go into the technical details, but these two changes make the design of templates and the construction of bindings  much simpler!

A further change is that we have implemented a simple “bindings bean” compiler: it takes a template definition and creates a java class, which allows sets of bindings to be created directly from Java, and serialized easily.  The aim of this compiler is to simplify the implementation of applications generating provenance.

The GitHub source code repository contains code for two further tutorials (Tutorial5 and Tutorial6). I will write up the text for these tutorials in the New Year.

2. Qualified Pattern for All PROV Relations

At the recent PROV: Three Years Later Workshop, I made the case for the Qualified Pattern  to be used for all PROV relations. My key motivation for this extension to PROV is my provenance summarisation algorithm, which generates a “summary provenance graph“, in which nodes and edges are annotated with weights indicating how frequently these kinds of nodes and edges  can be found in the original graph. To allow for such annotations to be added to specialization, alternate, and membership relations, they need to support the Qualified Pattern.

At this stage, it is the data model that is modified. Serialization to xml and provn is work in progress, and not supported in prov-json and prov-sql yet. Furthermore, there is no parsing yet. Three new interfaces have been defined in the package org.openprovenance.prov.model.extension.

3. Release Log

For full details of the changes, see the release log at

4. Conclusion

We keep on using ProvToolbox in various applications to generate provenance with templates and to undertake some analytics using the summarisation algorithm. This new release was critical to support these two use cases of ProvToolbox. Shortly, I will release two further blogs with new tutorials for prov-template.

As always, all relevant links can be found at, including binary installers for linux (rpm and debian) and macosx.

Seasonal greetings!



PhD studentships available

Are you interested in undertaking PhD studies in the area of provenance? Are you intrigued by some of the following topics?

  • Provenance architecture for the Internet of Things
  • Big data analytics of provenance
  • Streamed provenance
  • Tradeoff between provenance and privacy
  • Unforgeable provenance and blockchain

Contact me and we can discuss a PhD topic. We have fully-funded studentships for DTA-eligible students (see  details

Building 32 hosts the Web and Internet Science Group

Building 32 hosts the Web and Internet Science Group

What is in ProvToolbox 0.7.2?

1. Introduction

Yesterday, I released ProvToolbox 0.7.2, which includes the following novel features.

2. Novel Features

2.1. MacOS X Installer

Continuing our efforts of providing binary installers to facilitate installation of ProvToolbox, this release includes an installer for MacOS X.

Simply follow the link, you will then be given access to the installation image.

Installation Disk

Installation Disk

Click on the Installer. Note that you need to allow installation of programs from any sources in your security preferences. Then simply follow the instructions. The installer will install all libraries and executable in /Applications/provconvert (default location, which can be overriden), as well as a symbolic link making the provconvert executable available in your execution path. An Uninstaller is also available as an executable jar file /Applications/provconvert/Uninstaller/uninstaller.jar.

provconvert Installer

provconvert Installer

Et voila! The executable can be invoked directly from the command line.

provconvert -version

which should return provconvert version 0.7.2 (2015-09-15 20:16).

2.2. Templates

As we continue to use templates in our applications, two further requirements have been implemented. It is now possible to expand a template, and strip the result from any variable that has not been instantiated. For this, simply pass the option -allexpand to provconvert, to be used in conjunction with the -bindings option (see Tutorial 4 (part 1) and Tutorial 4 (part 2) on template processing in ProvToolbox). Furthermore, an error code is returned when not all variables have been expanded.

2.3. Interoperability

As we are integrating Provtoolbox, ProvStore and ProvStore in the inter-operability harness developed by the Software Sustainability Institute, we have fixed some minor issues to ensure interoperability between our software stacks.

2.4. provconvert artifact

The artifact toolbox has been renamed into provconvert, since we have plans for other artifacts out of ProvToolbox.

3. Conclusion

For all details about ProvToolbox, see the page

What is in ProvToolbox 0.7.1?

1. Introduction

Yesterday, I released ProvToolbox 0.7.1. It is a minor release, fixing minor bugs of 0.7.0, and including a useful new feature.

2. Novel Features

2.1. Debian Package

To facilitate installation, a new binary release format is now supported: Debian packaging to support binary release on Ubuntu and other Debian-based Linux distributions. You just need to run the following commands.

dpkg --install toolbox-0.7.1.deb

This is in addition to RPM support introduced in 0.6.2:

rpm -U

2.3 Visualization

Modification of the visualisation component prov-dot allow dge thickness, node size, and tooltips (on SVG) to be controlled. For this, the provenance graph nodes and edges need to be annotated with reserved attributes dot:size and dot:tooltip. The following figure illustrates the kind of graphs that can now be generated.

A summarisation of the provenance challenge workflow. Nodes are to be understood as provenance types. Thickness of edges and size of nodes reflect their frequency in the summarised document.

A summarisation of the provenance challenge workflow. Nodes are to be understood as provenance types. Thickness of edges and size of nodes reflect their frequency in the summarised document.

2.3 Bug fixes

I also fixed some minor bugs in qualified namespaces in the prov-sql package, and updated reserved namespace for provtoolbox.

3. Conclusion

Tell me how you use ProvToolbox and/or provconvert and for for which purpose. Share details of your projects with me, I will add them to

For all details about ProvToolbox, see the page