Difference between revisions of "Contextual Data"

From ECRIN-MDR Wiki
Jump to navigation Jump to search
(Geographical Data)
(Geographical Data)
Line 21: Line 21:
 
* 6  County (The second tier administrative area in the country - e.g. county, départment)
 
* 6  County (The second tier administrative area in the country - e.g. county, départment)
 
* 7  Mixed
 
* 7  Mixed
A section of the geog_entities is shown below:
+
A section of the geog_entities is shown in Figure 1:
 
+
Figure 1 provides an image of a few of the rows in the source_data_studies table.<br/>
 +
[[File:countries.png|center||]]
 +
<div style="text-align:center">'''Figure 1:''' Sample lines from the ctx.geog_entities table</div>
 
The geog_groupings table is designed to support aggregation of geographically related data. A small section of the table is shown below:
 
The geog_groupings table is designed to support aggregation of geographically related data. A small section of the table is shown below:
  

Revision as of 18:05, 17 November 2020

Introduction

All the systems and databases within the MDR need to access so called 'contextual' data, which is of two main types:

  • Context data: Data relating to the 'landscape' in which clinical research is carried out - organisations and people, countries and regions, systems for classifying keywords and topic etc.
  • Controlled terminology: Data relating to the options available for categorised questions within the MDR itself.

Rather than replicate such data in every database, all of these data are stored in a separate context database, in the ctx and lup (look up) schemas respectively.
Postgres does not allow statements made in one database context to reference objects in another DB directly, so when processes in one of the source data databases require access to contextual data, one or both context schemas are imported into the database within a 'foreign table wrapper'. This allows the data, but not the table designs, to be read and manipulated. It does risk, however, cluttering up the databases with additional schema, so within the MDR such foreign table wrappers are always created on a temporary basis, when required, and are torn down again once the processing is completed (see the Managing Links between databases section in Data_Structures).
The sections below provide further details on the various types of contextual data in the system.

Contextual Data

Organisation Data

Geographical Data

Geographical entities - countries, regions, continents, states, etc. - are referenced both within the main MDR, as a possible topic or keyword applied to some studies or papers, and within the contextual data , because organisations have a 'geographic scope'. They can be entered into the system under a variety of different names (US, U.S., USA. United States, United States of America, etc.) and it is therefore better to standardise and code them where possible.
The contextual data includes three tables - geog_entities, geog_entity_types, and geog_groupings - that list the countries and their codes, list the various types of entity available, and list the members of a composite entity (e.g. the states inside a country). The entity types are as follows:

  • 0  System
  • 1  Global
  • 2  Continental
  • 3  Regional (country grouping)
  • 4  National
  • 5  State (The largest of the administrative region types within a country, e.g. state, province, devolved nation, land)
  • 6  County (The second tier administrative area in the country - e.g. county, départment)
  • 7  Mixed

A section of the geog_entities is shown in Figure 1: Figure 1 provides an image of a few of the rows in the source_data_studies table.

Countries.png
Figure 1: Sample lines from the ctx.geog_entities table

The geog_groupings table is designed to support aggregation of geographically related data. A small section of the table is shown below:

Topic Data

Publisher Data

The context schema includes three tables that relate print issns (pissns), electronic issns (eissns) and journal names to a publisher id, and a fourth table that gives the publisher name against that id. The data is largely derived from CrossRef, although the publisher's id is derived from its entry in the organisations table. The tables are required because PubMed citations identify the journal name, and its pissn and / or eissn, but not the publisher. The publisher is usually the managing organisation for a journal paper - in the sense that it is the publisher rather than the editorial board that decides whether a journal or paper is open access or not. The publisher's identity is therefore an important part of the metadata for a published paper. These four look up tables are therefore used to identify the publisher.

People Data

Ideally, there would be a 'contextual' list of people that would relate people's names, titles and affiliations to ORCID identifiers, and only the ORCID Ids would be used within the system. Unfortunately only a very small proportion of the names in the source data (as study leads or contacts, or as journal paper authors) come with ORCID identifiers. Names come in a variety of different formats - sometimes with titles, sometimes with first name rather than initials, sometimes with the family name before the given name - and there is no obvious pattern or consistency to this, even within a single source. In general, therefore, standardisation and rationalisation of name and affiliation data is not possible in the MDR, and such data remains largely as it is presented in the source data (though titles are dropped when they can be identified). There are no plans at the moment to create a contextual resource for people, unless and until adoption of ORCID identifiers becomes much more widespread.

Controlled Terminology

Many of the data fields in the MDR system are categorised, i.e. they are constrained to hold one of a predefined set of possible values.
For example, a clinical study may fall into one of several different main ‘types’, according to the broad methodology used:

  • Interventional
  • Observational
  • Observational [Patient Registry]
  • Expanded access
  • Funded programme

Although clinical studies are usually categorised in similar ways, different systems can use different terms for those categories. Using the MDR’s controlled terminology, studies labelled as ‘clinical trial’, ‘randomised trial’, or ‘active intervention’ would all be mapped to a study type of ‘Interventional’, while other studies, described variously as ‘off-label’, ‘compassionate use’, or ‘pre-licence’, would all be mapped to ‘Expanded Access’. The options available for any categorised data item are stored in a 'look up' table.

Structure of Look Up tables

The MDR system uses a collection of these ‘look up’ tables. In the context of a user interface, each lookup table would often be used as the source of a dropdown or list box that displayed the categories for search or filtering purposes. Each look up table lists the options available for that data field, and almost all have a core common structure:

  • An integer id field, that provides a key for each option. The database only has to store the id to represent the category being used, making storage much more efficient.
  • A string name field, with the short name of the category, as it would appear – for instance – in a dropdown box that displayed the alternatives available.
  • A text description field, that provides, when necessary, a brief explanation of what the category meant.
  • An integer list_order field, which allows the default order in a display or report (normally that of the id, as the primary key) to be over-ridden, by ordering the records according to the value of this field.

A few look up tables have one or two additional fields but the great majority follow the pattern above.
The categories used are normally as already exist within DataCite or other key systems, augmented if necessary by ECRIN to better cover the full range of categories required for clinical research data objects. For instance the listed study types are taken from the study type classification found within the ClinicalTrials.gov trials registry. The lookup tables contain two additional ‘audit’ field:

  • A string source field, denoting the system from which the term was originally taken. Terms created by the MDR team are labelled as ‘ECRIN’
  • A date date_added field, that indicates the date a category was agreed as being required by the MDR system (though full implementation may be later). If a term has been proposed but not yet agreed this date is null.

The study_types table, below, illustrates how this structure works in the case of study categories.

id name description list_order source date_added
0 Not yet known Dummy value supplied by default on entity creation. 99 ECRIN 2019-02-08
11 Interventional A clinical trial. 10 ClinicalTrials.gov 2019-02-08
12 Observational Any form of non-interventional research. 20 ClinicalTrials.gov 2019-02-08
13 Observational Patient Registry Collecting data for a designated registry. 30 ClinicalTrials.gov 2019-02-08
14 Expanded access Off label usage of a new product for individuals. 40 ClinicalTrials.gov 2019-02-08
15 Funded programme With a single or linked series of grants. 50 ClinicalTrials.gov 2019-02-08

Listing of Look Up tables

At present the MDR system contains 29 look up tables, with most indicating that they refer to categorised fields by their name, usually <something>_types, occasionally _codes, categories or _classes.

Each of these tables is described in more detail on its own wiki page.