Darwin Core text guide

Title

Darwin Core text guide

Date version issued

2023-09-13

Date created

2009-02-12

Part of TDWG Standard

http://www.tdwg.org/standards/450

This version

http://rs.tdwg.org/dwc/terms/guides/text/2023-09-13

Latest version

http://rs.tdwg.org/dwc/terms/guides/text/

Previous version

http://rs.tdwg.org/dwc/terms/guides/text/2021-07-15

Abstract

Guidelines for implementing Darwin Core in Text files.

Contributors

Tim Robertson (Global Biodiversity Information Facility), Markus Döring (Global Biodiversity Information Facility), John Wieczorek (VertNet), Renato De Giovanni (Centro de Referência em Informação Ambiental), Dave Vieglais (KU Natural History Museum), Steve Baskauf (Vanderbilt University Libraries), Gail Kampmeier (Illinois Natural History Survey)

Creator

Darwin Core Maintenance Group

Bibliographic citation

Darwin Core Maintenance Group. 2023. Darwin Core text guide. Biodiversity Information Standards (TDWG). http://rs.tdwg.org/dwc/terms/guides/text/2023-09-13

1 Introduction

This document provides guidelines for formatting and sharing Darwin Core terms in fielded text formats, such as one or more comma separated value (CSV) files. Data conforming to the Simple Darwin Core (CSV format and having the first row include Darwin Core standard term names) MAY be shared in a single file, while a non-standard text file MAY be understood using an XML metafile to describe its contents and formatting. A Darwin Core Archive is an example of an implementation of the Darwin Core Text recommendation.

More complex structure MAY be shared in multiple related files. The description of content and relationships between files can be achieved using the metafile (meta.xml). This guideline makes recommendations for the simple case of a core file, upon which Darwin Core records are based, and extensions, which are linked to records in that core file. Specifically, extension records have a many-to-one relationship with records in the core file. For example, a core file might contain specimen records, with one specimen per row in the file, while an extension file contains one or more identifications for those specimens, with one identification per row in the extension file, and with an identifier to the specimen for each identification row. This example would allow many identifications to be associated with each specimen.

1.1 Status of the content of this document

All sections of this document are normative, except for examples, whose sections are marked as non-normative.

1.1.1 RFC 2119 key words

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they are written in capitals (as shown here).

1.2 Simple example metafile content (non-normative)

A simple comma separated values (CSV) data file named specimens.csv with the following content:

ID,Species,Count
123,"Cryptantha gypsophila Reveal & C.R. Broome",12
124,"Buxbaumia piperi",2

can be described with the following Darwin Core metafile (meta.xml):

<?xml version="1.0" encoding="UTF-8"?>
<archive xmlns="http://rs.tdwg.org/dwc/text/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xsi:schemaLocation="http://rs.tdwg.org/dwc/text/ http://rs.tdwg.org/dwc/text/tdwg_dwc_text.xsd">
    <core rowType="http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord" ignoreHeaderLines="1">
	<files>
	    <location>http://data.gbif.org/download/specimens.csv</location>
    	</files>
    	<field index="0" term="http://rs.tdwg.org/dwc/terms/occurrenceID" />
    	<field index="1" term="http://rs.tdwg.org/dwc/terms/scientificName" />
    	<field index="2" term="http://rs.tdwg.org/dwc/terms/individualCount" />
   	<!-- A constant value has no index, but applies to all rows -->
    	<field term="http://rs.tdwg.org/dwc/terms/datasetID" default="urn:lsid:tim.lsid.tdwg.org:collections:1"/>
    </core>
</archive>

These same data could be understood without the metafile if the first row of the CSV file contained Darwin Core term names, such as:

type,institutionCode,collectionCode,catalogNumber,scientificName,individualCount,datasetID
PhysicalObject,ANSP,PH,123,"Cryptantha gypsophila Reveal & C.R. Broome",12,urn:lsid:tim.lsid.tdwg.org:collections:1
PhysicalObject,ANSP,PH,124,"Buxbaumia piperi",2,urn:lsid:tim.lsid.tdwg.org:collections:1

2 Metafile content

The text metafile schema provides technical details for the structure of a metafile by defining the elements and attributes necessary to describe the contents and relationships between text files. These elements and attributes, with descriptions and specifications for their use in a metafile, are described in the following table. The metafile MUST be named meta.xml.

2.1 The <archive> element

The <archive> element is the container for the list of related files (one core and zero or more extensions). The <archive> element SHOULD have a metadata attribute.

2.1.1 Attributes

2.1.2 Elements

2.2 The <core> or <extension> element

2.2.1 Attributes

2.2.2 Elements

2.3 <files> element

The <files> element MUST contain one or more <location> elements, each defining where a file resides. Each <core> or <extension> entity MAY be composed from one or more files. If an entity has data in more than one file, the <location> element MUST be present once for each distinct <file> that makes up the entity.

2.3.1 Elements

2.4 The <field> element

The <field> element is used to specify the location and content of data within a <file>. There MUST be one <field> element for every term being shared for the entity, whether explicitly or through the use of a <default> value for all rows in the <file>.

2.4.1 Attributes

3 Implementation guide (non-normative)

A Darwin Core Archive is an example of an implementation of the Darwin Core Text recommendation.

3.1 Extension example (non-normative)

The following example illustrates the use of extensions. In this example there are three files in the archive, all of which are located in the same directory as the metafile. The whales.txt file acts as a core file of Taxon records. The whales.txt file is extended by two other files, types.txt and distribution.txt. The types.txt file contains records specified in an external definition at http://rs.gbif.org/terms/1.0/TypesAndSpecimen and consists of Dublin Core and Darwin Core terms, while the distribution.txt file contains records specified in the Species Distribution Extension at http://rs.gbif.org/terms/1.0/Distribution and consists of Darwin Core terms plus an additional term for threatStatus. Both extension files are related to the core file by the dwc:taxonID fields. This archive contains information about whale species, type specimen records for those species, and lists of countries and the threat status for those species in those countries.

<?xml version="1.0" encoding="UTF-8"?>
<archive xmlns="http://rs.tdwg.org/dwc/text/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xsi:schemaLocation="http://rs.tdwg.org/dwc/text/ http://rs.tdwg.org/dwc/text/tdwg_dwc_text.xsd">

    <core encoding="UTF-8" fieldsTerminatedBy="\t" linesTerminatedBy="\n" ignoreHeaderLines="1" rowType="http://rs.tdwg.org/dwc/terms/Taxon">
        <files>
            <location>whales.txt</location>
        </files>
        <id index="0" />
        <field index="0" term="http://rs.tdwg.org/dwc/terms/taxonID" />
        <field index="1" term="http://purl.org/dc/terms/modified" />
        <field index="2" term="http://rs.tdwg.org/dwc/terms/scientificName"/>
        <field index="3" term="http://rs.tdwg.org/dwc/terms/acceptedNameUsageID"/>
        <field index="4" term="http://rs.tdwg.org/dwc/terms/parentNameUsageID"/>
        <field index="5" term="http://rs.tdwg.org/dwc/terms/originalNameUsageID"/>
    </core>

    <extension encoding="UTF-8" fieldsTerminatedBy="," linesTerminatedBy="\n" fieldsEnclosedBy='"' ignoreHeaderLines="1" rowType="http://rs.gbif.org/terms/1.0/TypesAndSpecimen">
        <files>
            <location>types.csv</location>
        </files>
        <coreid index="0" />
        <field index="1" term="http://purl.org/dc/terms/bibliographicCitation"/>
        <field index="2" term="http://rs.tdwg.org/dwc/terms/catalogNumber"/>
        <field index="3" term="http://rs.tdwg.org/dwc/terms/collectionCode"/>
        <field index="4" term="http://rs.tdwg.org/dwc/terms/institutionCode"/>
        <field index="5" term="http://rs.tdwg.org/dwc/terms/typeStatus"/>
    </extension>

    <extension encoding="UTF-8" fieldsTerminatedBy="," linesTerminatedBy="\n" fieldsEnclosedBy='"' ignoreHeaderLines="1" rowType="http://rs.gbif.org/terms/1.0/Distribution">
        <files>
            <location>distribution.csv</location>
        </files>
        <coreid index="0" />
        <field index="1" term="http://rs.tdwg.org/dwc/terms/countryCode"/>
        <field index="2" term="http://rs.gbif.org/terms/1.0/threatStatus"/>
        <field index="3" term="http://rs.tdwg.org/dwc/terms/occurrenceStatus"/>
    </extension>
</archive>

4 Database example (non-normative)

4.1 MySQL

It is very easy to produce fielded text using the SELECT INTO … outfile command from MySQL. The encoding of the resulting file will depend on the server variables and collations used, and might need to be modified before the operation is done. Note that MySQL will export NULL values as \N by default. Use the IFNULL() function as shown in the following example to avoid this.

SELECT
    IFNULL(id, ''), IFNULL(scientific_name, ''), IFNULL(count,'')
        INTO outfile '/tmp/dwc.txt'
            FIELDS TERMINATED BY ','
            OPTIONALLY ENCLOSED BY '"'
            LINES TERMINATED BY '\n'
FROM
    dwc;