Bulkloader

New specimen records may be created from a single flat (non-relational) file, a text file in which all (or most) data for a single cataloged item are in a single row. This file can be created with any convenient client-side application. The file is then loaded into a similarly structured table on the server, and a server-side application (the bulkloader) parses the columns from each row into the relational structure of the database. The process provides an independent layer of data checking before new information is incorporated into the database proper. Original data that are received in electronic format may require minimal manipulation; you can sometimes merely add the necessary columns to build a file in the bulk-loading format.

Bulkloader templates should be created from the Bulkloader Builder in Arctos. All other means, including this documentation, may produce non-current data which will be rejected.

There is no standard method for moving data into table Bulkloader. You may import data from any file format, type the data into the table, write your own data entry screen, or use any other method you choose. We appreciate documentation, even for specialized datasets – contact us if you wish to contribute.

You may mix accessions, collections, or anything else in a single load.

The specimen Bulkloader will not alone handle every eventuality that may ever occur while entering data. (The suite of tools available should.) Use flags to mark incomplete records for further editing, tie to other bulkloaders with UUIDs, or talk to your friendly local Arctos development team BEFORE you make a mess.

Error messages should include more than enough information to allow you to locate and correct the problem. If that isn’t the case, contact us with the error message and a description of the action that caused the error message.

Arctos is case-sensitive. JOHN DOE is not the same value as John Doe. Leading and trailing spaces and other non-printing characters matter.

The web-based applications may not work well for very large loads. Contact us if you’re having problems.

Agent Names

Agent Names must match a unique namestring, not necessarily the preferred name. If you are loading “John Smith” and there are three John Smiths in Arctos, you might create a new name “John Smith (my project)” and use that namestring in your data. Once loaded, the records will display preferred name, and agent name “John Smith (my project)” may be removed.

Taxonomy

Special note primarily for botanists: The bulkloader requires taxonomy.scientific_name, not taxonomy.display_name. That is, “Carex bigelowii subsp. lugens” rather than “Carex bigelowii Torr. subsp. lugens (Holm) T.V. Egorova”.

Any of the following are acceptable taxon name values (current 23 Aug 2011, see code table for most current formulas):

• Formula “A”: An exact match to any accepted taxonomy.scientific_name
  • Sorex cinereus
  • Soricidae
• Formula “A cf.”: Any accepted taxonomy.scientific_name plus “ cf.”
  • Sorex cf.
• Formula “A ?”: Any accepted taxonomy.scientific_name plus “ ?”
  • Sorex ?
• Formula “A x B”: Any two accepted taxonomy.scientific_names separated by “ x “
  • Sorex cinereus x Sorex yukonicus
• Formula “A or B”: Any two accepted taxonomy.scientific_names separated by “ or “
  • Sorex cinereus or Sorex yukonicus
• Formula “A and B”: Any two accepted taxonomy.scientific_names separated by “ and “
  • Sorex cinereus and Sorex yukonicus
• Formula “A {string}”: Any valid taxonomy.scientific_name, followed by a space, followed by an opening curly bracket, followed by a verbatim identification, followed by a closing curly bracket.
  • Sorex {Sorex new species “my name”}
  • unidentifiable {granite}

Be sure anything coming from other applications (especially Microsoft products) has not changed field length, precision, or other attributes. Watch dates and non-integer numbers (such as decimal latitude) most closely.

Fields

The following table describes select individual fields in the Bulkloader. Check the Bulkloader Builder for the latest table structure. Do not attempt to use this as a template. Let us know if it’s out of date, incomplete, cryptic, or otherwise useless.

Primary Key Warning

Some values may be replaced by or require primary keys: locality_id, entered_by_agent_id, collecting_event_id, etc. These are internal database identifiers that exist only for convenience, and may be updated, transferred to another data object, or removed for seemingly arbitrary reasons and without warning. They’ll probably work over short time-periods, but we offer no guarantees.

Processing

Once a record is marked to load by making loaded NULL, a script periodically attempts to parse the record into the normalized core Arctos structure. This may result in two things: * the record is created and marked for cache refresh, or * an error is returned in the loaded column

Records which successfully load must be refreshed in the cache before appearing in the user interfaces. Records are refreshed in the order they enter the queue. This process often takes less than one minute, but in the case of many thousands of records being queued can take up to several days. Reports/Services—>View Statistics—>FLAT status provides a summary of the state of the cache, and may be useful in estimating processing time.

Note that there is a period of time between successful loading and the cache being refreshed where records are not visible in any user interface.