Into to the Crossref APIverse

Author

Affiliation

Luis Montilla

Crossref

Published

January 12, 2024

So you want to retrieve data from Crossref using R. If you want to jump to the coding part, click here, but if you are new to APIs and/or Crossref role in the scholarly metadata ecosystem, let me invite to read more below.

Crossref and the research nexus

At Crossref, we make research objects easy to find, cite, link, assess, and reuse. Our goal is to connect all research knowledge through open and persistent metadata. We have termed this interconnected knowledge ecosystem, the Research Nexus.

The metadata we collect and make available are the identifiers and properties of different items in the scholarly record. We don’t collect the full text or the actual datasets that constitute a scholarly paper, instead, we are interested in relationships between the paper, the authors, institutions, journals, funding agencies, protocols, and more.

We make this metadata open through our REST API

Our data is readily available. You have three access levels to the API metadata:

Public. Free, fully anonymous.
Polite. Free, you provide your email. (Recommended).

We can use this information to contact you in case of issues. We get rid of it after 90 days.
Plus. Our paid premium service. You get:
- A service level agreement guaranteeing you extra service and support, giving you a consistent and predictable experience.
- Additional features such as snapshots and priority service/rate limits.

What’s an API by the way?

An API is a software intermediary that allows two applications to talk to each other. This means that you don’t directly get the data as would happen when you download data associated to a research paper. In this document, we use the rcrossref package to build the requests that the Crossref API handles to return the data from Crossref servers.

flowchart LR
  A(Client) --> B{API}
  B --> C(Server)
  C --> B
  B --> A

Ready to dive into some code?

Libraries

We’ll make use of the rcrossref Chamberlain et al. (2022) package to interect with the Crossref API from the comfort of our R environment. We’ll also load gt to quickly visualize better-looking data tables.

Code

library(rcrossref)
library(tidyverse)
library(gt)
library(magrittr)

I will be polite. How do I provide my information?

We should add an email to ensure proper communication with Crossref in case our API requests break something. We can set this as a system variable:

Code

Sys.setenv(crossref_email = "yourname@yourorg.org")

Or we can directly type it in the environment file by first executing:

Code

file.edit("~/.Renviron")

Then, adding the email address to be shared with Crossref as crossref_email = name@example.com

Finally, save the file and restart your R session.

Let’s start with some examples:

Funder-related output

As a funding organization, you may wish to know how many publications are openly available to the public, e.g. the German Research Foundation.

First, we can do a general query to get the specific ID tied to the name of a given institution and store it in a variable that we’ll call funders.

Code

funders <- cr_funders(query = "German Research Foundation")

Typically, the output of these call is a JSON object, but with the rcrossref package we get nested tables that we can pass to gt, to quickly obtain better-looking data tables.

Notice that the specific organization that we are interested in is in the first row. We can also store this id for future use.

Code

funders$data |> 
  select(-tokens) |> 
  gt() |> 
  opt_interactive(use_search = TRUE,
  use_filters = TRUE,
  use_resizers = TRUE,
  use_highlight = TRUE,
  use_compact_mode = TRUE,
  use_page_size_select = TRUE)

Code

fund_id <- '501100001659' # this is the id corresponding to the  Deutsche Forschungsgemeinschaft

With this information, we can store the funder id of our interest in a new variable, and use it to retrieve additional information, e.g. how many papers are openly available to the public?

If we use our newly store funder id to retrieve data specifically from this organization, we’ll see that it has 223006 works associated with it:

Code

cr_funders(fund_id)$data$data |> 
  select(-tokens) |> 
  gt() |> 
  opt_interactive(use_search = TRUE,
  use_filters = TRUE,
  use_resizers = TRUE,
  use_highlight = TRUE,
  use_compact_mode = TRUE,
  use_page_size_select = TRUE)

The following code block shows you how we can add additional parameters to expand our results, for example, to retrieve records that are missing their full-text versions:

Code

1cr_funders(fund_id,
2           works = TRUE,
3           filter = c(has_full_text=TRUE),
4           limit = 25
           )

1: We use the id to retrieve works specifically linked to our funder.
2: This parameter let us retrieve works instead of a summary.
3: This filter let us retrieve works that have the full text available.
4: We can limit the number of results (for the sake of efficiency).

Of course, we can add additional parameter to refine our result, for example, we can include additional filters to specify that we want journal articles. The following code chunk also select some relevant columns out of the entire list of options.

Code

cr_funders(fund_id,
           works = TRUE,
           filter = c(has_full_text=TRUE,
                      type = "journal-article") ,
           limit = 25
           ) |> 
  pluck("data") |> 
  select(title,
         published.online,
         container.title, 
         doi, 
         publisher,
         member, 
         reference.count,
         is.referenced.by.count
) |> 
  gt() |> 
  opt_interactive(use_search = TRUE,
  use_filters = TRUE,
  use_resizers = TRUE,
  use_highlight = TRUE,
  use_compact_mode = TRUE,
  use_page_size_select = TRUE)

We can use filter_names() to print the list of filtering options that the package has currently available:

Code

filter_names()

 [1] "award_funder"              "award_number"             
 [3] "content_domain"            "from_accepted_date"       
 [5] "from_online_pub_date"      "from_posted_date"         
 [7] "from_print_pub_date"       "has_assertion"            
 [9] "has_authenticated_orcid"   "has_crossmark_restriction"
[11] "has_relation"              "location"                 
[13] "relation_object"           "relation_object_type"     
[15] "relation_type"             "until_accepted_date"      
[17] "until_online_pub_date"     "until_posted_date"        
[19] "until_print_pub_date"      "has_funder"               
[21] "funder"                    "prefix"                   
[23] "member"                    "from_index_date"          
[25] "until_index_date"          "from_deposit_date"        
[27] "until_deposit_date"        "from_update_date"         
[29] "until_update_date"         "from_pub_date"            
[31] "until_pub_date"            "has_license"              
[33] "license_url"               "license_version"          
[35] "license_delay"             "has_full_text"            
[37] "full_text_version"         "full_text_type"           
[39] "full_text_application"     "has_references"           
[41] "reference_visibility"      "has_archive"              
[43] "archive"                   "has_orcid"                
[45] "orcid"                     "issn"                     
[47] "isbn"                      "type"                     
[49] "directory"                 "doi"                      
[51] "updates"                   "is_update"                
[53] "has_update_policy"         "container_title"          
[55] "category_name"             "type_name"                
[57] "from_created_date"         "until_created_date"       
[59] "has_affiliation"           "assertion_group"          
[61] "assertion"                 "article_number"           
[63] "alternative_id"            "has_clinical_trial_number"
[65] "has_abstract"              "has_content_domain"       
[67] "has_domain_restriction"

Let’s try other options. For example, we might want to track articles related to clinical trials:

Code

cr_funders(fund_id, 
           works = TRUE, 
           filter = c(has_clinical_trial_number = TRUE,
                      type = "journal-article") ,
           limit = 25 
           ) |> 
  pluck("data") |>
  select(title,
         published.online,
         container.title, 
         doi, 
         publisher,
         member, 
         reference.count,
         is.referenced.by.count,
         `clinical-trial-number`) |> 
    unnest(`clinical-trial-number`) |> 
  gt() |>
  opt_interactive(use_search = TRUE,
  use_filters = TRUE,
  use_resizers = TRUE,
  use_highlight = TRUE,
  use_compact_mode = TRUE,
  use_page_size_select = TRUE)

Other resources

References

Chamberlain, Scott, Hao Zhu, Najko Jahn, Carl Boettiger, and Karthik Ram. 2022. Rcrossref: Client for Various ’CrossRef’ ’APIs’. https://CRAN.R-project.org/package=rcrossref.

License

Into to the Crossref APIverse by Luis Montilla is licensed under CC BY 4.0