methodshub.qmd

---
title: oolong - Create Validation Tests for Automated Content Analysis
format:
  html:
    embed-resources: true
  gfm: default
---

# oolong - Create Validation Tests for Automated Content Analysis
<!--
General specifications:
- This specification of the Methods Hub friendly README often uses the word 'should' to indicate the usual case. If you feel you need to do it differently, add a comment to argue for your case when you submit your method.
- A Methods Hub friendly README should contain all sections below that are not marked as optional, and can contain more sections.
- A Methods Hub friendly README should contain as few technical terms as possible and explain (or link to an explanation of) all used technical terms.
- A Methods Hub friendly README should link to all code files that it mentions using the [text](URL relative to this file) format. The relative URL (i.e., no "https://github.com") is neccessary for proper versioning in Methods Hub.
- A Methods Hub friendly README should contain an explanation (in the text) and an alternative for each image it contains (e.g., data models, pipeline, schema structure). Format: ![alternative text that describes what is visible in the image](URL relative to this file).
- A Methods Hub friendly README should link to authoritative sources rather than containing a copy of the information (e.g., documentation).
- A Methods Hub friendly README should use a uniform citation style for all references, for example APA7 https://apastyle.apa.org/style-grammar-guidelines/references/examples

Title:
1. The title must be the README's only first-level heading (line starting with a single '#').
2. The title should make the method's purpose clear.
3. The title (line 1 of this file) must be changed by you, but all other headings should be kept as they are.
4. The title must be appropriate (not harmful, derogatory, etc.).

Section templates:
The README template comes with text templates for each section (after each comment) that can be used, customized or removed as desired.
-->

## Description
<!--
1. Provide a brief and exact description of the method clearly mentioning its purpose i.e., what the method does or aims to achieve in abstract terms (avoiding technical details).
2. The focus should be on explaining the method in a way that helps users with different levels of expertise understand what it does, without going into technical details. It should clearly describe what inputs are needed and what outputs can be expected.
3. Briefly explain the input and output of the method and its note worthy features.
4. Provide link(s) to related papers from the social science domain using the method or similar methods for solving social science research questions. 
5. In a separate paragraph, highlight the reproducibility aspect of the method providing details or references to the resources used by the method, the data used in building the pre-trained modules etc.
6. It should also discuss the decisions and parameters controlling the behavior of the method.
-->

Intended to create standard human-in-the-loop validity tests for typical automated content analysis such as topic modeling and dictionary-based methods. This package offers a standard workflow with functions to prepare, administer and evaluate a human-in-the-loop validity test. This package provides functions for validating topic models using word intrusion, topic intrusion (Chang et al. 2009,  <https://papers.nips.cc/paper/3700-reading-tea-leaves-how-humans-interpret-topic-models>) and word set intrusion (Ying et al. 2021) [doi:10.1017/pan.2021.33](https://doi.org/10.1017/pan.2021.33) tests. This package also provides functions for generating gold-standard data which are useful for validating dictionary-based methods. The default settings of all generated tests match those suggested in Chang et al. (2009) and Song et al. (2020) [doi:10.1080/10584609.2020.1723752](https://doi.org/10.1080/10584609.2020.1723752).

## Keywords

* Validity
* Text Analysis
* Topic Model

## Use Cases
<!--
1. The use cases section should contain a list of use cases relevant to the social sciences.
2. Each use case should start with a description of a task and then detail how one can use the method to assist in the task.
3. Each use case may list publications in which the use case occurs (e.g., in APA7 style, https://apastyle.apa.org/style-grammar-guidelines/references/examples).
-->

This package was used in the literature to valid topic models and prediction models trained on text data, e.g. [Rauchfleisch et al. (2023)](https://doi.org/10.1080/17512786.2022.2110928), [Rothut, et al. (2023)](https://doi.org/10.1177/14614448231164409), [Eisele, et al. (2023)](https://doi.org/10.1080/19312458.2023.2230560).

## Input Data
<!--
1. The input data section should illustrate the input data format by showing a (possibly abbreviated) example item and explaining (or linking to an explanation of) the data fields.
2. The input data section should specify which parts of the input data are optional and what effect it has to not provide these.
3. The input data section should link to a small example input file in the same repository that can be used to test the method (this test should be described in the section "How to Use").
-->

A sample input is a model trained on text data, e.g.

```{r}
#| message: false
library(oolong)
library(seededlda)
abstracts_seededlda
```

A sample input is a model trained on text data, e.g.

```{r}
#| message: false
library(oolong)
library(seededlda)
abstracts_seededlda
```

## Output Data
<!--
1. The output data section should illustrate the output data format by showing a (possibly abbreviated) example item and explaining (or linking to an explanation of) the data fields.
2. The output data section should link to a small example output file in the same repository that can be re-created (as far as the method is non-random) from the input data (as described in the section "How to Use").
-->

The sample output is an oolong [R6 object](https://r6.r-lib.org/).

## Hardware Requirements
<!--
1. The hardware requirements section should list all requirements (storage, memory, compute, GPUs, cluster software, ...) that exceed the capabilities of a cheap virtual machine provided by cloud computing company (2 x86 CPU core, 4 GB RAM, 40GB HDD).
2. If the method requires a GPU, the hardware requirements section must list the minimal GPU requirements (especially VRAM).
-->

This package runs on any hardware that can run R.

## Environment Setup
<!--
1. The environment setup section should list all requirements and provide all further steps to prepare an environment for running the method (installing requirements, downloading files, creating directoriees, etc.).
2. The environment setup section should recommend to use a virtual environment or similar if the programming language supports one.
-->

With R installed:

```r
install.packages("oolong")
```

## How to Use
<!--
1. The how to use section should provide the list of steps that are necessary to produce the example output file (see section Output Data) after having set up the environment (see section Environment Setup).
2. The how to use section should explain how to customize the steps to one's own needs, usually through configuration files or command line parameters, or refer to the appropriate open documentation.
-->

Please refer to the [overview of this package](https://gesistsa.github.io/oolong/articles/overview.html) for a comprehensive introduction of all test types.

Suppose there is a topic model trained on some text data called `abstracts_seededlda`, which is included in the package.

```{r}
library(oolong)
abstracts_seededlda
```

Suppose one would like to conduct a word intrusion test (Chang et al. 2009) to validate this topic model. This test can be generated by the `wi()` function.

```{r}
oolong_test <- wi(abstracts_seededlda, userid = "Hadley")
oolong_test
```

One can then conduct the test following the instruction displayed, i.e. `oolong_test$$do_word_intrusion_test()`.

```r
oolong_test$do_word_intrusion_test()
```

One should see a graphic interface like the following and conduct the test.

<img src="man/figures/oolong_demo.gif" align="center" height="400" />

After the test, one can finalize the test by locking the test.

```{r}
#| include: false
### Mock this process
oolong_test$.__enclos_env__$private$test_content$wi$answer <- oolong_test$.__enclos_env__$private$test_content$wi$intruder
oolong_test$.__enclos_env__$private$test_content$wi$answer[1] <- "wronganswer"
```

```{r}
oolong_test$lock()
```

And then obtain the result of the test. For example:

```{r}
oolong_test
```

## Technical Details
<!--
1. The technical details section should proview a process overview, linking to key source code files at every step of the process.
2. In case a publication provides the details mentioned below, the technical details section should link to this publication using a sentence like "See the [publication](url-of-publication-best-using-doi) for ...". In this case, the mentioned technical details can be omitted from the section.
3. The technical details section should list all information needed to reproduce the method, including employed other methods and selected parameters.
4. The input data section should link to external data it uses, preferably using a DOI to a dataset page or to API documentation.
5. The technical details section should mention how other methods and their parameters were selected and which alternatives were tried.
6. The technical details section should for employed machine learning models mention on what kind of data they were trained.
-->

See the official [CRAN repository](https://cran.r-project.org/web/packages/oolong/) for further information about technical details.

## References
<!--
1. The references section is optional, especially if they are cited in a publication that explains the technical details (see section Technical Details).
2. The references section should provide references of publications related to this method (e.g., in APA7 style, https://apastyle.apa.org/style-grammar-guidelines/references/examples).
-->

Chan, C. H., & Sältzer, M. (2020). oolong: An R package for validating automated content analysis tools. The Journal of Open Source Software: JOSS, 5(55), 2461. <https://doi.org/10.21105/joss.02461>

## Contact Details
<!-- 
1. The contact details section should specify whom to contact for questions or contributions and how (can be separate entitites; for example email addresses or links to the GitHub issue board).
-->

Maintainer: Chung-hong Chan <chainsawtiney@gmail.com>
Issue Tracker: [https://github.com/gesistsa/oolong/issues](https://github.com/gesistsa/oolong/issues)