Introduction to org

Why org?

Research projects and data analyses can be hard to manage. These problems are common:

  • Inconsistent project structures across different analyses.
  • Mixed requirements for code (version control), results (sharing), and data (security).
  • Collaboration difficulties when team members use different folder structures.
  • Version tracking for research submissions and revisions.
  • Cross-platform compatibility issues with file paths.

The org package solves these problems. It provides a standardized framework that organizes R projects. The framework gives you a clear separation of concerns, and a consistent structure across all your analyses.

Installation

# Install from CRAN
install.packages("org")

# Or install development version from GitHub
# devtools::install_github("csids/org")

Quick start

The code below starts your first org project.

library(org)

# 1. Initialize your project structure
org::initialize_project(
  env = .GlobalEnv,
  home = "my_analysis",
  results = "my_results"
)

# 2. Access project paths
org::project$home          # Your code location
org::project$results_today # Today's results folder

# 3. Use org functions in your analysis
org::path("data", "file.csv")  # Cross-platform paths
org::ls_files("R")             # List R files

Concept

The concept behind org is straightforward. Most analyses have three main sections:

  • Code: Analysis scripts and functions.
  • Results: Output files and figures.
  • Data: Input data files.

Each section has unique requirements:

Code requirements

  • Code MUST be version controlled.
  • Code SHOULD be publicly accessible.
  • Code needs a single analysis pipeline that documents all steps.
  • Code SHOULD be organized into modular functions.

Results requirements

  • Results MUST be immediately shareable with collaborators.
  • Results SHOULD maintain a history of changes over time.
  • Results SHOULD be organized by date for tracking.
  • Results SHOULD be stored in a shared location (e.g., Dropbox).

Data requirements

  • Data SHOULD be encrypted if sensitive.
  • Data SHOULD NOT be stored in the cloud if sensitive.
  • Data SHOULD be organized by project or analysis.
  • Data SHOULD maintain clear separation from code and results.

Project structure

Core components

1. org::initialize_project

initialize_project() is the main function that sets up your project structure. It takes two or more arguments. It saves folder locations in org::project for use throughout your analysis:

  • home: Location of Run.R and the R/ folder (accessible via org::project$home).
  • results: Results folder that creates date-based subfolders (accessible via org::project$results_today).
  • ...: Additional folders as needed (e.g., data_raw, data_clean).
  • max_loc_per_file: The largest number of code lines one .R file may hold. initialize_project() stops with an error naming every file above the limit, before it sources any of them. The default is Inf, which checks nothing.

A code line is a physical line that is neither blank nor entirely a comment. The R parser identifies the comments, so a # inside a string does not hide a line. org::loc_per_file() applies the same count on its own, if you want to see the numbers without setting a limit.

2. Run.R

Run.R is your main analysis script. It orchestrates the entire workflow:

  • Data cleaning.
  • Analysis.
  • Result generation.

All code sections SHOULD be in functions in the R/ folder. You SHOULD NOT have multiple main files. Multiple main files create confusion when you return to your code later. You MAY have versioned files (e.g., Run_v01.R, Run_v02.R). A later version supersedes an earlier one.

3. R/ directory

All analysis functions SHOULD be defined in org::project$home/R. initialize_project() automatically sources every R script in org::project$home/R.

Example project structure

The code below is a complete example of a project structure.

# Initialize the project
org::initialize_project(
  env = .GlobalEnv,
  home = "/git/analyses/2019/analysis3/",
  results = "/dropbox/analyses_results/2019/analysis3/",
  data_raw = "/data/analyses/2019/analysis3/"
)

# Document changes in archived results
txt <- glue::glue("
  2019-01-01:
    Included:
    - Table 1
    - Table 2
  
  2019-02-02:
    Changed Table 1 from mean -> median
", .trim=FALSE)

org::write_text(
  txt = txt,
  file = fs::path(org::project$results, "info.txt")
)

# Load required packages
library(data.table)
library(ggplot2)

# Run analysis
d <- clean_data()  # Accesses data from org::project$data_raw
table_1(d)         # Saves to org::project$results_today
figure_1(d)        # Saves to org::project$results_today
figure_2(d)        # Saves to org::project$results_today

Research article versioning

A research article often needs multiple versions, such as the initial submission and the resubmissions. org manages this with date-based versioning.

  1. Initial submission.
    • Rename Run.R to Run_YYYY_MM_DD_submission_1.R.
    • Rename R/ to R_YYYY_MM_DD_submission_1/.
  2. Resubmission.
    • Create new files with updated dates.
    • Keep old versions for reference.

This preserves the code that produced the results for each submission. Every change is then deliberate and intentional.

Team collaboration

Team members often have different folder structures. You can specify several possible paths for one folder. org automatically selects the first path that exists:

# Team member setup - org will use the first existing path
org::initialize_project(
  env = .GlobalEnv,
  home = c(
    "/Users/teammate1/projects/analysis3/",  # Mac user
    "/home/teammate2/analysis3/",            # Linux user  
    "C:/Users/teammate3/analysis3/"          # Windows user
  ),
  results = c(
    "/Users/teammate1/Dropbox/results/",
    "/home/teammate2/dropbox/results/", 
    "C:/Users/teammate3/Dropbox/results/"
  ),
  data_raw = c(
    "/Users/teammate1/data/analysis3/",
    "/home/teammate2/data/analysis3/",
    "C:/shared_drive/data/analysis3/"
  )
)

The same initialization code then works across different team members’ machines, with no changes.

Best practices

Alternative structures

RMarkdown project

Use this structure for a project on a shared network drive, without GitHub or Dropbox.

project_name/              # org::project$home
├── Run.R
├── R/
│   ├── CleanData.R
│   ├── Descriptives.R
│   ├── Analysis1.R
│   └── Graphs1.R
├── paper/
│   └── paper.Rmd
├── results/              # org::project$results
│   └── 2018-03-12/      # org::project$results_today
│       ├── table1.xlsx
│       └── figure1.png
└── data_raw/            # org::project$data_raw
    └── data.xlsx

Single folder project

Use this structure for a project with limited access.

project_name/              # org::project$home
├── Run.R
├── R/
│   ├── clean_data.R
│   ├── descriptives.R
│   ├── analysis.R
│   └── figure_1.R
├── results/              # org::project$results
│   └── 2018-03-12/      # org::project$results_today
│       ├── table_1.xlsx
│       └── figure_1.png
└── data_raw/            # org::project$data_raw
    └── data.xlsx

Path naming conventions

Path components have standard names.

Component Name
/home/richard/test.src Absolute (file)path
richard/test.src Relative (file)path
/home/richard/ Absolute (directory) path
./richard/ Relative (directory) path
richard Directory
test.src Filename

A path specifies a location in a directory structure, while a filename only includes the file name itself. Directories only include directory name information.

Function reference

The org package provides several key functions for project management.

Core functions

  • org::initialize_project(): Set up project structure and source R files.
  • org::set_results(): Change the results folder after project initialization.
  • org::project: Environment containing all project folder locations.

File operations

  • org::path(): Construct cross-platform file paths.
  • org::ls_files(): List files with optional pattern matching.
  • org::loc_per_file(): Count the code lines in each of several R files.
  • org::move_directory(): Move directories safely.
  • org::write_text(): Write text files with consistent formatting.

Common workflows

Setting up a new analysis

# 1. Initialize project structure
org::initialize_project(
  env = .GlobalEnv,
  home = "/path/to/your/analysis/",
  results = "/path/to/results/",
  data_raw = "/path/to/data/"
)

# 2. Create analysis functions in R/ folder
# 3. Run analysis from Run.R  
# 4. Results automatically saved to org::project$results_today

Working with existing projects

# Reinitialize existing project
org::initialize_project(
  env = .GlobalEnv,
  home = "/existing/analysis/path/",
  results = "/existing/results/path/"
)

# Update results location if needed
org::set_results("/new/results/path/")

Environment management

Recommendation: you SHOULD always use .GlobalEnv. It is much easier. All your functions are then directly accessible, and you do not need to think about environment scoping.

# Recommended approach - use .GlobalEnv
org::initialize_project(env = .GlobalEnv, ...)

# Only use custom environments in special cases (e.g., package development)
my_env <- new.env()
org::initialize_project(env = my_env, ...)

Path construction and cross-platform compatibility

The org::path() function ensures your code works across different operating systems.

# Cross-platform path construction
data_file <- org::path(org::project$data_raw, "survey_data.csv")
output_file <- org::path(org::project$results_today, "analysis_results.xlsx")

# Handles multiple path components
nested_path <- org::path("folder1", "subfolder", "file.txt")

# Removes double slashes automatically
clean_path <- org::path("folder//", "//file.txt")  # Returns "folder/file.txt"

Troubleshooting

Common issues

Path issues

  • Always use org::path() for cross-platform compatibility.
  • Avoid hardcoded absolute paths in shared code.
  • Check that all specified directories exist and are accessible.
  • Make sure you have write permissions to results directories.

Sourcing problems

# If functions aren't loading from R/ folder:
# 1. Check that R files are in the correct directory
org::ls_files(org::path(org::project$home, "R"))

# 2. Verify file extensions are .R or .r
# 3. Check for syntax errors in R files
# 4. Restart R and reinitialize project if needed

Getting help