org-mode/rorg.org

rorg — R and org-mode

• Overview
  • Dan [2009-02-08 Sun]
    • What happens when the code is evaluated?
    • When is the code evaluated?
      • Latex
        • org → org* → latex
        • org → latex → \latex*
• Commentary
  • Eric <2009-02-06 Fri 15:41>
• Objectives
  • Send data to R from org
    • Implementations
      • naive
      • org-R
      • org-exp-blocks
      • RweaveOrg
  • Evaluate R code from org and deal with output appropriately
    • vector output
      • Implementations
        • org-R
        • org-exp-blocks
        • RweaveOrg
    • graphical output
      • Implementations
        • org-R
        • org-exp-blocks
        • RweaveOrg
  • Evaluate R code on export
• Notes
  • Special editing and evaluation of source code in R blocks
    • PROPOSED R-block proposal
      • src block evaluation w/org-eval-light
    • Source code blocks
    • dblocks
    • R blocks
  • Interaction with the R process
• Tasks
• buffer dictionary

Overview

Dan [2009-02-08 Sun]

This project is basically about putting source code into org files. This isn't just code to look pretty as a source code example, but code to be evaluated. Org files have 3 main export targets: org, html and latex. Thus the aim of this project is to produce files in those formats that have benefitted in some way from the evaluation of source code that is present in the source org file. We have a current focus on R code, but we are regarding that more as a working example than as a defining feature of the project.

The following are three important questions:

1. How is the code placed in the org file?
2. When is the code evaluated?
3. What happens when the code is evaluated?

Let's take them in reverse order, as 1 and 2 can be viewed as details of the implementation of 3.

What happens when the code is evaluated?

Code evaluation can have three relevant consequences

1. It produces text/numeric output
2. It produces graphical output
3. It alters the environment by side effect in some other way

I think our aims here are fairly straightforward to specify: in the case of (1) we (optionally) incorporate the text output as text in the target document. In the case of (2) we either place a link to the graphics or (html/latex) include them inline. We don't need to do anything about (3), except bear in mind that it's a possibility.

When is the code evaluated?

Let's take the three export targets in turn

Latex

Here we have some options. I'll use an asterisk to indicate content which includes the result of code evaluation, rather than the code itself.

org → org* → latex

This is what RweaveOrg does, e.g.

Sweave("testing.Rorg", driver=RweaveOrg, syntax=SweaveSyntaxOrg)

Here testing.Rorg is an org file, and Sweave generates another org file, but with the code evaluated.

This is also what org-R does.

org → latex → \latex*

This seems like a possibility to me, but AFAIK we don't have it working. So in the case of R, here you would use something like `org-export-as-latex', and then from R, run Sweave (with the RweaveLatex driver) on the exported latex to generate the latex* file with evaluation results. Since Sweave with RweaveLatex is an established technology that is heavily used in the R community, it would be nice to provide this route as an option, so at any point users can `bail out' and enter a workflow where they treat the latex/noweb as source code to be edited directly (rather than editing a org buffer).

Commentary

Eric <2009-02-06 Fri 15:41>

I think we're getting close to a comprehensive set of objectives (although since you two are the real R user's I leave that decision up to you). Once we've agreed on a set of objectives and agreed on at least to broad strokes of implementation, I think we should start listing out and assigning tasks.

Objectives

Send data to R from org

Org-mode includes orgtbl-mode, an extremely convenient way of using tabular data in a plain text file. Currently, spreadsheet functionality is available in org tables using the emacs package calc. It would be a boon both to org users and R users to allow org tables to be manipulated with the R programming language. Org tables give R users an easy way to enter and display data; R gives org users a powerful way to perform vector operations, statistical tests, and visualization on their tables.

Implementations

naive

Naive implementation would be to use (org-export-table "tmp.csv") and (ess-execute "read.csv('tmp.csv')").

org-R

org-R passes data to R from two sources: org tables, or csv files. Org tables are first exported to a temporary csv file using org-R-export-to-csv.

org-exp-blocks

org-exp-blocks uses /ndwarshuis/org-mode/src/commit/cf31c1f13423d24095f8808db07f84c8f1e2798e/org-interblock-R-command-to-string to send commands to an R process running in a comint buffer through ESS. org-exp-blocks has no support for dumping table data to R process, or vice versa.

RweaveOrg

NA

Evaluate R code from org and deal with output appropriately

vector output

When R code evaluation generates vectors and 2-dimensional arrays, this should be formatted appropriately in org buffers (orgtbl-mode) as well as in export targets (html, latex)

Agreed, if we can convert the vector data to lists then we can use the many orgtbl-to-* functions to convert the list to whatever output format we desire. See `orgtbl-to-orgtbl, `orgtbl-to-latex', `orgtbl-to-html', `orgtbl-to-csv', etc…

Implementations

org-R

org-R converts R output (vectors, or matrices / 2d-arrays) to an org table and stores it in the org buffer, or in a separate org file (csv output would also be perfectly possible).

org-exp-blocks

RweaveOrg

graphical output

R can generate graphical output on a screen graphics device (e.g. X11, quartz), and in various standard image file formats (png, jpg, ps, pdf, etc). When graphical output is generated by evaluation of R code in Org, at least the following two things are desirable:

1. output to screen for immediate viewing is possible
2. graphical output to file is linked to appropriately from the

org file This should have the automatic consequence that it is included appropriately in subsequent export targets (html, latex).

Implementations

org-R

org-R does (1) if no output file is specified and (2) otherwise

org-exp-blocks

org-exp-blocks tries to do 2, but I don't think that part was every really working

RweaveOrg

Evaluate R code on export

At first I was leaning towards leaving the exporting to Sweave, but it seems that once we have evaluation or R working, it will not be difficult to implement full evaluation of R blocks, one-liners, and creation of R graphics on export directly in elisp.

I think that this would be worth the effort since it would eliminate the need for using Sweave, and would allow for exportation to any target format supported by org-mode.

Notes

Special editing and evaluation of source code in R blocks

Unfortunately org-mode how two different block types, both useful. In developing RweaveOrg, a third was introduced.

Eric is leaning towards using the #+begin_src blocks, as that is really what these blocks contain: source code. Austin believes that specifying export options at the beginning of a block is useful functionality, to be preserved if possible.

Note that upper and lower case are not relevant in block headings.

PROPOSED R-block proposal

I (Eric) propose that we use the syntax of source code blocks as they currently exist in org-mode with the addition of evaluation, header-arguments, exportation, single-line-blocks, and references-to-table-data.

1. evaluation: These blocks can be evaluated through \C-c\C-c with a slight addition to the code already present and working in org-eval-light.el. All we should need to add for R support would be an appropriate entry in /ndwarshuis/org-mode/src/commit/cf31c1f13423d24095f8808db07f84c8f1e2798e/org-eval-light-interpreters with a corresponding evaluation function. For an example usinga org-eval-light see /ndwarshuis/org-mode/src/commit/cf31c1f13423d24095f8808db07f84c8f1e2798e/%2A%20src%20block%20evaluation%20w/org-eval-light.
2. header-arguments: These can be implemented along the lines of Austin's header arguments in org-sweave.el.
3. exportation: Should be as similar as possible to that done by Sweave, and hopefully can re-use some of the code currently present in org-exp-blocks.el.
4. single-line-blocks: It seems that it is useful to be able to place a single line of R code on a line by itself. Should we add syntax for this similar to Dan's #+R: lines? I would lean towards something here that can be re-used for any type of source code in the same manner as the #+begin_src R blocks, maybe #+src_R?
5. references-to-table-data: I get this impression that this is vital to the efficient use of R code in an org file, so we should come up with a way to reference table data from a single-line-block or from an R source-code block. It looks like Dan has already done this in org-R.el.

What do you think? Does this accomplish everything we want to be able to do with embedded R source code blocks?

src block evaluation w/org-eval-light

here's an example using org-eval-light.el

first load the org-eval-light.el file

elisp:(load (expand-file-name "org-eval-light.el&%2334;%20(expand-file-name%20&%2334;existing_tools&%2334;%20(file-name-directory%20buffer-file-name))))

then press \C-c\C-c inside of the following src code snippet. The results should appear in a comment immediately following the source code block. It shouldn't be too hard to add R support to this function through the `org-eval-light-interpreters' variable.

date

Source code blocks

Org has an extremely useful method of editing source code and examples in their native modes. In the case of R code, we want to be able to use the full functionality of ESS mode, including interactive evaluation of code.

Source code blocks look like the following and allow for the special editing of code inside of the block through `org-edit-special'.

,## hit C-c ' within this block to enter a temporary buffer in r-mode.

,## while in the temporary buffer, hit C-c C-c on this comment to
,## evaluate this block
a <- 3
a

,## hit C-c ' to exit the temporary buffer

dblocks

dblocks are useful because org-mode will automatically call `org-dblock-write:dblock-type' where dblock-type is the string following the #+BEGIN: portion of the line.

dblocks look like the following and allow for evaluation of the code inside of the block by calling \C-c\C-c on the header of the block.

R blocks

In developing RweaveOrg, Austin created org-sweave.el. This allows for the kind of blocks shown in testing.Rorg. These blocks have the advantage of accepting options to the Sweave preprocessor following the #+BEGIN_R declaration.

Interaction with the R process

We should take care to implement this in such a way that all of the different components which have to interactive with R including:

• evaluation of source code blocks
• automatic evaluation on export
• evaluation of \R{} snippets
• evaluation of single source code lines
• sending/receiving vector data

I think we currently have two implementation of interaction with R processes; org-R.el and org-exp-blocks.el. We should be sure to take the best of each of these approaches.

Tasks

buffer dictionary

LocalWords: DBlocks dblocks