Adds functionality to ob-comint.el to implement async session eval on
a per-language basis. Adds a reference implementation for ob-python.
* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.
* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.
*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
* lisp/ob-python.el (org-babel-python--exec-tmpfile): Rename tmpfile handle
(org-babel-python-format-session-value): Rename tmpfile handle
Opening the exec tmpfile as a `f' variable shadows any such variable
that might by defined by the Python session context. e.g. my Org babel
files commonly pass single letter variables inside a session which is
broken by this behavior.
The new name `__org_babel_python_tmpfile' is in line with other org
mode specific Python variables set by ob-python. This is unlikely to
conflict with the user's Python code.
TINYCHANGE
* lisp/ob-python.el (org-babel-python--exec-tmpfile): Rename tmpfile handle
(org-babel-python-format-session-value): Rename tmpfile handle
Opening the exec tmpfile as a `f' variable shadows any such variable
that might by defined by the Python session context. e.g. my Org babel
files commonly pass single letter variables inside a session which is
broken by this behavior.
The new name `__org_babel_python_tmpfile' is in line with other org
mode specific Python variables set by ob-python. This is unlikely to
conflict with the user's Python code.
TINYCHANGE
* lisp/ob-python.el (org-babel-python-eoe-indicator): Add back
variable for end-of-expression indicator.
(org-babel-python--shift-right): New function to indent source body.
(org-babel-python-evaluate-external-process): Call new function org-babel-python--shift-right.
(org-babel-python--send-string): Wrap body in try-except-finally,
print indicator token, and find end of output via the token.
cf https://orgmode.org/list/871rjcan53.fsf@kyleam.com/
* lisp/ob-python.el (org-babel-python--eval-ast): Removed.
(org-babel-python-format-session-value): New function that returns
Python code to evaluate for sessions with value results.
(org-babel-python-evaluate-session): Replace
org-babel-python--eval-ast with org-babel-python-format-session-value.
Motivation is that the new function is more flexible than the old
format string. We can pass in the full result-params, which can be
used to add more formatting options in future. We could also add more
optional arguments in the future to extend this functionality.
This function will be particularly helpful for a couple patches after
9.4:
- ob-reticulate
- https://orgmode.org/list/875z98gj4f.fsf@gmail.com/
- Improved formatting
- https://orgmode.org/list/87eenpfe77.fsf@gmail.com/
However, I add this function in now 9.4, so those future patches can
rely on it in 9.4. In particular, ob-reticulate will probably be
packaged separately from org-mode, and I want to provide a stable
interface to it so that it can be released after 9.4.
* lisp/ob-python.el (py-send-string-no-output): Remove external function.
(py-shell-send-string): Add external function.
(org-babel-python--eval-ast): Remove blank lines.
(org-babel-python--send-string): New function to send string to Python
process and return output.
(org-babel-python-evaluate-session): Call
org-babel-python--send-string to evaluate code.
New function to send code to Python session. It has the following
desirable properties:
- Wait for evaluation to finish before returning.
- Input can have arbitrary spaces/newlines.
- Avoid echoing input to the session.
- Echo all output to the session.
- Output is also captured and returned separately.
- Avoid adding extra prompts to the session.
- Work well with both Python and IPython repls.
The implementation borrows from `python-shell-send-string-no-output'
and `org-babel-comint-with-output'. This commit is related to
632ceabb1, which tried to implement the same thing, but also
introduced bugs, and had to be partially reverted in b5709ddc9.
* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Remove
workaround for emacs<24.1. Sleep after starting Python, to prevent
startup messages leaking into output results.
* testing/lisp/test-ob-python.el (test-ob-python/session-multiline):
Remove workaround for startup message leaking into results.
* lisp/ob-python.el (py-shell-send-string): Remove unneeded
declare-function.
(org-babel-python-eoe-indicator): Add back this const.
(org-babel-python--eval-ast): Fix spacing so the block can be entered
line by line into the interpreter.
(org-babel-python-evaluate-session): Revert previous changes to
evaluation for value results.
* testing/lisp/test-ob-python.el (test-ob-python/session-value-sleep):
New test for session blocks that don't return immediately.
632ceabb1 introduced a bug where session blocks don't wait for value
results if not returned immediately. This reverts part of that
commit, specifically for evaluation of session blocks with value
results. It leaves other parts of that commit alone, for example the
changes to session output results, which is more robust handling
prompts from ipython interpreter.
A downside of reverting the session value implementation is that the
ugly code from org-babel-python--eval-ast is again printed in the
console, but this is less bad than the new bug that was introduced by
the change, hence reverted.
* lisp/ob-python.el: Require python.el at top-level.
(org-babel-python-eoe-indicator): Remove unused variable.
(org-babel-python-initiate-session-by-key): Rename
python-buffer to avoid obsolete warning.
(org-babel-python-evaluate-external-process): Remove unnecessary
require.
(org-babel-python--exec-tmpfile): Simplify this template.
(org-babel-python--eval-ast): Add printing of results to this
template, requiring additional string escapes.
(org-babel-python-evaluate-session): Simplify to use adjusted
templates, and call out to functionality in python.el or
python-mode.el.
(org-babel-python-evaluate-session): Simplified to use adjusted templates.
This commit refactors and cleans up code related to session
evaluation. python.el is now required at the top-level. Python
templates for wrapping code are simplified. Instead of directly
pasting code to the REPL, functionality from python.el and
python-mode.el are used; this fixes issues with code being echoed to
the REPL, and should be generally more robust. Finally, in the
:results value case, special handling of exceptions is removed, and we
no longer print None when the last statement isn't an expression.
* lisp/ob-python.el (org-babel-python-evaluate-session): Fix
discrepancy between how single-line and multiline code blocks return
output.
For example, before this commit, the 2 blocks behaved differently:
+begin_src python :session :results output
pass
"foo"
+end_src
+RESULTS:
+begin_src python :session :results output
"foo"
+end_src
+RESULTS:
: 'foo'
But now, they both behave as the former.
* lisp/ob-python.el (org-babel-python-evaluate-session): Process
temporary file name with `org-babel-process-file-name' before
inserting it into code Python code snippets.
Before this change, the entire temporary filename was sent to the
Python session for execution, causing a 'No such file' error when the
filename had a Tramp format such as
/ssh:user@server:/tmp/python-ABCDEF.
TINYCHANGE
* lisp/ob-python.el (org-babel-python-evaluate-external-process): Use
functions from python.el to indent lines, avoiding multiline strings.
* testing/lisp/test-ob-python.el (test-ob-python/multiline-var): Set
test as expected to succeed.
(test-ob-python/multiline-str): Add test for multiline string in body.
(test-ob-python/header-var-assignment): Test that :var is in correct
scope and can be assigned to.
cf. https://orgmode.org/list/87tv009l9a.fsf@gmail.com/#t
* lisp/ob-python.el (org-babel-execute:python,
org-babel-python-evaluate): Move variable assignment back to body from
preamble.
* testing/lisp/test-ob-python.el (test-ob-python/multiline-var): Mark
unit test for multiline-variable as expected to fail.
Reverts 6149b6cb6 which fixed multiline vars in ob-python, but caused
a new bug (https://lists.gnu.org/archive/html/emacs-orgmode/2020-05/msg00719.html).
* lisp/ob-python.el (org-babel-execute:python): Add variable
assignment to preamble instead of body.
(org-babel-python-evaluate): Concatenate preamble (which now includes
variable assignment) to body before session evaluation.
Fixes
https://lists.gnu.org/archive/html/emacs-orgmode/2020-03/msg00126.html. For
non-session evaluation, ob-python adds indentation to the body inside
main(), but this was adding spurious indentation for multiline strings
passed through :var. This commit fixes the issue by moving variable
assignment out of the body and into the preamble.
* lisp/ob-python.el (org-babel-python-evaluate-session): Fix a few
related issues with :session :results value blocks, including broken
if-else statements, indented blocks with blank lines, and returning
the wrong value when underscore has been used.
(org-babel-python--eval-ast): New constant variable, a string
consisting of Python code to execute a source block using ast.
Previously, python blocks with parameters ":session :results value"
were entered line-by-line into the Python session, which could cause
issues around indentation and new lines. Now, such python blocks are
written to temp files, then the built-in ast python module is used to
parse and execute them, and to extract the last line separately to
return as a result. Introduces a change in behavior, requiring that
the last line must be a top-level expression statement if its result
is to be saved (otherwise, the result is None).
* lisp/ob-python.el (org-bable-python-evaluate-session):
Syntax error occurs when evaluating the following code block:
\#+begin_src python :session
if True:
1
2
\#+end_src
A blank line is required for top level module code to end an indented block, such as a for loop, try/except, or if statement.
https://www.python.org/dev/peps/pep-0008/#blank-line
TINYCHANGE
* lisp/ob-python.el (orb-babel-python-evaluate-session): When :session
:results output, send multiline code blocks to tmpfile and execute in
Python with exec().
(org-babel-python--exec-tmpfile): New variable.
* testing/lisp/test-ob-python.el (test-ob-python/session-multiline):
Test for :session with multiple lines and indentation.
* lisp/ob-python.el (org-babel-execute:python):
`org-babel-python-command` should be set before calling
`org-babel-python-initiate-session`.
TINYCHANGE
Bastien
Jack Kamm
Kyle Meyer
Adrian Kummerlaender
Christian Vanderwall
stardiviner
Juanma Barranquero
Aaron Ecay
Nicolas Goaziou
Gong Qijian
Tim Landscheidt
Paul Eggert
Nathaniel Nicandro