* lisp/ob-python.el (org-babel-python-associate-session): New function
setting `python-shell-buffer-name' in *Org Src* buffer according to
source block's :session parameter. This function will be triggered by
`org-src-associate-babel-session'.
* etc/ORG-NEWS (ob-python now sets ~python-shell-buffer-name~ in Org
edit buffers): Announce the change.
Co-authored-by: Liu Hui <liuhui1610@gmail.com>
Link: https://orgmode.org/list/CAOQTW-MdC=jiGf+3bEVtfww+izSZix7csBJ+mZ4eZ2BQHDR42w@mail.gmail.com
ob-python will now use the same settings as `run-python' when starting
interactive sessions, by default.
* lisp/ob-python.el (org-babel-python-command): Changed to have
additional `auto' option, which is the new default value.
(org-babel-python-command-session): New option to control default
session Python command.
(org-babel-python-command-nonsession): New option to control default
nonsession Python command.
(org-babel-python--command): New function to determine the command to
run Python.
(org-babel-python-initiate-session-by-key): Call `run-python' without
CMD arg. Instead, set `python-shell-interpreter' and
`python-shell-interpreter-args' from `org-babel-python--command' when
needed.
(org-babel-python-evaluate-external-process): Use
`org-babel-python--command' to start Python.
*
lisp/ob-python.el (org-babel-python--python-util-comint-end-of-output-p):
Backport python-util-comint-end-of-output-p for emacs28.
(org-babel-python-initiate-session-by-key): Use backported version of
python-util-comint-end-of-output-p.
* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Wait
for first prompt for pre-existing python processes. Document behavior
for pre-existing python processes.
(org-babel-python-initiate-session): Document behavior for
pre-existing python processes.
*
testing/lisp/test-ob-python.el (test-ob-python/session-with-existing-inferior-python):
New test for working with existing python processes.
(org-babel-default-header-args:python):
(org-babel-header-args:python): Rename.
`org-babel-header-args:python' is the correct name for extra
language-specific allowed header arguments.
This is an attempt to mitigate occasional leaky prompts in CI. My
current hypothesis is that this could happen if the comint-output ends
between `org-babel-python-eoe-indicator' and the new prompt. Then the
new prompt could be injected into the output of the next unit test. To
avoid this, check we are at an ending prompt before exiting
`org-babel-python-send-string'.
https://list.orgmode.org/873506j7ky.fsf@localhost/
* lisp/ob-python.el (org-babel-python-send-string): Add check for
`python-shell-comint-end-of-output-p' to stop while loop accepting
process output.
(org-babel-execute:python):
(org-babel-variable-assignments:python):
(org-babel-python-format-session-value): Document all the function
arguments in the docstrings.
(org-babel-python-with-earmuffs):
(org-babel-python-without-earmuffs): Add docstring.
See: https://list.orgmode.org/87ttsnh5bx.fsf@localhost/T/#t
* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Switch
from `org-babel-comint-wait-for-output' to `sleep-for' while waiting
for `org-babel-python--initialized', to prevent hanging on restarted
Python process.
Thanks to Ihor Radchenko and Liu Hui for valuable feedback on this
patch.
* lisp/ob-python.el (org-babel-execute:python): Parse graphics-file
from params, and pass it to `org-babel-python-evaluate'.
(org-babel-python--output-graphics-wrapper): New constant. Python
code to save graphical output.
(org-babel-python--def-format-value): New constant. Python function
to format and save value results to file. Includes handling for
graphics, dataframes, and arrays.
(org-babel-python-table-or-string): Prevent `org-babel-script-escape'
from mangling dict results.
(org-babel-python--setup-session): New function to
run setup code within Python session.
(org-babel-python-initiate-session-by-key): Replace lambda with
`org-babel-python--setup-session' in `python-shell-first-prompt-hook',
or run it directly if the session was started outside Org.
(org-babel-python-wrapper-method): Removed. Instead use part of the
string directly in `org-babel-python-evaluate-external-process'.
(org-babel-python-pp-wrapper-method): Removed. Pretty printing is now
handled by `org-babel-python--def-format-value'.
(org-babel-python--exec-tmpfile): Removed. Instead use the raw string
directly in `org-babel-python-evaluate-session'.
(org-babel-python-format-session-value): Updated to use
`org-babel-python--def-format-value' for formatting value result.
(org-babel-python-evaluate): New parameter graphics-file. Pass
graphics-file onto downstream helper functions.
(org-babel-python-evaluate-external-process): New parameter
graphics-file. Use `org-babel-python--output-graphics-wrapper' for
graphical output. For value result, use
`org-babel-python--def-format-value'. Don't call `org-trim' on
results, to prevent misalignment of pandas DataFrames.
(org-babel-python-evaluate-session): New parameter graphics-file. Use
`org-babel-python--output-graphics-wrapper' for graphical output.
Replace the removed constant `org-babel-python--exec-tmpfile' with the
string directly. Rename local variable tmp-results-file to
results-file, which may take the value of graphics-file when provided.
(org-babel-python-async-evaluate-session): New parameter
graphics-file. Use `org-babel-python--output-graphics-wrapper' for
graphical output. Rename local variable tmp-results-file to
results-file, which may take the value of graphics-file when provided.
* lisp/ob-python.el (python-shell-buffer-name): Remove unneeded
defvar.
(org-babel-python-initiate-session-by-key): Check if session already
existed before run-python. Only wait for initialization if it's a
newly started session. Also simplify the code a bit by combining
multiple setq and let statements into a single let statement. Also
add a comment about why adding to `python-shell-first-prompt-hook'
after `run-python' should be safe from race conditions.
* ob-R.el (ob-session-async-org-babel-R-evaluate-session): Use
`org-id-uuid' instead of `md5' as results placeholder.
* ob-python.el (org-babel-python-async-evaluate-session): Use
`org-id-uuid' instead of `md5' as results placeholder.
* lisp/ob-python.el (org-babel-python-mode): Moved to ob-compat.el,
and changed to a constant.
(org-babel-python-initiate-session-by-key): Remove code to support
python-mode.el.
(org-babel-python-send-string): Renamed from
`org-babel-python--send-string', turning it into a public function to
accommodate ob-python-mode-mode which advises this function. Also,
remove some code for python-mode.el.
(org-babel-python-evaluate-session): Update calls to renamed function
`org-babel-python-send-string'.
* lisp/ob-compat.el (org-babel-python-mode): Moved from ob-python.el,
and changed to a constant.
* lisp/ob-python.el (org-babel-python--initialized): New internal flag
used to indicate that python session has been initialized in buffer.
(org-babel-python-initiate-session-by-key): Set
`org-babel-python--initialized' in `python-shell-first-prompt-hook'
when using built-in python.el. Wait until the hook is fired before we
finish initiating the session.
This patch intends to fix CI test failures where the CPU allocation is
limited and python loading is extremely slow.
* lisp/ob-python.el (py-choose-shell):
(py-toggle-shells): Use the new function name.
(py-default-interpreter): Remove the variable that is no longer
provided by python-mode.el.
(org-babel-python-initiate-session-by-key): Use `py-choose-shell'.
Reported-by: Christian Köstlin <christian.koestlin@gmail.com>
* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Do not
rely on 10ms delay to initialize Python. Wait until python process is
initialized using `org-babel-comint-wait-for-output'.
Emacs now advises using "website" (instead of "homepage") and
consequently to use the "URL" comment header instead of "Homepage".
* README:
* README_ELPA:
* doc/org-guide.org (Creating Footnotes):
* doc/org-manual.org (Creating Footnotes):
* etc/styles/README (URL): Prefer "website" to "homepage".
* lisp/*.el:
* mk/org-fixup.el: Replace the "Homepage" header comment with "URL".
Adds functionality to ob-comint.el to implement async session eval on
a per-language basis. Adds a reference implementation for ob-python.
* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.
* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.
*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
* lisp/ob-python.el (org-babel-python--exec-tmpfile): Rename tmpfile handle
(org-babel-python-format-session-value): Rename tmpfile handle
Opening the exec tmpfile as a `f' variable shadows any such variable
that might by defined by the Python session context. e.g. my Org babel
files commonly pass single letter variables inside a session which is
broken by this behavior.
The new name `__org_babel_python_tmpfile' is in line with other org
mode specific Python variables set by ob-python. This is unlikely to
conflict with the user's Python code.
TINYCHANGE
* lisp/ob-python.el (org-babel-python--exec-tmpfile): Rename tmpfile handle
(org-babel-python-format-session-value): Rename tmpfile handle
Opening the exec tmpfile as a `f' variable shadows any such variable
that might by defined by the Python session context. e.g. my Org babel
files commonly pass single letter variables inside a session which is
broken by this behavior.
The new name `__org_babel_python_tmpfile' is in line with other org
mode specific Python variables set by ob-python. This is unlikely to
conflict with the user's Python code.
TINYCHANGE
* lisp/ob-python.el (org-babel-python-eoe-indicator): Add back
variable for end-of-expression indicator.
(org-babel-python--shift-right): New function to indent source body.
(org-babel-python-evaluate-external-process): Call new function org-babel-python--shift-right.
(org-babel-python--send-string): Wrap body in try-except-finally,
print indicator token, and find end of output via the token.
cf https://orgmode.org/list/871rjcan53.fsf@kyleam.com/
* lisp/ob-python.el (org-babel-python--eval-ast): Removed.
(org-babel-python-format-session-value): New function that returns
Python code to evaluate for sessions with value results.
(org-babel-python-evaluate-session): Replace
org-babel-python--eval-ast with org-babel-python-format-session-value.
Motivation is that the new function is more flexible than the old
format string. We can pass in the full result-params, which can be
used to add more formatting options in future. We could also add more
optional arguments in the future to extend this functionality.
This function will be particularly helpful for a couple patches after
9.4:
- ob-reticulate
- https://orgmode.org/list/875z98gj4f.fsf@gmail.com/
- Improved formatting
- https://orgmode.org/list/87eenpfe77.fsf@gmail.com/
However, I add this function in now 9.4, so those future patches can
rely on it in 9.4. In particular, ob-reticulate will probably be
packaged separately from org-mode, and I want to provide a stable
interface to it so that it can be released after 9.4.
* lisp/ob-python.el (py-send-string-no-output): Remove external function.
(py-shell-send-string): Add external function.
(org-babel-python--eval-ast): Remove blank lines.
(org-babel-python--send-string): New function to send string to Python
process and return output.
(org-babel-python-evaluate-session): Call
org-babel-python--send-string to evaluate code.
New function to send code to Python session. It has the following
desirable properties:
- Wait for evaluation to finish before returning.
- Input can have arbitrary spaces/newlines.
- Avoid echoing input to the session.
- Echo all output to the session.
- Output is also captured and returned separately.
- Avoid adding extra prompts to the session.
- Work well with both Python and IPython repls.
The implementation borrows from `python-shell-send-string-no-output'
and `org-babel-comint-with-output'. This commit is related to
632ceabb1, which tried to implement the same thing, but also
introduced bugs, and had to be partially reverted in b5709ddc9.
* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Remove
workaround for emacs<24.1. Sleep after starting Python, to prevent
startup messages leaking into output results.
* testing/lisp/test-ob-python.el (test-ob-python/session-multiline):
Remove workaround for startup message leaking into results.
* lisp/ob-python.el (py-shell-send-string): Remove unneeded
declare-function.
(org-babel-python-eoe-indicator): Add back this const.
(org-babel-python--eval-ast): Fix spacing so the block can be entered
line by line into the interpreter.
(org-babel-python-evaluate-session): Revert previous changes to
evaluation for value results.
* testing/lisp/test-ob-python.el (test-ob-python/session-value-sleep):
New test for session blocks that don't return immediately.
632ceabb1 introduced a bug where session blocks don't wait for value
results if not returned immediately. This reverts part of that
commit, specifically for evaluation of session blocks with value
results. It leaves other parts of that commit alone, for example the
changes to session output results, which is more robust handling
prompts from ipython interpreter.
A downside of reverting the session value implementation is that the
ugly code from org-babel-python--eval-ast is again printed in the
console, but this is less bad than the new bug that was introduced by
the change, hence reverted.
* lisp/ob-python.el: Require python.el at top-level.
(org-babel-python-eoe-indicator): Remove unused variable.
(org-babel-python-initiate-session-by-key): Rename
python-buffer to avoid obsolete warning.
(org-babel-python-evaluate-external-process): Remove unnecessary
require.
(org-babel-python--exec-tmpfile): Simplify this template.
(org-babel-python--eval-ast): Add printing of results to this
template, requiring additional string escapes.
(org-babel-python-evaluate-session): Simplify to use adjusted
templates, and call out to functionality in python.el or
python-mode.el.
(org-babel-python-evaluate-session): Simplified to use adjusted templates.
This commit refactors and cleans up code related to session
evaluation. python.el is now required at the top-level. Python
templates for wrapping code are simplified. Instead of directly
pasting code to the REPL, functionality from python.el and
python-mode.el are used; this fixes issues with code being echoed to
the REPL, and should be generally more robust. Finally, in the
:results value case, special handling of exceptions is removed, and we
no longer print None when the last statement isn't an expression.
* lisp/ob-python.el (org-babel-python-evaluate-session): Fix
discrepancy between how single-line and multiline code blocks return
output.
For example, before this commit, the 2 blocks behaved differently:
+begin_src python :session :results output
pass
"foo"
+end_src
+RESULTS:
+begin_src python :session :results output
"foo"
+end_src
+RESULTS:
: 'foo'
But now, they both behave as the former.
* lisp/ob-python.el (org-babel-python-evaluate-session): Process
temporary file name with `org-babel-process-file-name' before
inserting it into code Python code snippets.
Before this change, the entire temporary filename was sent to the
Python session for execution, causing a 'No such file' error when the
filename had a Tramp format such as
/ssh:user@server:/tmp/python-ABCDEF.
TINYCHANGE
* lisp/ob-python.el (org-babel-python-evaluate-external-process): Use
functions from python.el to indent lines, avoiding multiline strings.
* testing/lisp/test-ob-python.el (test-ob-python/multiline-var): Set
test as expected to succeed.
(test-ob-python/multiline-str): Add test for multiline string in body.
(test-ob-python/header-var-assignment): Test that :var is in correct
scope and can be assigned to.
cf. https://orgmode.org/list/87tv009l9a.fsf@gmail.com/#t
* lisp/ob-python.el (org-babel-execute:python,
org-babel-python-evaluate): Move variable assignment back to body from
preamble.
* testing/lisp/test-ob-python.el (test-ob-python/multiline-var): Mark
unit test for multiline-variable as expected to fail.
Reverts 6149b6cb6 which fixed multiline vars in ob-python, but caused
a new bug (https://lists.gnu.org/archive/html/emacs-orgmode/2020-05/msg00719.html).
Ihor Radchenko
Kyle Meyer
Po Lu
Jack Kamm
Sébastien Miquel
Gerard Vermeulen
Liu Hui
Matthew Trzcinski
Eli Zaretskii
Ihor Radchenko
Stefan Kangas
Bastien
Adrian Kummerlaender
Christian Vanderwall