• table of contents
• overview
  • features and use cases
  • for new users
  • config structure
• library
  • external
    • dash
    • dash functional
  • macros
  • functions
  • interactive
• user interface
  • theme
  • modeline
    • delight
  • remove interface bars
  • startup screen
  • windows
    • popup windows
    • ace-window
  • navigation
    • helm
    • helm-swoop
    • avy
  • cursor
  • misc
    • line wrap
    • smooth scrolling
    • imagemagick
    • yes-no prompt
• low-level config
  • user information
  • autosave
  • async
• editing
  • standardization
    • tabs and alignment
    • short column width
    • spell checking
    • syntax checking
    • packaging
  • auto completion
    • yasnippet
  • undo
  • parenthesis matching
  • sudo edit
  • formats and languages
    • Elisp
    • ESS (Emacs Speaks Statistics)
    • Python
    • Ruby
    • Haskell
      • major mode and intero
      • camelCase
    • Lua
    • TeX
      • AUCTeX
      • external viewers
      • folding and outlining
      • auto completion
      • syntax check
      • spell check
      • line wrap
      • BibTeX
        • database management
        • citation search and insertion
    • CSS
    • Javascript
    • markdown
    • R-markdown
    • csv files
• org-mode
  • low-level config
    • modules
    • directory
    • autosave
  • buffer interface
    • line wrap
    • indentation
    • special key behavior
    • bullets
    • font height
    • src blocks
    • todo insertion
    • table of contents
    • column view
  • extra commands
    • org buffer
    • org agenda
  • calfw
  • window splitting
    • todo selection
    • tag selection
    • capture
  • exporting
    • latex to pdf command
    • custom output directory
    • html5
  • gantt charts
  • gtd implementation
    • overview
      • workflow
        • collect
        • process
        • organize
        • review
        • execute
      • file hierarchy and structure
      • repetition
      • block agenda views
        • projects
        • repeaters
      • terms and definitions
        • headings
        • time
    • todo states
      • sequences
      • colors
    • tags
      • alist
      • colors
    • properties
    • capture
      • templates
      • insert mode
    • refile
    • clocking
      • general
      • modeline
    • clustering
      • extraction filters
      • timestamp extraction and filtering
      • conflict detection
      • overload detection
      • frontend
    • agenda
      • targets
      • appearence
        • sticky agendas
        • tag alignment
        • prefix format
        • modeline
        • misc
      • interactive filters
        • custom filtering functions
        • filter advice
      • bulk actions
      • holidays and birthdays
      • block agenda library
        • constants
        • variables
        • task helper functions
          • org-element
          • timestamps
          • task level testing
          • property testing
          • relational testing
          • project level testing
          • repeater testing
        • skip functions
          • helper skip functions and macros
          • headings
          • tasks
          • atomic tasks
          • repeaters
          • project tasks
          • heading-level errors
          • projects
        • sorting and filtering
        • block view building macros
        • interactive functions
        • advising
          • org-tags-view done keywords
      • block agenda views
        • default sorting
        • custom commands
  • gtd next generation
    • logging
      • drawer
      • events
      • repeated tasks
      • creation time
    • sqlite backend
      • load path
      • customized variables
• tools
  • printing
  • magit
  • dired
    • no confirm
    • compression
    • formatting for humans
    • mu4e attachments
    • directory sized
    • mounted devices
    • filtering
  • mu4e
  • shell
  • ediff
  • jupyter
• keybindings
  • setup
  • whichkey
  • evil
    • base
    • search
    • motion
    • evil state defaults
    • enhancements
      • surround
      • commentary
      • replace with register
    • unbind emacs keys
    • evil-org
    • evil-magit
    • visual line mode
    • comint
      • interactive functions
      • ess
      • intero
    • collection
      • dired
      • helm
      • term
  • local
    • org-mode
    • mu4e
    • dired
    • helm-prefix
    • outline-magic
    • ess
  • global
    • function
    • control/meta

This is my personal emacs config. It is quite massive. Please use the table of contents below for easy navigation ;)

table of contents   TOC

• overview

  • features and use cases
  • for new users
  • config structure

• library

  • external
  • macros
  • functions
  • interactive

• user interface

  • theme
  • modeline
  • remove interface bars
  • startup screen
  • windows
  • navigation
  • cursor
  • misc

• low-level config

  • user information
  • autosave
  • async

• editing

  • standardization
  • auto completion
  • undo
  • parenthesis matching
  • sudo edit
  • formats and languages

• org-mode

  • low-level config
  • buffer interface
  • extra commands
  • calfw
  • window splitting
  • exporting
  • gantt charts
  • gtd implementation
  • gtd next generation

• tools

  • printing
  • magit
  • dired
  • mu4e
  • shell
  • ediff

• keybindings

  • setup
  • whichkey
  • evil
  • local
  • global

overview

features and use cases

• full GTD implementation with org-mode to help me stay organized
• unified interface for common linux tools (dired, shell, git, ediff)
• fully customizable email client with mu4e
• optimizations for some of my favorite languages (R, Lisp, Haskell, Lua, Python)
• document preparation with latex

for new users

Feel free to take bits and pieces for your own configuration file. Like many things in emacs, the config file is quite self documenting; however, there are some useful ramblings that decribe why I made some design choices over others. As someone who learned from countless emacs configs of other experienced users, I thought it was extremely beneficial to see the thought process behind their workflow and code, and I hope my annotations pay that forward. Finally, please don't just blindly copy this config into your ~/.emacs.d. I don't care if you do, but you will learn more if you build from scratch.

config structure

The "config file" is actually two files.

The "root" is init.el which is the file explicitly loaded by emacs. Most users have their entire config in this file but I put most of my actuall settings in another file as explained in the next paragraph. Here init.el has minimum functionality, including setting the repositories, configuring use-package (which installs all other packages and ensures they are available, useful if I move this elsewhere), and load paths for other config file.

Once loaded, the init.el pulls in another file called conf.el with the function org-babel-load-file. conf.el is actually sourced from an org file called conf.org.

Using an org file like this offers several advantages. First, org files are foldable in emacs which makes navigation easy. Second, they allow code snippets (the bit that actually go into conf.el) which allows for explanatory prose to be written around them, making documentation easy and clear. Third, org-mode has an automatic table of contents through the toc-org package, which makes naviagation even easier. Fourth, github itself is awesome enough to recognize org files as valid markdown and will render all the text, code snippets, headers, and table of contents in the nice html that you are reading now if on github. The result is a nearly self-documenting, self-organizing configuration that is easy to maintain and also easy to view for other users. Using the init.el itself would just have plain eLisp, which gets cluttered quickly. Some people break the init.el down into multiple files to keep everything sane, but I personally think it is easier to use one giant file that itself can be folded and abstracted to reduce the clutter.

library

This is code that is used generally throughout the emacs config

external

dash

(use-package dash
  :ensure t
  :config
  (setq dash-enable-fontlock t))

dash functional

(use-package dash-functional
  :ensure t
  :config)

macros

;; lovingly stolen from aaron harris
(defmacro nd/with-advice (adlist &rest body)
  "Execute BODY with temporary advice in ADLIST.

Each element of ADLIST should be a list of the form
  (SYMBOL WHERE FUNCTION [PROPS])
suitable for passing to `advice-add'.  The BODY is wrapped in an
`unwind-protect' form, so the advice will be removed even in the
event of an error or nonlocal exit."
  (declare (debug ((&rest (&rest form)) body))
           (indent 1))
  `(progn
     ,@(mapcar (lambda (adform)
                 (cons 'advice-add adform))
               adlist)
     (unwind-protect (progn ,@body)
       ,@(mapcar (lambda (adform)
                   `(advice-remove ,(car adform) ,(nth 2 adform)))
                 adlist))))

(defmacro nd/when-os (os &rest body)
  "Execute BODY if the operating system is OS.
OS is one of those in `system-type'."
  `(if (eq system-type ,os) (progn ,@body)
     (print "Skipping OS-restricted code")))

(defmacro nd/when-not-os (os &rest body)
  "Execute BODY if the operating system is not OS.
OS is one of those in `system-type'."
  `(when (not (eq system-type ,os)) (progn ,@body)
     (print "Skipping OS-restricted code")))

(defmacro nd/when-bin (bin &rest body)
  "Execute BODY if the program BIN exists."
  `(if (executable-find ,bin) (progn ,@body)
     (print (format "Executable %s not found. Skipping." ,bin))))

functions

(defun nd/filter-list-prefix (prefix str-list)
  "Return a subset of STR-LIST whose first characters are PREFIX."
  (seq-filter (lambda (i)
                (and (stringp i)
                     (string-prefix-p prefix i)))
              str-list))

(defun nd/move-key (keymap-from keymap-to key)
  "Move KEY from KEYMAP-FROM keymap to KEYMAP-TO keymap."
  (define-key keymap-to key (lookup-key keymap-from key))
  (define-key keymap-from key nil))

(defun nd/get-apps-from-mime (mimetype)
  "Return all applications that can open a given MIMETYPE.
The list is comprised of alists where pairs are of the form (name . command)."
  (let* ((case-fold-search nil)
         (mime-regex (concat "^MimeType=.*" mimetype ";.*$"))
         (desktop-dirs '("/usr/share/applications"
                         "/usr/local/share/applications"
                         "~/.local/share/applications"))
         (desktop-files (mapcan (lambda (d) (directory-files d t ".*\\.desktop" t)) desktop-dirs))
         (app-list))
    (dolist (file desktop-files app-list)
      (with-temp-buffer
        (insert-file-contents file)
        (let* ((tb (buffer-string)))
          (if (string-match mime-regex tb)
              (let* ((exec (progn (string-match "^Exec=\\(.*\\)$" tb)
                                  (match-string 1 tb)))
                     (name (or
                            (progn (string-match "^Name=\\(.*\\)$" tb)
                                   (match-string 1 tb))
                            exec)))
                (setq app-list (cons `(,name . ,exec) app-list)))))))))

(defun nd/get-apps-bulk-from-mime (mimetype)
  "Like `nd/get-apps-from-mime' but only includes apps that can open
multiple files at once for given MIMETYPE."
  (let ((case-fold-search nil))
    (seq-filter (lambda (a) (string-match ".*%[FU].*" (car a))) (nd/get-apps-from-mime mimetype))))
    
(defun nd/execute-desktop-command (cmd file)
  "Opens FILE using CMD in separate process where CMD is from a 
desktop file exec directive."
  (let* ((cmd-arg (replace-regexp-in-string "%[fuFU]" file cmd t t)))
    (call-process-shell-command (concat cmd-arg " &"))))
  
(defun nd/get-mime-type (file)
  "Get the mime type of FILE."
  (let* ((cmd (concat "file --mime-type -b " file))
         (mt (shell-command-to-string cmd)))
    (replace-regexp-in-string "\n\\'" "" mt)))

(defvar nd/device-mount-dir (concat "/media/" (user-login-name)))

(defun nd/get-mounted-directories (&optional mount-path)
  "Scan MOUNT-PATH (defaults to /media/$USER for devices that have
been mounted by udevil."
  (seq-filter #'file-directory-p (directory-files nd/device-mount-dir t "^\\([^.]\\|\\.[^.]\\|\\.\\..\\)")))

(defun nd/device-mountable-p (devpath)
  "Returns label or uuid if device at DEVPATH is has a readable 
filesystem and is a usb drive."
  (let ((devprops (shell-command-to-string (concat "udevadm info --query=property " devpath))))
    (and (string-match-p (regexp-quote "ID_FS_TYPE") devprops)
         (string-match-p (regexp-quote "ID_BUS=usb") devprops)
         (progn
           (or (string-match "ID_FS_LABEL=\\(.*\\)\n" devprops)
               (string-match "ID_FS_UUID=\\(.*\\)\n" devprops))
           (match-string 1 devprops)))))

(defun nd/get-mountable-devices ()
  "Return paths of all mountable devices. (see `nd/device-mountable-p')."
  (seq-filter #'car
              (mapcar (lambda (d) `(,(nd/device-mountable-p d) . ,d))
                      (directory-files "/dev" t "sd.[0-9]+"))))

(defun nd/mount-device (dev &rest opts)
  "Mount device DEV using udevil."
  (call-process "udevil" nil nil nil "mount" dev))

(defun nd/get-mountpoint (dev)
  "Get the filesystem mountpoint for device DEV."
  (let ((mp (shell-command-to-string (concat "printf %s \"$(findmnt -n -o TARGET " dev ")\""))))
    (and (not (equal "" mp)) mp)))

(defun nd/print-args (orig-fun &rest args)
  "Prints ARGS of ORIG-FUN. Intended as :around advice."
  (print args)
  (apply orig-fun args))

(defun nd/plist-put-append (plist prop value &optional front)
  "Like `plist-put' but append VALUE to current values in PLIST for PROP.
If FRONT is t, append to the front of current values instead of the back."
  (let* ((cur (plist-get plist prop))
         (new (if front (append value cur) (append cur value))))
    (plist-put plist prop new)))

(defun nd/plist-put-list (plist prop value &optional front)
  "Like `plist-put' but append (list VALUE) to current values in PLIST for PROP.
If FRONT is t, do to the front of current values instead of the back."
  (let* ((cur (plist-get plist prop))
         (new (if front (append (list value) cur) (append cur (list value)))))
    (plist-put plist prop new)))

interactive

(defun nd/split-and-follow-horizontally ()
  "Split window horizontally and move focus."
  (interactive)
  (split-window-below)
  (balance-windows)
  (other-window 1))

(defun nd/split-and-follow-vertically ()
  "Split window vertically and move focus."
  (interactive)
  (split-window-right)
  (balance-windows)
  (other-window 1))
    
(defun nd/switch-to-previous-buffer ()
  "Switch the buffer to the last opened buffer."
  (interactive)
  (switch-to-buffer (other-buffer (current-buffer) 1)))
  
(defun nd/config-reload ()
  "Reloads main configuration file at runtime."
  (interactive)
  (org-babel-load-file nd/conf-main))

(defun nd/config-visit ()
  "Opens the main conf.org file (the one that really matters)."
  (interactive)
  (find-file nd/conf-main))

(defun nd/kill-current-buffer ()
  "Kill the current buffer."
  (interactive)
  (kill-buffer (current-buffer)))

(defun nd/close-all-buffers ()
  "Kill all buffers without regard for their origin."
  (interactive)
  (mapc 'kill-buffer (buffer-list)))

(defun nd/org-close-all-buffers ()
  "Kill all org buffers."
  (interactive)
  (mapc 'kill-buffer (org-buffer-list)))

(defun nd/open-urxvt ()
  "Launch urxvt in the current directory."
  (interactive)
  (let ((cwd (expand-file-name default-directory)))
    (call-process "urxvt" nil 0 nil "-cd" cwd)))

user interface

The general look and feel, as well as interactive functionality

theme

This theme has good functionality for many different modes without being over-the-top or overly complex. It also comes with an easy way to set custom colors.

(use-package spacemacs-theme
  :ensure t
  :defer t
  :config
  (setq spacemacs-theme-custom-colors '((lnum . "#64707c"))))

Since I run emacs in client/server mode, the loaded theme can change depending on if the client is a terminal or server (terminals have far fewer colors). This makes the theme reset when terminal is loaded before gui or vice versa.

(defvar nd/theme 'spacemacs-dark)
(defvar nd/theme-window-loaded nil)
(defvar nd/theme-terminal-loaded nil)

;; required for emacsclient/daemon setup
(if (daemonp)
    (add-hook 'after-make-frame-functions
              (lambda (frame)
                (select-frame frame)
                (if (window-system frame)
                    (unless nd/theme-window-loaded
                      (if nd/theme-terminal-loaded
                          (enable-theme nd/theme)
                        (load-theme nd/theme t))
                      (setq nd/theme-window-loaded t))
                  (unless nd/theme-terminal-loaded
                    (if nd/theme-window-loaded
                        (enable-theme nd/theme)
                      (load-theme nd/theme t))
                    (setq nd/theme-terminal-loaded t)))))
  (progn
    (load-theme nd/theme t)
    (if (display-graphic-p)
        (setq nd/theme-window-loaded t)
      (setq nd/theme-terminal-loaded t))))

modeline

This modeline goes along with the spacemacs-theme. It also has nice integration with evil-mode (see keybindings below).

(use-package spaceline
  :ensure t
  :config
  (require 'spaceline-config)
  (setq powerline-default-separator 'arrow
        spaceline-buffer-size-p nil
        spaceline-buffer-encoding-abbrev-p nil)
  (spaceline-spacemacs-theme))

(line-number-mode 1)
(column-number-mode 1)

delight

I like to keep the modeline clean and uncluttered. This package prevents certain mode names from showing in the modeline (it also has support for use-package through the :delight keyword)

(use-package delight
  :ensure t)

remove interface bars

Emacs comes with some useless garbage by default. IMHO (in my haughty opinion), text editors should be boxes with text in them. No menu bars, scroll bars, or toolbars (and certainly no ribbons).

(tool-bar-mode -1)
(menu-bar-mode -1)
(scroll-bar-mode -1)

startup screen

Default startup screen is silly

(setq inhibit-startup-screen t)

Instead use a dashboard, and display days until predicted death…you know, as a pick-me-up ;)

(defvar nd/user-birthday 727506000
  "User date of birth in unix time")

(defvar nd/predicted-age-at-death 71.5
  "Expected age that user will die.")

(defun nd/deathclock (list-size) 
  (let ((death-ut (-> nd/predicted-age-at-death
                      (* 31557600)
                      (+ nd/user-birthday))))
    (insert (--> (float-time)
                 (- death-ut it)
                 (/ it 86400)
                 (round it)
                 (format "%s days until death" it)))))
  
(use-package dashboard
  :ensure t
  :config 
  (setq dashboard-banner-logo-title nil
        dashboard-startup-banner "~/.emacs.d/dashlogo.png"
        dashboard-items '(deathclock))
  (add-to-list 'dashboard-item-generators '(deathclock . nd/deathclock))
  (dashboard-setup-startup-hook))

windows

popup windows

Some modes like to make popup windows (eg ediff). This prevents that.

(setq pop-up-windows nil)

ace-window

This is an elegant window selector. It displays a number in the corner when activated, and windows may be chosen by pressing the corresponding number. Note that spacemacs fails to make the numbers look nice so the theme code is a workaround to make them smaller and prettier.

(use-package ace-window
  :ensure t
  :config
  (setq aw-background t)
  (custom-set-faces '(aw-leading-char-face 
                      ((t (:foreground "#292b2e"
                           :background "#bc6ec5"
                           :height 1.0
                           :box nil))))))

navigation

helm

One of the best packages for emacs. Helm is basically a search and completion engine (other exanples being ido-mode and ivy-mode) which is mainly used for finding files and selecting commands (which are obviously used often). It also integrates well with many other modes such as evil-mode and org-mode.

(use-package helm
  :ensure t
  :delight
  :init
  (helm-mode 1)
  :config
  (setq helm-autoresize-max-height 0
        helm-autoresize-max-height 40
        helm-M-x-fuzzy-match t
        helm-buffers-fuzzy-matching t
        helm-recentf-fuzzy-match t
        helm-semantic-fuzzy-match t
        helm-imenu-fuzzy-match t
        helm-scroll-amount 8)
  (add-to-list 'display-buffer-alist
               `(,(rx bos "*helm" (* not-newline) "*" eos)
                 (display-buffer-in-side-window)
                 (inhibit-same-window . t)
                 (window-height . 0.4)))
  (helm-autoresize-mode 1)
  (require 'helm-config))

helm-swoop

(use-package helm-swoop
  :ensure t)

avy

Allows jumping to any character in any window with a few keystrokes. Goodbye mouse :)

  (use-package avy
    :ensure t
    :config
    (setq avy-background t))

cursor

This makes a nice glowy effect on the cursor when switching window focus. Very elegant way of saving time in finding where you left off.

(use-package beacon
  :ensure t
  :delight
  :init
  (beacon-mode 1))

misc

line wrap

I don't like line wrap

(set-default 'truncate-lines t)

smooth scrolling

This makes scrolling smoother

(setq scroll-conservatively 100)

imagemagick

(when (fboundp 'imagemagick-register-types)
  (imagemagick-register-types))

yes-no prompt

Some prompts require literal "yes" or "no" to decide action. Life is short and I would rather not waste keystrokes typing whole words. This makes all "yes/no" prompts only require "y" or "n."

(defalias 'yes-or-no-p 'y-or-n-p)

low-level config

General configuation for behind-the-scenes behavior

user information

(setq user-full-name "Dwarshuis, Nathan J")

autosave

Saving files continuously is actually really annoying and clutters my disk. Turn it off.

(setq make-backup-files nil)
(setq auto-save-default nil)

async

Allows certain processes to run in multithreaded manner. For things like IO this makes sense.

(use-package async
  :ensure t
  :delight dired-async-mode
  :init
  (dired-async-mode 1))

editing

For options that specifically affect programming or editing modes

standardization

tabs and alignment

Who uses tabs in their programs? Make tabs actually equal 4 spaces. Also, alledgedly I could make more money if I use spaces :)

(setq-default indent-tabs-mode nil
              tab-width 4)

short column width

Alot of languages at least semi-adhere to the 80-characters-per-line rule. fci-mode displays a line as a guide for column width.

(use-package fill-column-indicator
  :ensure t
  :config
  (setq fci-rule-use-dashes t)
  :hook
  (prog-mode . fci-mode))

spell checking

I use the built-in flyspell-mode to handle spellchecking. Obviously I am going to use helm when I spellcheck something.

(use-package flyspell-correct-helm
  :ensure t
  :after (helm flyspell))

This will spell-check comments in programming languages.

(add-hook 'prog-mode-hook #'flyspell-prog-mode)
(setq flyspell-issue-message-flag nil)

Since flyspell mode is enabled in so many buffers, use a short modeline alias.

(delight 'flyspell-mode "σ" "flyspell")

Additionally, I want to automatically highlight errors whenever flyspell-mode is enabled.

;; (add-hook 'flyspell-mode-hook 'flyspell-buffer)

syntax checking

Flycheck will highlight and explain syntax errors in code and formatting.

(use-package flycheck
  :ensure t
  :hook
  (prog-mode . flycheck-mode)
  :config
  (setq flycheck-check-syntax-automatically '(save
                                              idle-change
                                              mode-enabled)
        flycheck-idle-change-delay 2
        flycheck-error-list-minimum-level 'warning
        flycheck-navigation-minimum-level 'warning)
  (delight 'flycheck-mode "ϕ" "flycheck"))

packaging

(use-package flycheck-package
  :ensure t
  :after flycheck
  :config
  (eval-after-load 'flycheck '(flycheck-package-setup)))

auto completion

Company provides a dropdown of completion options. It has many backends which are configured in each language and format elsewhere.

(use-package company
  :ensure t
  :delight " ©"
  :config
  (setq company-idle-delay 0
        company-minimum-prefix-length 3))

yasnippet

(use-package yasnippet
  :ensure t)

(use-package yasnippet-snippets
  :ensure t
  :after yasnippet
  :hook
  ((prog-mode . yas-minor-mode))
  :config
  (yas-reload-all))

undo

I find it weird that most programs do not have a tree-like tool to navigate undo information…because this is literally how most programs store this data.

undo-tree package adds a nice undo tree buffer to visualize history and also displays diffs to easily show what changed.

(use-package undo-tree
  :ensure t
  :delight
  :config
  (setq undo-tree-visualizer-diff t)
  (global-undo-tree-mode))

parenthesis matching

This color-codes matching parenthesis. Enable pretty much everywhere.

(use-package rainbow-delimiters
  :ensure t
  :delight
  :hook
  ((prog-mode . rainbow-delimiters-mode)
   (inferior-ess-mode . rainbow-delimiters-mode)
   (ess-mode . rainbow-delimiters-mode)
   (LaTeX-mode . rainbow-delimiters-mode)
   (Tex-latex-mode . rainbow-delimiters-mode)))

Use pretty symbols (like lambda in lisp)

(add-hook 'prog-mode-hook #'prettify-symbols-mode)

sudo edit

Allows opening a file with sudo elevation.

  (use-package sudo-edit
    :ensure t)

formats and languages

Elisp

Elisp can use vanilla company with no plugins

(add-hook 'emacs-lisp-mode-hook 'company-mode)

ESS (Emacs Speaks Statistics)

For me this means R but ess also supports S-plus, SAS, Stata, and other statistical black-magic languages. Note that ESS is not part of prog-mode so it must be added manually to hooks.

A few caveats when using R

• ess-mode requires a running R process for company-mode to work
• flycheck-mode requries r-lintr

(defun nd/init-ess-company ()
  "Set the company backends for ess modes."
  (setq-local company-backends '((company-R-objects company-R-args))))

(use-package ess
  :ensure t
  :init
  (load "ess-site")
  :hook
  ((ess-mode . flycheck-mode)
   (ess-mode . company-mode)
   (ess-mode . nd/init-ess-company)
   (ess-mode . prettify-symbols-mode)
   (ess-mode . fci-mode)

   (inferior-ess-mode . company-mode)
   (inferior-ess-mode . nd/init-ess-company)
   (inferior-ess-mode . prettify-symbols-mode))
  :config
  (setq inferior-R-args "--quiet --no-save"
        ess-history-file "session.Rhistory"
        ess-history-directory (substitute-in-file-name "${XDG_CONFIG_HOME}/r/")))

Python

(elpy-enable)

;; make python tabs 4 chars
(add-hook 'python-mode-hook
      (lambda ()
        (setq indent-tabs-mode t)
        (setq tab-width 4)
        (setq python-offset 4)))
        
(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "--simple-prompt")

Ruby

(use-package inf-ruby
  :ensure t)

Haskell

major mode and intero

Haskell is covered just with the basic major mode and intero (provides company and flycheck) which integrates well with stack.

(use-package haskell-mode
  :ensure t
  :config
  (setq haskell-interactive-popup-errors nil))
  
(use-package intero
  :ensure t
  :after haskell-mode
  :hook
  (haskell-mode . intero-mode))

camelCase

The defacto style for haskell mandates camelcase, so use subword mode.

(add-hook 'haskell-mode-hook #'subword-mode)
(delight 'subword-mode nil "subword")

Lua

(use-package lua-mode
  :ensure t)

TeX

AUCTeX

Install auctex through emacs as this is OS independent and more automatic. Note that the Tex package libraries (eg TeXLive) still need to be installed to do anything useful.

(use-package tex
  :ensure auctex)

external viewers

AUCTeX can launch external viewers to show compiled documents. I use Okular for PDFs.

(setq TeX-view-program-selection '(((output-dvi has-no-display-manager)
                                    "dvi2tty")
                                   ((output-dvi style-pstricks)
                                    "dvips and gv")
                                   (output-dvi "xdvi")
                                   (output-pdf "Okular")
                                   (output-html "xdg-open")))

folding and outlining

I like how org-mode folds with the TAB key, so bring the same thing to AUCTeX here with outline-magic.

(add-hook 'LaTeX-mode-hook (lambda () (outline-minor-mode 1)))
(add-hook 'Tex-latex-mode-hook (lambda () (outline-minor-mode 1)))

(use-package outline-magic
  :ensure t
  :after outline)

Also, the section fonts are too big by default. Now the sizes are all kept equal with hatchet, axe, and saw :)

(setq font-latex-fontify-sectioning 'color)

auto completion

There are two backends which (kinda) complement each other. The company-math package should privide completion for math symbols and the company-auctex package should cover pretty much everything else.

(defun nd/init-company-auctex ()
  "Set the company backends for auctex modes."
  (setq-local company-backends '((company-auctex-labels
                                  company-auctex-bibs
                                  company-auctex-macros
                                  company-auctex-symbols
                                  company-auctex-environments
                                  ;; company-latex-commands
                                  company-math-symbols-latex
                                  company-math-symbols-unicode))))

(use-package company-math
  :ensure t
  :after company
  :config
  (setq company-math-allow-unicode-symbols-in-faces '(font-latex-math-face)
        company-math-disallow-latex-symbols-in-faces nil))

(use-package company-auctex
  :ensure t
  :after (company company-math)
  :hook
  ((LaTeX-mode . company-mode)
   (LaTeX-mode . nd/init-company-auctex)
   (Tex-latex-mode . company-mode)
   (Tex-latex-mode . nd/init-company-auctex)))

syntax check

Flycheck should work out of the box.

(add-hook 'LaTeX-mode-hook #'flycheck-mode)
(add-hook 'Tex-latex-mode-hook #'flycheck-mode)

spell check

Spell checking is important for prose

(add-hook 'LaTeX-mode-hook (lambda () (flyspell-mode 1)))

line wrap

I like having my lines short and readable (also easier to git). Turn on autofill here and also make a nice vertical line at 80 chars (visual-line-mode).

(defun nd/turn-on-auto-fill-maybe ()
  "Prompts user to turn on `auto-fill-mode'."
  (when (y-or-n-p "Activate Auto Fill Mode? ")
    (turn-on-auto-fill)))
  
(add-hook 'LaTeX-mode-hook #'nd/turn-on-auto-fill-maybe)
(add-hook 'LaTeX-mode-hook #'fci-mode)

BibTeX

database management

(use-package ebib
  :ensure t
  :config
  (setq ebib-autogenerate-keys t
        ebib-uniquify-keys t))

citation search and insertion

Together, org-ref and helm-bibtex provide a nice pipeline to search a BibTex database and insert citations.

(use-package org-ref
  :ensure t
  :after org
  :config
  (setq reftex-default-bibliography (expand-file-name "~/BibTeX/master.bib")
        org-ref-bibliography-notes (expand-file-name "~/BibTeX/notes.org")
        org-ref-default-bibliography (expand-file-name "~/BibTeX/master.bib")))
        
(use-package helm-bibtex
  :ensure t
  :after helm
  :config
  (setq bibtex-completion-bibliography (expand-file-name "~/BibTeX/master.bib")
        bibtex-completion-library-path (expand-file-name "~/BibTeX/pdf")
        bibtex-completion-pdf-field "File"))

CSS

Overlays hex color codes with matching colors in certain modes like css and html.

(use-package rainbow-mode
  :ensure t)

Javascript

(use-package js-comint
  :ensure t)

markdown

Make font sizes smaller and less intrusive for headers

(add-hook 'markdown-mode-hook
          (lambda ()
            (let ((heading-height 1.15))
              (set-face-attribute 'markdown-header-face-1 nil :weight 'bold :height heading-height)
              (set-face-attribute 'markdown-header-face-2 nil :weight 'semi-bold :height heading-height)
              (set-face-attribute 'markdown-header-face-3 nil :weight 'normal :height heading-height)
              (set-face-attribute 'markdown-header-face-4 nil :weight 'normal :height heading-height)
              (set-face-attribute 'markdown-header-face-5 nil :weight 'normal :height heading-height))))

R-markdown

R-markdown is enabled via polymode, which allows multiple modes in one buffer (this is actually as crazy as it sounds). In this case, the modes are yaml, R, markdown, and others. Installing poly-R will pull in all required dependencies.

(use-package poly-R
  :ensure t
  :mode
  (("\\.Rmd\\'" . poly-markdown+r-mode)
   ("\\.rmd\\'" . poly-markdown+r-mode)))

csv files

This adds support for csv files. Almost makes them editable like a spreadsheet. The lambda function enables alignment by default.

(use-package csv-mode
  :ensure t
  :hook (csv-mode . (lambda () (csv-align-fields nil (point-min) (point-max)))))

org-mode

low-level config

modules

Org has several extensions in the form of loadable modules. org-protocol is used as a backend for external programs to communicate with org-mode. org-habit allows the habit todoitem which is used as a more flexible recurring task.

(setq org-modules '(org-habit org-protocol))
(require 'org)
;; make sure everything else works that I have customly defined
(require 'org-agenda)
(require 'org-protocol)
(require 'org-habit)
(require 'org-clock)

directory

I keep all my org files in one place.

(setq org-directory "~/Org")

autosave

Save all org buffers 1 minute before the hour.

(defun nd/org-save-all-org-buffers ()
  "Save org buffers without confirmation or message (unlike default)."
  (save-some-buffers t (lambda () (derived-mode-p 'org-mode)))
  (when (featurep 'org-id) (org-id-locations-save)))

(run-at-time "00:59" 3600 #'nd/org-save-all-org-buffers)

buffer interface

line wrap

I often write long, lengthy prose in org buffers, so use visual-line-mode to make lines wrap in automatic and sane manner.

(add-hook 'org-mode-hook #'visual-line-mode)
(delight 'visual-line-mode nil 'simple)

indentation

By default all org content is squished to the left side of the buffer regardless of its level in the outline. This is annoying and I would rather have content indented based on its level just like most bulleted lists. This is what org-indent-mode does.

(setq org-startup-indented t)
(delight 'org-indent-mode nil "org-indent")

special key behavior

TODO: These don't work in evil mode (using the usual line commands).

(setq org-special-ctrl-a/e t
      org-special-ctrl-k t
      org-yank-adjusted-subtrees t)

bullets

These are just so much better to read

(use-package org-bullets
  :ensure t
  :hook
  (org-mode . org-bullets-mode))

font height

The fonts in org headings bug me; make them smaller and less invasive.

(add-hook 'org-mode-hook
          (lambda ()
            (let ((heading-height 1.15))
              (set-face-attribute 'org-level-1 nil :weight 'bold :height heading-height)
              (set-face-attribute 'org-level-2 nil :weight 'semi-bold :height heading-height)
              (set-face-attribute 'org-level-3 nil :weight 'normal :height heading-height)
              (set-face-attribute 'org-level-4 nil :weight 'normal :height heading-height)
              (set-face-attribute 'org-level-5 nil :weight 'normal :height heading-height))))

src blocks

Enable shortcuts for embedding code in org text bodies.

(setq org-src-window-setup 'current-window
      org-src-fontify-natively t
      org-edit-src-content-indentation 0)

(add-to-list 'org-structure-template-alist
             '("el" "#+BEGIN_SRC emacs-lisp\n?\n#+END_SRC"))

todo insertion

Make todo insertion respect contents

(setq org-insert-heading-respect-content t)

table of contents

Since I use org mode as my config file, makes sense to have a table of contents so others can easily naviagate this crazy empire I have created :)

(use-package toc-org
  :ensure t
  :hook
  (org-mode . toc-org-mode))

column view

  (setq org-columns-default-format
        "%25ITEM %4TODO %TAGS %5Effort{:} %DELEGATE(DEL)")

  (set-face-attribute 'org-column nil :background "#1e2023")
  ;; org-columns-summary-types

extra commands

org buffer

Some useful additional commands for org buffers.

(defun nd/mark-subtree-keyword (new-keyword &optional exclude no-log)
  "Mark all tasks in a subtree with NEW-KEYWORD unless original
keyword is in the optional argument EXCLUDE."
  (let ((subtree-end (save-excursion (org-end-of-subtree t)))
        (org-todo-log-states (unless no-log org-todo-log-states)))
    (if (not (listp exclude))
        (error "exlude must be a list if provided"))
    (save-excursion
      (while (< (point) subtree-end)
        (let ((keyword (nd/is-todoitem-p)))
          (if (and keyword (not (member keyword exclude)))
              (org-todo new-keyword)))
        (outline-next-heading)))))

(defun nd/mark-subtree-done ()
  "Mark all tasks in subtree as DONE unless they are already CANC."
  (interactive)
  (nd/mark-subtree-keyword "DONE" '("CANC")))

(defun nd/org-clone-subtree-with-time-shift (n &optional shift)
  "Like `org-clone-subtree-with-time-shift' except it resets checkboxes
and reverts all todo keywords to TODO."
  (interactive "nNumber of clones to produce: ")
    
  (let ((shift (or (org-entry-get nil "TIME_SHIFT" 'selective)
                   (read-from-minibuffer
                    "Date shift per clone (e.g. +1w, empty to copy unchanged): "))))
    (condition-case err
          (save-excursion
          ;; clone once and reset
            (org-clone-subtree-with-time-shift 1 shift)
            (org-forward-heading-same-level 1 t)
            (org-reset-checkbox-state-subtree)
            (nd/mark-subtree-keyword "TODO" nil t)
            (call-interactively 'nd/org-log-delete)
            (org-cycle)
            ;; clone reset tree again if we need more than one clone
            (if (> n 1)
                (let ((additional-trees (- n 1)))
                  (org-clone-subtree-with-time-shift additional-trees shift)
                  (dotimes (i additional-trees)
                    (org-forward-heading-same-level 1 t)
                    (org-cycle)))))
      (error (message "%s" (error-message-string err))))))

(defun nd/org-clone-subtree-with-time-shift-toplevel (n)
  "Go to the last item underneath an iterator and clone using
`nd/org-agenda-clone-subtree-with-time-shift'. Assumes point starts on
the top level headline and only looks at the second level of
headlines to clone."
  (interactive "nNumber of clones to produce: ")
  ;; do nothing if there is nothing to clone
  (unless (eq :uninit
              (or (and (nd/is-iterator-heading-p)
                       (nd/get-iterator-status))
                  (and (nd/is-periodical-heading-p)
                       (nd/get-periodical-status))))
    ;; goto last item in the second level
    (save-excursion
      (let ((current-point (point)))
        (outline-next-heading)
        (while (< current-point (point))
          (setq current-point (point))
          (org-forward-heading-same-level 1 t)))
      (nd/org-clone-subtree-with-time-shift n))))

(defun nd/org-log-delete ()
  "Delete logbook drawer of subtree."
  (interactive)
  (save-excursion
    (goto-char (org-log-beginning))
    (when (save-excursion
            (save-match-data
              (beginning-of-line 0)
              (search-forward-regexp org-drawer-regexp)
              (goto-char (match-beginning 1))
              (looking-at "LOGBOOK")))
      (org-mark-element)
      (delete-region (region-beginning) (region-end))
      (org-remove-empty-drawer-at (point)))))

(defun nd/org-delete-subtree ()
  "Delete the entire subtree under the current heading without sending to kill ring."
  (interactive)
  (org-back-to-heading t)
  (delete-region (point) (+ 1 (save-excursion (org-end-of-subtree)))))

(defun nd/org-clock-range (&optional arg)
  "Add a completed clock entry to the current heading.
Does not touch the running clock. When called with one C-u prefix
argument, ask for a range in minutes in place of the second date."
  (interactive "P")
  (let* ((t1 (-> (org-read-date t t) float-time))
         (diff (if (equal arg '(4))
                   (-some-> (read-string "Length in minutes: ")
                            (cl-parse-integer :junk-allowed t)
                            (* 60))
                 (-> (org-read-date t t nil nil t1)
                      float-time
                      round
                      (- t1)))))
    (cond
     ((not diff) (message "Invalid range given!"))
     ((< diff 0) (message "Second timestamp earlier than first!"))
     (t
      (let* ((h (-> diff (/ 3600) floor))
             (m (-> diff (- (* h 3600)) (/ 60) floor))
             (new-clock
              (concat
               org-clock-string " "
               (format-time-string (org-time-stamp-format t t) t1)
               "--"
               (format-time-string (org-time-stamp-format t t) (+ t1 diff))
               " => "
               (format "%2d:%02d" h m))))
        (save-excursion
          (org-clock-find-position nil)
          (insert-before-markers "\n")
	      (backward-char 1)
	      (org-indent-line)
          (insert new-clock)))))))

org agenda

These are executed directly from agenda views and affect their source org buffers. The trick is that all of them must somehow go back to the heading to which they allude, execute, then update the agenda view with whatever changes have been made.

(defmacro nd/org-agenda-cmd-wrapper (get-head &rest body)
  "Wraps commands in BODY in necessary code to allow commands to be
called from the agenda buffer. Particularly, this wrapper will
navigate to the original header, execute BODY, then update the agenda
buffer."
  '(org-agenda-check-no-diary)
  `(let* ((hdmarker (or (org-get-at-bol 'org-hd-marker)
			           (org-agenda-error)))
	     (buffer (marker-buffer hdmarker))
	     (pos (marker-position hdmarker))
	     (inhibit-read-only t)
	     newhead)
    (org-with-remote-undo buffer
	  (with-current-buffer buffer
	    (widen)
	    (goto-char pos)
	    (org-show-context 'agenda)
        ,@body
	    (when ,get-head (setq newhead (org-get-heading))))
	  (if ,get-head
          (org-agenda-change-all-lines newhead hdmarker)
        (org-agenda-redo))
	  (beginning-of-line 1))))
  
(defun nd/org-agenda-toggle-checkbox ()
  "Toggle checkboxes in org agenda view using `org-toggle-checkbox'."
  (interactive)
  (nd/org-agenda-cmd-wrapper
   t
   (call-interactively #'org-toggle-checkbox)))

(defun nd/org-agenda-clone-subtree-with-time-shift ()
  "Apply `nd/org-clone-subtree-with-time-shift' to an agenda entry.
It will clone the last entry in the selected subtree."
  (interactive)
  (nd/org-agenda-cmd-wrapper
   nil
   (call-interactively #'nd/org-clone-subtree-with-time-shift-toplevel)))

(defun nd/org-agenda-delete-subtree ()
  "Apply `nd/org-delete-subtree' to an agenda entry."
  (interactive)
  (nd/org-agenda-cmd-wrapper
   nil
   (call-interactively #'nd/org-delete-subtree)))

(defun nd/org-agenda-clock-range ()
  "Apply `nd/org-clock-range' to an agenda entry"
  (interactive)
  (nd/org-agenda-cmd-wrapper
   nil
   (call-interactively #'nd/org-clock-range)))

calfw

This is a nifty calendar…sometimes way faster than the agenda buffer for looking at long term things.

(use-package calfw
  :ensure t
  :config
  (setq cfw:fchar-junction ?╋
        cfw:fchar-vertical-line ?┃
        cfw:fchar-horizontal-line ?━
        cfw:fchar-left-junction ?┣
        cfw:fchar-right-junction ?┫
        cfw:fchar-top-junction ?┯
        cfw:fchar-top-left-corner ?┏
        cfw:fchar-top-right-corner ?┓))

(use-package calfw-org
  :ensure t
  :after calfw
  :config
  (setq cfw:org-agenda-schedule-args
        '(:deadline :timestamp)))

window splitting

Org mode is great and all, but the windows never show up in the right place. The solutions here are simple, but have the downside that the window sizing must be changed when tags/capture templates/todo items are changed. This is because the buffer size is not known at window creation time and I didn't feel like making a function to predict it

todo selection

I only need a teeny tiny window below my current window for todo selection

(defun nd/org-todo-position (buffer alist)
  (let ((win (car (cl-delete-if-not
                   (lambda (window)
                     (with-current-buffer (window-buffer window)
                       (memq major-mode
                             '(org-mode org-agenda-mode))))
                   (window-list)))))
    (when win
      (let ((new (split-window win -4 'below)))
        (set-window-buffer new buffer)
        new))))

(defun nd/org-todo-window-advice (orig-fn)
  "Advice to fix window placement in `org-fast-todo-selection'."
  (let  ((override '("\\*Org todo\\*" nd/org-todo-position)))
    (add-to-list 'display-buffer-alist override)
    (nd/with-advice
        ((#'org-switch-to-buffer-other-window :override #'pop-to-buffer))
      (unwind-protect (funcall orig-fn)
        (setq display-buffer-alist
              (delete override display-buffer-alist))))))

(advice-add #'org-fast-todo-selection :around #'nd/org-todo-window-advice)

tag selection

By default, the tag selection window obliterates all but the current window…how disorienting :/

(defun nd/org-tag-window-advice (orig-fn current inherited table &optional todo-table)
  "Advice to fix window placement in `org-fast-tags-selection'."
  (nd/with-advice
      ((#'delete-other-windows :override #'ignore)
       ;; pretty sure I just got lucky here...
       (#'split-window-vertically :override #'(lambda (&optional size)
                                                (split-window-below (or size -9)))))
    (unwind-protect (funcall orig-fn current inherited table todo-table))))

(advice-add #'org-fast-tag-selection :around #'nd/org-tag-window-advice)

capture

Capture should show up in the bottom of any currently active buffer

(defun nd/org-capture-position (buffer alist)
  (let ((new (split-window (get-buffer-window) -14 'below)))
    (set-window-buffer new buffer)
    new))

(defun nd/org-capture-window-advice (orig-fn table title &optional prompt specials)
  "Advice to fix window placement in `org-capture-select-template'."
  (let  ((override '("\\*Org Select\\*" nd/org-capture-position)))
    (add-to-list 'display-buffer-alist override)
    (nd/with-advice
        ((#'org-switch-to-buffer-other-window :override #'pop-to-buffer))
      (unwind-protect (funcall orig-fn table title prompt specials)
        (setq display-buffer-alist
              (delete override display-buffer-alist))))))

(advice-add #'org-mks :around #'nd/org-capture-window-advice)

exporting

latex to pdf command

Use latexmk instead of pdflatex as it is more flexible and doesn't require running the process zillion times just to make a bibliography work. Importantly, add support here for BibTeX as well as the custom output directory (see below).

(setq org-latex-pdf-process (list "latexmk -output-directory=%o -shell-escape -bibtex -f -pdf %f"))

custom output directory

By default org export files to the same location as the buffer. This is insanity and clutters my org directory with .tex and friends. Force org to export to a separate location.

(defvar nd/org-export-publishing-directory
  (expand-file-name "org-exports" (getenv "XDG_CACHE_HOME"))
  "The target directory to for all org exports.")

(defun nd/org-export-output-file-name (orig-fun extension &optional subtreep pub-dir)
  "Change the target export directory for org exports."
  (unless pub-dir
    (setq pub-dir nd/org-export-publishing-directory)
    (unless (file-directory-p pub-dir)
      (make-directory pub-dir)))
  (apply orig-fun extension subtreep pub-dir nil))

(advice-add 'org-export-output-file-name :around #'nd/org-export-output-file-name)

html5

The default is XHTML for some reason (which few use and makes certain barbaric word processors complain). Use the much-superior html5.

(setq org-html-doctype "html5")

gantt charts

This is custom, non-MELPA package, so it must be loaded manually. See here for guide.

(add-to-list 'load-path "~/.emacs.d/untracked/org-gantt/")
(require 'org-gantt)

It is also useful to define a block template for gantt chart creation

(add-to-list 'org-structure-template-alist
             '("og" "#+BEGIN: org-gantt-chart\n?\n#+END"))

gtd implementation

overview

This section is meant to be a big-picture overview of how GTD works in this setup. For specifics, see each section following this for further explanation and code. I should also say that most of the ideas for the code came from Bernt Hansen's very detailed guide.

workflow

GTD as described in its original form is divided into five steps as explained further below. Here I attempt to explain how I implement each of these into org-mode.

collect

The whole point of GTD is to get stuff out of one's head, and this is purpose of the collect step. Basically if a thought or task pops in my head or interrupts me, I record it somewhere. These thoughts can happen any time and anywhere, so it is important to keep them out of consciousness so that I can concentrate on whatever I am doing.

When org-mode is in front of me, I use org-capture (see below for org-capture-templates). The "things" that could be collected include anything from random ideas, things I remember to do, appointments I need to attend, etc. I also capture emails with mu4e (which links to org-mode through org-protocol). Everythign collected with org-capture gets sent to a dedicated file where I deal with it later (see process step).

When org-mode is not in front of me, I record my thoughts in the Orgzly app on my android. It doesn't really sync so I transfer everything manually.

process

Collecting only records things; it doesn't make decisions. The point of the process step is to decide if the task/note is worth my time and when. This involves several key questions.

The first question to ask is if the task is actionable. If yes, it gets moved to a project file or a general task file. If not, I ask it can either be moved to the "incubator" (a place for things I might do), be moved any number of reference files (for storing inportant information), or flat-out deleted if I think it is stupid or no longer relevant.

In org-mode these decisions are made and recorded by moving headings between files with org-refile. To facilitate this process I have an agenda view to filter out captured tasks. From there it is easy to refile to wherever the headers need to go.

This step happens daily along with organize below.

organize

The organize step is basically the second half of the process step (I honestly think of these as a single task because that's how they are implemented in org-mode, but the original GTD workflow describes them seperately).

After refiling with org-refile, the next step is to add any remaining meta information to each task, which is later used to decide what to do and when. This information includes context, effort, delegation, and timestamps. In the case of projects this also includes choosing a NEXT tasks if one hasn't been chosen already.

Delegation (assingning something to someone else) is simple and is represented by a simple property which is filled with the initials of the person doing the work. It filter and view this with org-columns and org-agenda-columns.

When tasks don't have a specific date, GTD outlines a four-criteria model for deciding what to do: context, required time, available energy, and priority. Context describes required locations and resources for tasks, and I represent them with tags (see org-tags-alist). Required time is represented by the Effort property (see org-default-properties below). Available energy is subjective and not represented in org-mode. Priority is again represented with tags, here chosen from one of seven "life categories."

In assigning timestamps, org-mode offers several possibilities out of the box. Putting a plain active timestamp denotes an appointment (something at which I need to show up). A scheduled timestamp denotes a task that I want to work on starting at a certain time. A deadline denotes a task that must be finished by a certain time. I try to only use these for "hard" times as anything "soft" risks me not fulfilling to the timestamp and hence diminishing the value of timestamps in general.

I have three main agenda views for handling this. The first is a daily view that shows the tasks needed for today, including anything with a timestamp. The second has all tasks that are not timestamps (eg things that can be done at any time). The third is a project view that shows the top level headings for collections of tasks (this is where I find any projects that need a NEXT task).

The organize step may seem like it requires alot of work but luckily org-mode allows enough automation that some of this meta information can be added in the collect and process phases. For instance, timestamps and tags can be added (forcibly) in org-capture depending on what template is used. Furthermore, the priority tag and some context tags are added when the task is refiled to its proper file or project; this happens via tag inheritance, defined at either the file level or a parent heading (for instance, a computer-related tasks may be filed under environmental/computer where environment has the _env tag and computer has the #laptop tag).

review

In order to keep the entire workflow moving smoothly, it is necessary to do a high-level review.

This happens weekly and involves several things.

• Scheduling important tasks and resolve conflicts. For this I use calfw (basically a calendar) to look at the next week and check if anything overlaps and move things around. I also "reload" repeater tasks using nd/org-clone-subtree-with-timeshift.
• Moving tasks to the archive as they are available. This keeps org-mode fast and uncluttered.
• Reviewing the incubator and moving tasks out that I actually decide to do.
• Reviewing reference material and moving it to appropriate tasks.
• Assessing projects based on their status (see below for the definition of "status"). Ideally all projects are "active," and if they are not I try to make them active by assigning NEXT.

I have specialized agenda views and commands for facilitating all of this.

execute

Execute involves doing the predefined work laid out in the previous four steps. Generally I work through two agenda views (in order). The first being all my tasks that need to get done in the day, and the second being all tasks with no specific timestamp.

Besides physically doing the tasks here, the other special thing in org-mode that I use is clocking. In addition to tracking time spent, it also encourages clean breaks between tasks (eg no multitasking).

file hierarchy and structure

All org files are kept in one place (see org-directory). This is futher subdivided into directories for project (as per terms and definitions, these are any tasks that involve at least on subtask) and reference files. At the top level are files for incubated tasks, captured tasks, and catchall general tasks (which also includes small projects that don't fit anywhere else).

In order to make sorting easier and minimize work during processing, the files are further subdivided using tags at the file level and heading level that will automatically categorize tasks when they are refiled to a certain location. For example, some project may be to create a computer program, so I would set #+FILETAGS: #laptop because every task in this project will require a laptop. See the tags section below for more information on tags.

repetition

This deserves special attention because it comprises a significant percentage of tasks I do (and likely everyone does). I personally never liked the org's repeated task functionality. It is way too temporally rigid to be useful to me, and offers very little flexibility in mutating a task as it moves forward. Habits (which I use) are a partial fix for the first problem but do not aleviate the mutability problem.

My (somewhat convoluted) solution was to use org-clone-subtree-with-time-shift, which creates an easy way to make repeated tasks from some template, but also allows modification. The only problem with the vanilla implementation is that it lacks automation and agenda-block awareness (they all get treated as regular tasks which I don't want). This is partially fixed with my own nd/org-clone-subtree-with-time-shift which automaticlly resets tasks which are cloned (eg clearing checkboxes and resetting todo state). The remainding problems I fixed by defining several properties to be applied to repeated groupings under a heading (see properties).

The first property is called PARENT_TYPE and has two values iterator and periodical. The first applies to repeated tasks and second which applies to timestamped headings such as appointments. These are mostly useful for agenda sorting, where I have views specifically for managing repeated tasks. The second property is TIME_SHIFT; nd/org-clone-subtree-with-time-shift is aware of this value and automatically shifts cloned tasks accordingly if available.

In practice, I use this for tasks like workouts, paying bills, maintenance, grocery shopping, work meetings, GTD reviews, etc. These are all almost consistent but may change slightly in their timing, action items, effort, context, etc. If any of these change, it is easy enough to modify one heading without disrupting the rest.

In an org tree these look like this:

 ***** clean room
:PROPERTIES:
:PARENT_TYPE: iterator
:TIME_SHIFT: +1m
:END:
 ****** DONE clean room [0/2]
CLOSED: [2018-11-21 Wed 22:13] SCHEDULED: <2018-10-29 Mon>
:PROPERTIES:
:Effort:   0:15
:END:
- [ ] vacuum
- [ ] throw away trash
 ****** TODO clean room [0/2]
SCHEDULED: <2018-11-29 Thu>
:PROPERTIES:
:Effort:   0:30
:END:
- [ ] vacuum room
- [ ] throw away trash

block agenda views

The heart of this implementation is an army of block agenda views (basically filters on the underlying org trees that bring whatever I need into focus). These have become tailored enough to my workflow that I don't even use the built-in views anymore (I also have not found an "easy" way to turn these off). Besides projects, these agenda views are primarily driven using skip functions.

projects

When it comes to the agenda view, I never liked how org-mode by default handled "projects" (see how that is defined in "terms and definitions"). It mostly falls short because of the number of todo keywords I insist on using. The solution I implemented was to used "statuscodes" (which are just keywords in lisp) to define higher-level descriptions based on the keyword content of a project. For example a "stuck" project (with statuscode :stuck) is a project with only TODO keywords. Adding a NEXT status turns the statuscode to :active. Likewise WAIT makes :waiting. This seems straightforward, except that NEXT trumps WAIT, WAIT trumps HOLD, etc. Furthermore, there are errors I wish to catch to ensure subtrees get efficiently cleaned out, such as a project heading with DONE that still has a TODO underneath.

I used to take care of this problem with lots of skip functions, but it turned out to be unmaintainable and offered poor performance (eg if I wanted a block agenda for N statuscodes, I needed to scan the entire org tree N times). A far easier way to implement this was to embed the statuscodes in text properties in each agenda line, which could then be sorted and the prefix string formatted with the status code for identification in the block agenda view. Since this only requires one block, it only requires one scan, and is very fast.

repeaters

Similarly to projects, repeaters (eg iterators and periodicals) are assessed via a statuscode (after all they are a group of headings and thus depending on the evaluation of todo keywoards and timestamps in aggregate). These prove much simpler than projects as essentially all I need are codes for uninitialized (there is nothing in the repeater), empty (all subheadings are in the past and therefore irrelevant), and active (there are some subtasks in the future).

terms and definitions

These conventions are used throughout to be precise when naming functions/variables and describing their effects

headings

• heading: the topmost part after the bullet in an org outline. Org-mode cannot seem to make up it's mind in calling it a header, heading, or headline, so I picked heading
• todoitem: any heading with a todo keyword

• task: a todoitem with no todoitem children

  • atomic: further specifies that the task is not part of a project

• project: a todoitem with that has todoitem children or other projects

  • status(code): a keyword used to describe the overall status of a project. See skip functions in the block agenda section for their implementation.

time

• stale: refers to timestamps that are in the past/present

  • archivable: further specifies that the timestamp is older than some cutoff that defines when tasks can be archived (usually 30 days)
• fresh: refers to timestamps that are in the future

todo states

sequences

These keywords are used universally for all org files (see below on quick explanation for each, they are all quite straightforward). Note that projects have a more specific meaning for these keywords in defining project status (see the library of agenda function). Also, it looks way better in the agenda buffer when they are all the same number of chars.

In terms of logging, I like to record the time of each change upon leaving any state, and I like recording information in notes when waiting, holding, or canceling (as these usually have some external trigger or barrier that should be specified).

(setq org-todo-keywords
      '((sequence
         ;; default undone state
         "TODO(t/!)"

         ;; undone but available to do now (projects only)
         "NEXT(n/!)" "|"

         ;; done and complete
         "DONE(d/!)")

        (sequence
         ;; undone and waiting on some external dependency
         "WAIT(w@/!)"
         
         ;; undone but signifies tasks on which I don't wish to focus at the moment
         "HOLD(h@/!)" "|"

         ;; done but not complete
         "CANC(c@/!)")))

colors

Aesthetically, I like all my keywords to have bold colors.

(setq   org-todo-keyword-faces
      '(("TODO" :foreground "light coral" :weight bold)
        ("NEXT" :foreground "khaki" :weight bold)
        ("DONE" :foreground "light green" :weight bold)
        ("WAIT" :foreground "orange" :weight bold)
        ("HOLD" :foreground "violet" :weight bold)
        ("CANC" :foreground "deep sky blue" :weight bold)))

tags

alist

I use tags for agenda filtering (primarily for GTD contexts, see below). Each tag here starts with a symbol to define its group (note, only the special chars "_", "@", "#", and "%" seem to be allowed; anything else will do weird things in the hotkey prompt). Some groups are mutually exclusive. By convention, any tag not part of these groups is ALLCAPS (not very common) and set at the file level.

(setq org-tag-alist
      ;; (@) gtd location context
      '((:startgroup)
        ("@errand" . ?e)
        ("@home" . ?h)
        ("@work" . ?w)
        ("@travel" . ?r)
        (:endgroup)
        
        ;; (#) gtd resource context 
        ("#laptop" . ?l)
        ("#tcult" . ?t)
        ("#phone" . ?p)
        
        ;; (%) misc tags 
        ;; denotes reference information
        ("%note" . ?n)
        
        ;; incubator
        ("%inc" . ?i)
        
        ;; denotes tasks that need further subdivision to turn into true project
        ("%subdiv" . ?s)
        
        ;; catchall to mark important headings, usually for meetings
        ("%flag" . ?f)
        
        ;; (_) life categories, used for gtd priorities
        (:startgroup)
        ("_env" . ?E) ;; environmental
        ("_fin" . ?F) ;; financial
        ("_int" . ?I) ;; intellectual
        ("_met" . ?M) ;; metaphysical
        ("_phy" . ?H) ;; physical
        ("_pro" . ?P) ;; professional
        ("_rec" . ?R) ;; recreational
        ("_soc" . ?S) ;; social
        (:endgroup)))

colors

Each group also has its own color, defined by its prefix symbol.

(defun nd/add-tag-face (fg-name prefix)
  "Adds list of cons cells to org-tag-faces with foreground set to fg-name.
  Start and end specify the positions in org-tag-alist which define the tags
  to which the faces are applied"
  (dolist (tag (nd/filter-list-prefix prefix (mapcar #'car org-tag-alist)))
    (push `(,tag . (:foreground ,fg-name)) org-tag-faces)))

(setq org-tag-faces '())

(nd/add-tag-face "PaleGreen" "@")
(nd/add-tag-face "SkyBlue" "#")
(nd/add-tag-face "PaleGoldenrod" "%")
(nd/add-tag-face "violet" "_")

properties

The built-in effort is used as the fourth and final homonymous GTD context (the other three being covered above using tags). It is further restricted with Effort_All to allow easier filtering in the agenda.

Also here are the properties for repeated tasks and a few others (see comments in code).

(mapc (lambda (i) (add-to-list 'org-default-properties i))
      ;; defines a repeater group
      '("PARENT_TYPE"
        ;; defines the time shift for repeater groups

        "TIME_SHIFT"
        ;; assigns another person/entity to a task (experimental)

        "DELEGATE"

        ;; defines a goal
        "GOAL"

        ;; date of header creation
        "CREATED"))

(setq org-global-properties
      '(("PARENT_TYPE_ALL" . "periodical iterator")
        ("Effort_ALL" . "0:05 0:15 0:30 1:00 1:30 2:00 3:00 4:00 5:00 6:00"))

      org-use-property-inheritance
      '("PARENT_TYPE" "TIME_SHIFT"))

capture

templates

As per Bernt's guide, capture is meant to be fast. The dispatcher is bound to F2 (see keybindings section) which allows access in just about every mode and brings a template up in two keystrokes.

(defun nd/org-timestamp-future (days)
  "Inserts an active org timestamp DAYS after the current time."
  (format-time-string (org-time-stamp-format nil)
                      (time-add (current-time) (days-to-time 1))))

(let ((capfile "~/Org/capture.org"))
  (setq org-capture-templates
        ;; regular TODO task
        `(("t" "todo" entry (file ,capfile)
           "* TODO %?\n")

          ;; for useful reference information that may be grouped with tasks
          ("n" "note" entry (file ,capfile)
           "* %?  :\\%note:\n%U\n")

          ;; for non-actionable events that happen at a certain time
          ("a" "appointment" entry (file ,capfile)
           "* %?\n%U\n%^t\n")

          ;; like appointment but multiple days
          ("s" "appointment-span" entry (file ,capfile)
           "* %?\n%U\n%^t--%^t\n")

          ;; task with a deadline
          ("d" "deadline" entry (file ,capfile)
           "* TODO %?\nDEADLINE: %^t\n")

          ;; for converting mu4e emails to tasks, defaults to next-day deadline
          ("e" "email" entry (file ,capfile)
           "* TODO Respond to %:fromname; Re: %:subject  :#laptop:\nDEADLINE: %(nd/org-timestamp-future 1)\n%a\n")

          ;; for interruptions that produce useful reference material
          ("m" "meeting" entry (file ,capfile)
           "* meeting with%?  :\\%note:\n%U\n")

          ;; for capturing web pages with web browser
          ("p" "org-protocol" entry (file ,capfile)
           "* %^{Title} :\\%note:\n%u\n#+BEGIN_QUOTE\n%i\n#+END_QUOTE"
           :immediate-finish t)

          ;; or capturing links with web browser
          ("L" "org-protocol link" entry (file ,capfile)
           "* %^{Title} :\\%note:\n[[%:link][%:description]]\n%U"
           :immediate-finish t))))

insert mode

To save one more keystroke (since I use evil mode), trigger insert mode upon opening capture template.

(add-hook 'org-capture-mode-hook (lambda () (evil-append 1)))

refile

Refile (like capture) should be fast, and I search all org file simultaneously using helm (setting org-outline-path-complete-in-steps to nil makes search happen for entire trees at once and not just the current level). Refiling is easiest to do from a block agenda view (see below) where headings can be moved in bulk.

(setq org-refile-targets '((nil :maxlevel . 9)
                           ("~/Org/reference/idea.org" :maxlevel . 9)
                           (org-agenda-files :maxlevel . 9))
      org-refile-use-outline-path t
      org-outline-path-complete-in-steps nil
      org-refile-allow-creating-parent-nodes 'confirm
      org-indirect-buffer-display 'current-window)

Prevent accidental refiling under tasks with done keywords

(setq org-refile-target-verify-function
      (lambda () (not (member (nth 2 (org-heading-components)) org-done-keywords))))

;; TODO this no work, although does work if var is global
;; redfining the targets works for now
(add-hook 'org-agenda-mode-hook
          (lambda ()
            (when (equal (buffer-name) "*Org Agenda(A)*")
              (setq-local org-refile-targets
                          '(("~/Org/journal/goals.org" :maxlevel . 9))))))
;;                           (lambda () (when (org-entry-get nil "GOAL") t))))))
;; (setq org-refile-targets '((nil :maxlevel . 9)
;;                            ("~/Org/reference/idea.org" :maxlevel . 9)
;;                            ("~/Org/journal/goals.org" :maxlevel . 9)
;;                            (org-agenda-files :maxlevel . 9))

clocking

general

Clocking is still new and experimental (I'm not a ninja like Bernt yet). I mostly use clocking now as a way to make clean breaks between tasks (eg to discourage "mixing" tasks which is a slippery multitasking slope). I bound F4 to org-clock-goto as an easy way to find my current/last clocked task in any mode (see keybindigs).

(setq org-clock-history-length 23
      org-clock-out-when-done t
      org-clock-persist t
      org-clock-report-include-clocking-task t)

modeline

The modeline is a nice place to indicate if something is clocked in or out. Unfortunately, sometimes is is so crowded that I can't see the text for the currently clocked task. Solution, use colors.

(defface nd/spaceline-highlight-clocked-face
  `((t (:background "chartreuse3"
        :foreground "#3E3D31"
        :inherit 'mode-line)))
  "Default highlight face for spaceline.")
  
(defun nd/spaceline-highlight-face-clocked ()
  "Set the spaceline highlight color depending on if the clock is running."
  (if (and (fboundp 'org-clocking-p) (org-clocking-p))
      'nd/spaceline-highlight-clocked-face
    'spaceline-highlight-face))

(setq spaceline-highlight-face-func 'nd/spaceline-highlight-face-clocked)

clustering

Org mode has no way of detecting if conflicts exist. It also has no way of alerting someone if they have overbooked their schedule

extraction filters

These control which types of headlines are processed by org-cluster

(defvar nd/org-cluster-filter-files t
  "Set to t if files should be filtered in org-cluster.
This option does nothing unless `nd/org-cluster-filtered-files' is 
also non-nil.")

(defconst nd/org-cluster-filtered-files
  '("incubator" "peripheral")
  "Files that should be excluded from org-cluster analysis.
These are pattern-matched so they do not need to be exact names
or paths.")

(defvar nd/org-cluster-filter-todo t
  "Set to t if todo keywords should be filtered in org-cluster.
This option does nothing unless `nd/org-cluster-filtered-todo' is 
also non-nil.")

(defconst nd/org-cluster-filtered-todo
  '("CANC" "DONE")
  "TODO keywords that should be filtered from org-cluster analysis.")

(defvar nd/org-cluster-filter-past t
  "Set to t to exclude files from before now in org-cluster analysis.")

(defvar nd/org-cluster-filter-habit nil
  "Set to t to exclude habits from org-cluster analysis.")

timestamp extraction and filtering

Conflicts and overloads begin with the same list to process, which is created using org-element-parse-buffer and a variety of filtering functions to extract relevent timestamps.

The main object that is passed around during extraction and processing is the timestamp-plist as described in nd/org-cluster-make-tsp below.

(defun nd/org-cluster-make-tsp (unixtime range offset fp hardness
                                      &optional type)
  "Construct a timestamp plist to be used in further processing.
UNIXTIME is the unixtime of the timestamp as an integer, RANGE is the 
duration of the timestamp (could be 0), OFFSET is the character offset
of the timestamp in the file represented with filepath FP, HARDNESS
is a boolean denoting if the timestamp is 'hard' (has minutes and
hours) or 'soft' (only a date). TYPE can be optionally supplied to
denote kinds of timestamps (only 'scheduled' for now)."
  (list :unixtime (round unixtime)
        :range (or range 0)
        :offset offset
        :type type
        :hardness hardness
        :filepath fp))
        
(defun nd/org-cluster-ts-hard-p (ts)
  "Return non-nil if the timestamp TS has hours/minutes."
  (org-element-property :hour-start ts))
  
(defun nd/org-cluster-parse-ts (ts hl fp)
  "Parse a timestamp TS belonging to headline HL and filepath FP.
TS is an object as described in the org-element API. Only active
or active-range types are considered. Returns a new timestamp-plist
for TS."
  (when ts
    (let* ((offset (org-element-property :begin hl))
           (hardness (nd/org-cluster-ts-hard-p ts))
           (split
            (lambda (ts &optional end)
              (--> ts
                   (org-timestamp-split-range it end)
                   (org-element-property :raw-value it)
                   (org-2ft it))))
           (start (funcall split ts)))
      (if (eq (org-element-property :type ts) 'active-range)
          (let ((range (--> ts (funcall split it t) (- it start))))
            (nd/org-cluster-make-tsp start range offset fp hardness))
        (nd/org-cluster-make-tsp start 0 offset fp hardness)))))
        
(defun nd/org-cluster-effort-seconds (effort-str)
  "Convert EFFORT-STR into an integer in seconds from HH:MM format."
  (let ((effort-str (string-trim effort-str)))
    (save-match-data
      (cond
       ((string-match "^\\([0-9]+\\):\\([0-6][0-9]\\)$" effort-str)
        (let ((hours (->> effort-str
                          (match-string 1)
                          string-to-number
                          (* 60))))
          (->> effort-str
               (match-string 2)
               string-to-number
               (+ hours)
               (* 60))))
       (t (error (format "Unknown effort: %s'" effort-str)))))))

(defun nd/org-cluster-extract (acc fun objs &rest args)
  "Run FUN on each of OBJS and put results into accumulator ACC.
FUN is a function that takes the accumulator as its first argument,
one member of OBJS as the second, and ARGS as the rest if supplied."
  (while objs
    (setq acc (apply fun acc (car objs) args)
          objs (cdr objs)))
  acc)

(defun nd/org-cluster-extract-hl-sched (acc hl fp)
  "Extract scheduled timestamp from headline HL in filepath FP.
Create a new timestamp-plist and add to accumulator ACC."
  (let* ((ts (org-element-property :scheduled hl))
         (unixtime (->> ts (org-element-property :raw-value) org-2ft))
         (range (-some->> hl
                          (org-element-property :EFFORT)
                          nd/org-cluster-effort-seconds))
         (hardness (nd/org-cluster-ts-hard-p ts))
         (offset (org-element-property :begin hl)))
    (if (= 0 unixtime) acc
      (-> unixtime
          (nd/org-cluster-make-tsp range offset fp hardness 'scheduled)
          (cons acc)))))

(defun nd/org-cluster-extract-hl-ts (acc hl fp)
  "Extract timestamps from headline HL in filepath FP.
All active timestamps that are not in drawers or the planning header 
are considered. Each timestamp is converted into a new timestamp-plist
and added to accumulator ACC."
  (--> hl
       (assoc 'section it)
       (org-element-contents it)
       (--remove
        (or (eq 'planning (org-element-type it))
            (eq 'property-drawer (org-element-type it))
            (eq 'drawer (org-element-type it)))
        it)
       (org-element-map it 'timestamp #'identity)
       (--filter
        (or (eq 'active (org-element-property :type it))
            (eq 'active-range (org-element-property :type it)))
        it)
       (--map (nd/org-cluster-parse-ts it hl fp) it)
       (append acc it)))

(defun nd/org-cluster-extract-hl (acc hl fp)
  "Extract timestamps from headline HL in filepath FP and store in ACC."
  (-> acc
      (nd/org-cluster-extract-hl-sched hl fp)
      (nd/org-cluster-extract-hl-ts hl fp)))

(defun nd/org-cluster-filter-todo (hls)
  "Filter certain TODO keywords from headline list HLS."
  (if (not nd/org-cluster-filter-todo) hls
    (--remove
     (member (org-element-property :todo-keyword it)
                      nd/org-cluster-filtered-todo)
     hls)))

(defun nd/org-cluster-filter-files (fps)
  "Filter certain file names from files list FPS."
  (if (not nd/org-cluster-filter-files) fps
    (--remove
     (-find (lambda (s) (string-match-p s it)) nd/org-cluster-filtered-files)
     fps)))

(defun nd/org-cluster-filter-past (tsps)
  "Filter out timestamp-plists in list TSPS if they start in the past."
  (if (not nd/org-cluster-filter-past) tsps
    (let ((ft (float-time)))
      (--remove (< (plist-get it :unixtime) ft) tsps))))

(defun nd/org-cluster-filter-habit (hls)
  "Filter headlines from headline list HLS that are habits."
  (if (not nd/org-cluster-filter-habit) hls
    (--remove (org-element-property :STYLE it) hls)))

(defun nd/org-cluster-extract-file (acc fp)
  "Extract timestamps from filepath FP and add to accumulator ACC."
  (-->
   fp
   (find-file-noselect it t)
   (with-current-buffer it (org-element-parse-buffer))
   (org-element-map it 'headline #'identity)
   (nd/org-cluster-filter-todo it)
   (nd/org-cluster-filter-habit it)
   (nd/org-cluster-extract acc #'nd/org-cluster-extract-hl it fp)))
  
(defun nd/org-cluster-get-unprocessed ()
  "Return a list of timestamp-plists with desired filter settings."
  (->>
   ;; (list "~/Org/reference/testconflict.org")
   (org-agenda-files)
   nd/org-cluster-filter-files
   (nd/org-cluster-extract nil #'nd/org-cluster-extract-file)   
   nd/org-cluster-filter-past))

conflict detection

This algorithm builds a list of pairs, with each pair being a two tasks that conflict and should be O(n) (best case/no conflicts) to O(n^2) (worst case/everything conflicts).

Steps for this:

1. make a list of all entries containing timestamps (active and scheduled)
2. sort timestamp list
3. Walk through list and compare entries immediately after (sorting ensures that entries can be skipped once one non-conflict is found). If conflicts are found push the pair to new list.

(defun nd/org-cluster-conflicting-p (tsp-a tsp-b)
  "Return t if timestamps TS-A and TS-B conflict."
  ;; assume that ts-a starts before ts-b
  (let* ((start-a (plist-get tsp-a :unixtime))
         (start-b (plist-get tsp-b :unixtime))
         (end-a (-> tsp-a (plist-get :range) (+ start-a))))
    (or (= start-a start-b) (< start-b end-a))))

(defun nd/org-cluster-find-conflict (tsp tsps conlist)
  "Test if timestamp-plist TSP conflicts with any in TSPS.
If found, anything in TSPS is cons'd with TSP and added to CONLIST
as a pair. New CONLIST is returned."
  (->> tsps
       (--take-while (nd/org-cluster-conflicting-p tsp it))
       (--map (cons tsp it))
       (append conlist)))
  
(defun nd/org-cluster-build-conlist (tsps)
  "Build a list of conflict pairs from timestamp-plist TSPS."
  (let ((conlist))
    (while (< 1 (length tsps))
      (setq conlist (nd/org-cluster-find-conflict (car tsps)
                                                  (cdr tsps)
                                                  conlist)
            tsps (cdr tsps)))
    conlist))

(defun nd/org-cluster-get-conflicts ()
  "Return a list of cons cells representing conflict pairs.
Each member in the cons cell is a timestamp-plist."
  (->>
   (nd/org-cluster-get-unprocessed)
   (--filter (plist-get it :hardness))
   (--sort (< (plist-get it :unixtime) (plist-get other :unixtime)))
   nd/org-cluster-build-conlist))

overload detection

Overloads are defined as days that have more than 24 hours worth of scheduled material. The algorithm is O(n) as it is basically just a bunch of filtering functions that walk through the list.

Steps for the algorithm:

1. filter only ranged entries (unranged entries have zero time)
2. maybe split timestamps if they span multiple days
3. sort from earliest to latest starting time
4. sum the range of timestamps in each day, keeping those that exceed 24 hours

(defun nd/org-cluster-split-day-bounds (tsps)
  "Split timestamp-plists in TSPS via daily boundaries.
Returns a new timestamp-plist with equal or greater length depending
on how many members needed splitting."
  (letrec
      ((new
        (lambda (start end tsp)
          (nd/org-cluster-make-tsp start
                                      (- end start)
                                      (plist-get tsp :offset)
                                      (plist-get tsp :filepath)
                                      (plist-get tsp :hardness)
                                      (plist-get tsp :type))))
       ;; need to temporarily offset the epoch time so day
       ;; boundaries line up in local time
       (split
        (lambda (start end tsp)
          (let* ((tzs-a (-> start current-time-zone car))
                 (tzs-b (-> end current-time-zone car))
                 (start* (-> end (+ tzs-b) (ceiling 86400) 1- (* 86400) (- tzs-b))))
            (if (> start* (-> start (+ tzs-a) (floor 86400) (* 86400) (- tzs-a)))
                (cons (funcall new start* end tsp)
                      (funcall split start start* tsp))
              (list (funcall new start end tsp))))))
       (split-maybe
        (lambda (tsp)
          (let* ((start (plist-get tsp :unixtime))
                 (end (+ start (plist-get tsp :range)))
                 (tzs (-> start current-time-zone car)))
            (if (< (-> start (+ tzs) (ceiling 86400)) end)
                (funcall split start end tsp)
              tsp)))))
    (--mapcat (funcall split-maybe it) tsps)))

(defun nd/org-cluster-daily-split (tsps)
  "Group timestamp-plist TSPS into sublists for each day."
  (letrec ((tz-shift (lambda (tsp) (-> tsp current-time-zone car (+ tsp)))))
    (->>
     tsps
     (--partition-by (--> it
                          (plist-get it :unixtime)
                          (funcall tz-shift it)
                          (floor it 86400))))))
                          
(defun nd/org-cluster-overloaded-p (tsps)
  "Return t if total time of timestamp-plists in TSPS exceeds 24 hours.
It is assumed the TSPS represents tasks and appointments within one
day."
  (letrec ((ts2diff
            (lambda (tsp)
              (let ((start (plist-get tsp :unixtime)))
                (- (-> tsp (plist-get :range) (+ start)) start)))))
    (->> tsps (--map (funcall ts2diff it)) -sum (<= 86400))))
                          
(defun nd/org-cluster-get-overloads ()
  "Return list of lists of timestamp-plists grouped by day.
Anything present represents all the tasks in a single day if that day
is overloaded. If a day is not overloaded there will be nothing for it
in the returned list."
  (->>
   (nd/org-cluster-get-unprocessed)
   (--filter (< 0 (plist-get it :range)))
   nd/org-cluster-split-day-bounds
   (--sort (< (plist-get it :unixtime) (plist-get other :unixtime)))
   nd/org-cluster-daily-split
   (--filter (nd/org-cluster-overloaded-p it))))

frontend

I could just fetch the org headings and throw them into a new buffer. But that's boring, and quite limiting. I basically want all the perks of an agenda buffer…tab-follow, the nice parent display at the bottom, time adjust hotkeys, etc. So the obvious and hacky solution is to throw together a quick-n-dirty agenda buffer.

(defun nd/org-cluster-headline-text (ts-entry)
  "Return string with text properties representing the org header for
MARKER for use in the conflict agenda view."
  (let* ((offset (plist-get ts-entry :offset))
         (ts-marker (--> ts-entry
                         (plist-get it :filepath)
                         (find-file-noselect it)
                         (with-current-buffer it
                           (copy-marker offset))))
         (props (list
                 'face nil
		         'done-face 'org-agenda-done
		         'org-not-done-regexp org-not-done-regexp
		         'org-todo-regexp org-todo-regexp
		         'org-complex-heading-regexp org-complex-heading-regexp
		         'mouse-face 'highlight))
	     marker priority category level tags todo-state
	     ts-date ts-date-type ts-date-pair
	     txt beg end inherited-tags todo-state-end-pos)

    (with-current-buffer (marker-buffer ts-marker) 
      (save-excursion
	    (goto-char ts-marker)

	    (setq marker (org-agenda-new-marker (point))
	          category (org-get-category)
	          ts-date-pair (org-agenda-entry-get-agenda-timestamp (point))
	          ts-date (car ts-date-pair)
	          ts-date-type (cdr ts-date-pair)
	          txt (org-get-heading t)
	          inherited-tags
	          (or (eq org-agenda-show-inherited-tags 'always)
		          (and (listp org-agenda-show-inherited-tags)
		               (memq 'todo org-agenda-show-inherited-tags))
		          (and (eq org-agenda-show-inherited-tags t)
		               (or (eq org-agenda-use-tag-inheritance t)
			               (memq 'todo org-agenda-use-tag-inheritance))))
	          tags (org-get-tags-at nil (not inherited-tags))
	          level (make-string (org-reduced-level (org-outline-level)) ? )
	          txt (org-agenda-format-item "" txt level category tags t)
	          priority (1+ (org-get-priority txt)))
        
	    (org-add-props txt props
	      'org-marker marker 'org-hd-marker marker
	      'priority priority
	      'level level
	      'ts-date ts-date
	      'type "timestamp")))))

(defun nd/org-cluster-ts-fmt (ts)
  (let ((fmt "[%Y-%m-%d]"))
    (--> ts (plist-get it :unixtime) (format-time-string fmt it))))

;; TODO...waaaaay too wet (not DRY)
(defun nd/org-cluster-show-conflicts (&optional arg)
  (interactive "P")

  (if org-agenda-overriding-arguments
      (setq arg org-agenda-overriding-arguments))

  (if (and (stringp arg) (not (string-match "\\S-" arg))) (setq arg nil))

  (let* ((today (org-today))
	     (date (calendar-gregorian-from-absolute today))
	     (completion-ignore-case t)
         (org-agenda-prefix-format '((agenda . "  %-12:c %-5:e ")))
	     rtn rtnall files file pos)

    (catch 'exit
      (when org-agenda-sticky (setq org-agenda-buffer-name "*Org Conflicts*"))

      (org-agenda-prepare)
      ;; (org-compile-prefix-format 'todo)
      (org-compile-prefix-format 'agenda)
      ;; (org-set-sorting-strategy 'todo)

      (setq org-agenda-redo-command '(nd/org-cluster-show-conflicts))

	  (insert "Conflicting Headings: \n")
	  (add-text-properties (point-min) (1- (point))
			               (list 'face 'org-agenda-structure
				                 'short-heading "Conflicts"))
	  (org-agenda-mark-header-line (point-min))

      (-some->
       (nd/org-cluster-get-conflicts)
       (--each
           (insert (concat
                    "At " (nd/org-cluster-ts-fmt (car it)) "\n"
                    (nd/org-cluster-headline-text (car it)) "\n"
                    (nd/org-cluster-headline-text (cdr it)) "\n"
                    "\n"))))

      ;; clean up and finalize
      (goto-char (point-min))
      (or org-agenda-multi (org-agenda-fit-window-to-buffer))
      (add-text-properties
       (point-min) (point-max)
	   `(org-agenda-type agenda
					     org-last-args ,arg
					     org-redo-cmd ,org-agenda-redo-command
					     org-series-cmd ,org-cmd))
      (org-agenda-finalize)
      (setq buffer-read-only t))))

(defun nd/org-cluster-show-overloads (&optional arg)
  (interactive "P")

  (if org-agenda-overriding-arguments
      (setq arg org-agenda-overriding-arguments))

  (if (and (stringp arg) (not (string-match "\\S-" arg))) (setq arg nil))

  (let* ((today (org-today))
	     (date (calendar-gregorian-from-absolute today))
	     (completion-ignore-case t)
         (org-agenda-prefix-format '((agenda . "  %-12:c %-5:e ")))
	     rtn rtnall files file pos)

    (catch 'exit
      (when org-agenda-sticky (setq org-agenda-buffer-name "*Org Overloads*"))

      (org-agenda-prepare)
      ;; (org-compile-prefix-format 'todo)
      (org-compile-prefix-format 'agenda)
      ;; (org-set-sorting-strategy 'todo)

      (setq org-agenda-redo-command '(nd/org-cluster-show-overloads))

	  (insert "Overloaded Days: \n")
	  (add-text-properties (point-min) (1- (point))
			               (list 'face 'org-agenda-structure
				                 'short-heading "Overloads"))
	  (org-agenda-mark-header-line (point-min))

      (-some->
       (nd/org-cluster-get-overloads)
       (--each
           (insert (concat
                    "On " (nd/org-cluster-ts-fmt (car it)) "\n"
                    (mapconcat #'nd/org-cluster-headline-text it "\n")
                    "\n"))))

      ;; clean up and finalize
      (goto-char (point-min))
      (or org-agenda-multi (org-agenda-fit-window-to-buffer))
      (add-text-properties
       (point-min) (point-max)
	   `(org-agenda-type agenda
					     org-last-args ,arg
					     org-redo-cmd ,org-agenda-redo-command
					     org-series-cmd ,org-cmd))
      (org-agenda-finalize)
      (setq buffer-read-only t))))

agenda

targets

The agenda files are limited to as few as possible to keep scanning and startup reasonably fast.

(setq org-agenda-files '("~/Org"
                        "~/Org/projects"
                        "~/Org/reference/peripheral.org"))

appearence

sticky agendas

I personally like having sticky agendas by default so I can use multiple windows

(setq org-agenda-sticky t)

tag alignment

(setq org-agenda-tags-column 'auto)

prefix format

This controls what each line on the block agenda looks like. This is reformated to include effort and remove icons.

(setq org-agenda-prefix-format
      '((agenda . "  %-12:c %-5:e %?-12t% s")
        (todo . "  %-12:c")
        (tags . "  %-12:c %-5:e ")
        (search . "  %-12:c")))

modeline

Hide the various modules that may be present

(defun nd/org-agenda-trim-modeline (orig-fn &rest args)
  "Advice to remove extra information from agenda modeline name."
  (let ((org-agenda-include-diary nil)
        (org-agenda-include-deadlines nil)
        (org-agenda-use-time-grid nil)
        (org-habit-show-habits nil))
    (apply orig-fn args)))

(advice-add #'org-agenda-set-mode-name :around #'nd/org-agenda-trim-modeline)

misc

These are just some options to enable/disable some aesthetic things.

(setq org-agenda-dim-blocked-tasks nil
      org-agenda-compact-blocks t
      org-agenda-window-setup 'current-window
      org-agenda-start-on-weekday 0
      org-agenda-span 'day
      org-agenda-current-time-string "### -- NOW -- ###")

Based on my screen size and usage patterns, this seems to be a good value to enable the maximum habit history to be shown without compromising aesthetics.

(setq org-habit-graph-column 50)

interactive filters

Rather than define infinite views for different tasks (I already have plenty of views) I use filtering to sort through the noise. Some of the built-in filters don't cut it, so I made a few of my own.

custom filtering functions

Some custom filters that are applied to the agenda view. Note that some of these use alternative filter types that are implemented via advising functions (see below).

(defun nd/org-agenda-filter-non-context ()
  "Filter all tasks with context tags."
  (interactive)
  (let* ((tags-list (mapcar #'car org-tag-alist))
         (context-tags (append
                        (nd/filter-list-prefix "@" tags-list)
                        (nd/filter-list-prefix "#" tags-list))))
    (setq org-agenda-tag-filter
          (mapcar (lambda (tag) (concat "-" tag)) context-tags))
    (org-agenda-filter-apply org-agenda-tag-filter 'tag)))

(defun nd/org-agenda-filter-non-peripheral ()
  "Filter all tasks that don't have peripheral tags."
  (interactive)
  (let* ((peripheral-tags '("PERIPHERAL")))
    (setq org-agenda-tag-filter
          (mapcar (lambda (tag) (concat "-" tag)) peripheral-tags))
    (org-agenda-filter-apply org-agenda-tag-filter 'tag)))
    
(defun nd/org-agenda-filter-non-effort ()
  "Filter agenda by non-effort tasks."
  (interactive)
  (setq org-agenda-hasprop-filter '("-Effort"))
  (org-agenda-filter-apply org-agenda-hasprop-filter 'hasprop))

(defun nd/org-agenda-filter-delegate ()
  "Filter agenda by tasks with an external delegate."
  (interactive)
  (setq org-agenda-hasprop-filter '("+DELEGATE"))
  (org-agenda-filter-apply org-agenda-hasprop-filter 'hasprop))

filter advice

In order to implement the hasprop filter, the functions org-agenda-filter-make-matcher and org-agenda-filter-remove-all need to be advised in order to add the functionality for the hasprop filter type.

As it is, this allows any filter using hasprop to be applied and removed using the standard org-agenda-filter-apply function with the org-agenda-hasprop-filter variable (obviously these can all be extended to different filter types). Note this does not give a shiny indicator at the bottom of spaceline like the built-in filter does…oh well.

;; initialize new filters
(defvar org-agenda-hasprop-filter nil)

(defun nd/org-agenda-filter-make-matcher-prop
    (filter type &rest args)
  "Return matching matcher form for FILTER and TYPE where TYPE is not
in the regular `org-agenda-filter-make-matcher' function. This is
intended to be uses as :before-until advice and will return nil if
the type is not valid (which is currently 'prop')"
  (let (f f1)
    ;; has property
    (cond
     ((eq type 'hasprop)
      (dolist (x filter)
        (push (nd/org-agenda-filter-make-matcher-hasprop-exp x) f))))
    (if f (cons 'and (nreverse f)))))

(defun nd/org-agenda-filter-make-matcher-hasprop-exp (h)
 "Returns form to test the presence or absence of properties H.
H is a string like +prop or -prop"
 (let (op)
   (let* ((op (string-to-char h))
          (h (substring h 1))
          (f `(save-excursion
                (let ((m (org-get-at-bol 'org-hd-marker)))
                  (with-current-buffer
                      (marker-buffer m)
                    (goto-char m)
                    (org-entry-get nil ,h))))))
     (if (eq op ?-) (list 'not f) f))))

(defun nd/org-agenda-filter-show-all-hasprop nil
  (org-agenda-remove-filter 'hasprop))

(advice-add #'org-agenda-filter-make-matcher :before-until
            #'nd/org-agenda-filter-make-matcher-prop)

(advice-add #'org-agenda-filter-remove-all :before
            (lambda () (when org-agenda-hasprop-filter
                    (nd/org-agenda-filter-show-all-hasprop))))

bulk actions

These add to the existing bulk actions in the agenda view.

(setq org-agenda-bulk-custom-functions
      '((?D nd/org-agenda-delete-subtree)))

holidays and birthdays

If I don't include this, I actually forget about major holidays.

(setq holiday-bahai-holidays nil
      holiday-hebrew-holidays nil
      holiday-oriental-holidays nil
      holiday-islamic-holidays nil)

(setq calendar-holidays (append holiday-general-holidays
                                holiday-christian-holidays))

block agenda library

These are functions and variables exclusively for agenda block manipulation within the context of org-custom-agenda-commands.

constants

(defconst nd/iter-future-time (* 7 24 60 60)
  "Iterators must have at least one task greater into the future to be active.")
  
(defconst nd/archive-delay-days 30
  "The number of days to wait before tasks are considered archivable.")

(defconst nd/inert-delay-days 90
  "The number of days to wait before tasks are considered inert.")
  
;; TODO ;unscheduled should trump all
(defconst nd/iter-statuscodes '(:uninit :empt :actv :project-error :unscheduled)
  "Iterators can have these statuscodes.")
 
(defconst nd/peri-future-time nd/iter-future-time
  "Periodicals must have at least one heading greater into the future to be fresh.")

(defconst nd/peri-statuscodes '(:uninit :empt :actv :unscheduled))

(defconst nd/project-invalid-todostates
  '("WAIT" "NEXT")
  "Projects cannot have these todostates.")
  
(defconst nd/org-agenda-todo-sort-order
  '("NEXT" "WAIT" "HOLD" "TODO")
  "Defines the order in which todo keywords should be sorted.")
  
(defconst nd/project-skip-todostates
  '("HOLD" "CANC")
  "These keywords override all contents within their subtrees.
Currently used to tell skip functions when they can hop over
entire subtrees to save time and ignore tasks")

variables

(defvar nd/agenda-limit-project-toplevel t
  "If true, filter projects by all levels or top level only.")

(defvar nd/agenda-hide-incubator-tags t
  "If true, don't show incubator headings.")

task helper functions

These are the building blocks for skip functions.

org-element

(defun nd/org-element-parse-headline (&optional granularity subtree)
  "Like `org-element-parse-buffer' but on only one headline. Assumes 
that point is currently on the starting line of the headline in
question. if SUBTREE is t, return all the subheadings under this
heading."
   ;; (line-beginning-position)
  (let ((start (point))
        (end (if subtree
                 (save-excursion (org-end-of-subtree))
               (save-excursion (outline-next-heading) (point)))))
    (-> (org-element--parse-elements
         start end 'first-section nil granularity nil nil)
        car)))

(defun nd/org-element-first-lb-entry (headline)
  "Get the first logbook entry of the headline under point."
  (letrec
      ((get-ts
        (lambda (obj)
          (if (eq 'clock (org-element-type obj))
              (--> obj
                   (org-element-property :value it)
                   ;; assume this will return the latest even if
                   ;; not a range
                   (org-timestamp-split-range it t))
            (->>
             obj
             org-element-contents
             car
             org-element-contents
             car
             ;; this assumes that the log timestamps are always
             ;; at the end of the first line
             (--take-while (not (eq 'line-break (org-element-type it))))
             (--last (eq 'timestamp (org-element-type it))))))))
  (-some-->
   headline
   (org-element-contents it)
   (car it)
   (org-element-contents it)
   (--first
    (equal org-log-into-drawer (org-element-property :drawer-name it))
    it)
   (org-element-contents it)
   (car it)
   (funcall get-ts it)
   (org-element-property :raw-value it))))

timestamps

(defun nd/get-date-property (timestamp-property)
  "Get TIMESTAMP-PROPERTY on current heading and convert to a number.
If it does not have a date, it will return nil."
  (let ((ts (org-entry-get nil timestamp-property)))
        (when ts (org-2ft ts))))

(defun nd/heading-compare-timestamp (timestamp-fun
                                     &optional ref-time future)
  "Returns the timestamp (from TIMESTAMP-FUM on the current heading) 
if timestamp is futher back in time compared to a REF-TIME (default to 
0 which is now, where negative is past and positive is future). If the 
FUTURE flag is t, returns timestamp if it is in the future compared 
to REF-TIME. Returns nil if no timestamp is found."
  (let* ((timestamp (funcall timestamp-fun))
        (ref-time (or ref-time 0)))
    (if (and timestamp
             (if future
                 (> (- timestamp (float-time)) ref-time)
               (<= (- timestamp (float-time)) ref-time)))
        timestamp)))
        
(defun nd/is-created-heading-p ()
  "Return heading's CREATED property timestamp or nil."
  (nd/get-date-property "CREATED"))

(defun nd/is-timestamped-heading-p ()
  "Get active timestamp of current heading."
  (nd/get-date-property "TIMESTAMP"))

(defun nd/is-scheduled-heading-p ()
  "Get scheduled timestamp of current heading."
  (nd/get-date-property "SCHEDULED"))

(defun nd/is-deadlined-heading-p ()
  "Get deadline timestamp of current heading."
  (nd/get-date-property "DEADLINE"))

(defun nd/is-closed-heading-p ()
  "Get closed timestamp of current heading."
  (nd/get-date-property "CLOSED"))

(defun nd/is-stale-heading-p (&optional ts-prop)
  "Return timestamp for TS-PROP (TIMESTAMP by default) if current heading is stale."
  (nd/heading-compare-timestamp
   (lambda () (let ((ts (org-entry-get nil (or ts-prop "TIMESTAMP"))))
           (when (and ts (not (find ?+ ts))) (org-2ft ts))))))

(defun nd/is-fresh-heading-p ()
  "Return timestamp if current heading is fresh."
  (nd/heading-compare-timestamp 'nd/is-timestamped-heading-p nil t))

(defun nd/is-archivable-heading-p ()
  "Return timestamp if current heading is archivable."
  (nd/heading-compare-timestamp
   'nd/is-closed-heading-p
    (- (* 60 60 24 nd/archive-delay-days))))

(defun nd/is-inert-p ()
  "Return most recent timestamp if headline is inert."
  (let* ((recent-ft (-some->> (nd/org-element-parse-headline)
                              nd/org-element-first-lb-entry
                              org-2ft)))
    (-some--> (or recent-ft (nd/get-date-property "CREATED"))
              (- (float-time) it)
              (when (> it (* 86400 nd/inert-delay-days)) it))))

task level testing

(defun nd/is-todoitem-p ()
  "Return todo keyword if heading has one."
  (let ((keyword (nth 2 (org-heading-components))))
    (if (member keyword org-todo-keywords-1)
        keyword)))

(defun nd/is-project-p ()
  "Return todo keyword if heading has todoitem children."
  (and (nd/heading-has-children 'nd/is-todoitem-p) (nd/is-todoitem-p)))

(defun nd/is-task-p ()
  "Return todo keyword if heading has no todoitem children."
  (and (not (nd/heading-has-children 'nd/is-todoitem-p)) (nd/is-todoitem-p)))

(defun nd/is-project-task-p ()
  "Return todo keyword if heading has todoitem parents."
  (and (nd/heading-has-parent 'nd/is-todoitem-p) (nd/is-task-p)))

(defun nd/is-atomic-task-p ()
  "Return todo keyword if heading has no todoitem parents or children."
  (and (not (nd/heading-has-parent 'nd/is-todoitem-p)) (nd/is-task-p)))

(defun nd/task-status ()
  "Return the status of the headline under point."
  (let ((kw (nd/is-task-p)))
    (when kw
      (cond 
       ((nd/is-archivable-heading-p)
        :arch)
       ((nd/is-inert-p)
        :inrt)
       ((and (member kw org-done-keywords) (not (nd/is-closed-heading-p)))
        :done-unclosed)
       ((and (not (member kw org-done-keywords)) (nd/is-closed-heading-p))
        :closed-undone)
       ((member kw org-done-keywords)
        :comp)
       (t :actv)))))

property testing

(defun nd/is-periodical-heading-p ()
  "Return t if heading is a periodical."
  (equal "periodical" (org-entry-get nil "PARENT_TYPE" t)))

(defun nd/is-iterator-heading-p ()
  "Return t if heading is an iterator."
  (equal "iterator" (org-entry-get nil "PARENT_TYPE" t)))

(defun nd/heading-has-effort-p ()
  "Return t if heading has an effort."
  (org-entry-get nil "Effort"))

(defun nd/heading-has-context-p ()
  "Return t if heading has a context."
  (let ((tags (org-get-tags-at)))
    (or (> (length (nd/filter-list-prefix "#" tags)) 0)
        (> (length (nd/filter-list-prefix "@" tags)) 0))))

(defun nd/heading-has-tag-p (tag)
  "Return t if heading has tag TAG."
  (member tag (org-get-tags-at)))

relational testing

Returns t if heading has certain relationship to other headings

(defun nd/heading-has-children (heading-test)
  "Return t if heading has a child for whom HEADING-TEST is t."
  (let ((subtree-end (save-excursion (org-end-of-subtree t)))
        has-children previous-point)
    (save-excursion
      (setq previous-point (point))
      (outline-next-heading)
      (while (and (not has-children)
                  (< previous-point (point) subtree-end))
        (when (funcall heading-test)
          (setq has-children t))
        (setq previous-point (point))
        (org-forward-heading-same-level 1 t)))
    has-children))

(defun nd/heading-has-parent (heading-test)
  "Return t if heading has parent for whom HEADING-TEST is t."
  (save-excursion (and (org-up-heading-safe) (funcall heading-test))))

(defun nd/has-discontinuous-parent ()
  "Return t if heading has a non-todoitem parent which in turn has a todoitem parent."
  (let ((has-todoitem-parent)
        (has-non-todoitem-parent))
    (save-excursion
      (while (and (org-up-heading-safe)
                  (not has-todoitem-parent))
        (if (nd/is-todoitem-p)
            (setq has-todoitem-parent t)
          (setq has-non-todoitem-parent t))))
    (and has-todoitem-parent has-non-todoitem-parent)))

project level testing

Projects are tested according to their statuscodes, which in turn are a function of the todo keywords and timestamps of their individual subtasks.

(defmacro nd/compare-statuscodes (op sc1 sc2 sc-list)
  "Compare position of statuscodes SC1 and SC2 in SC-LIST using operator OP."
  `(,op (position ,sc1 ,sc-list) (position ,sc2 ,sc-list)))

(defun nd/descend-into-project
    (allowed-statuscodes trans-tbl get-task-status callback-fun)
  "Loop through (sub)project and return overall statuscode.

The returned statuscode is chosen from list ALLOWED-STATUSCODES where 
later entries in the list trump earlier ones. 

When a subproject is encountered, this function will obtain the 
statuscode of that project and use TRANS-TBL to translate the 
subproject statuscode to one in ALLOWED-STATUSCODES (if not found an 
error will be raised). TRANS-TBL is given as an alist of two-member 
cons cells where the first member is the subproject statuscode and the
 second is the index in ALLOWED-STATUSCODES to which the subproject 
statuscode will be translated.

When a task is encountered, function GET-TASK-STATUS will be applied to
obtain a statuscode-equivalent of the status of the tasks."
  ;; define "breaker-status" as the last of the allowed-statuscodes
  ;; when this is encountered the loop is broken because we are done
  ;; (the last entry trumps all others)
  (let ((project-status (first allowed-statuscodes))
        (breaker-status (-last-item allowed-statuscodes))
        (previous-point))
    (save-excursion
      (setq previous-point (point))
      (outline-next-heading)
      ;; loop through subproject tasks until breaker-status found
      (while (and (not (eq project-status breaker-status))
                  (> (point) previous-point))
        (let ((keyword (nd/is-todoitem-p)))
          (if keyword
              (let ((new-status
                     ;; if project then descend recursively
                     (if (nd/heading-has-children 'nd/is-todoitem-p)
                         (let ((n (funcall callback-fun)))
                           ;; if project returns an allowed status
                           ;; then use that
                           (or (and (member n allowed-statuscodes) n)
                               ;; otherwise look up the value in the
                               ;; translation table and return error
                               ;; if not found
                               (nth (or (alist-get n trans-tbl)
                                        (error (concat "status not found: " n)))
                                    allowed-statuscodes)))
                       ;; if tasks then use get-task-status to obtain status
                       (nth (funcall get-task-status keyword)
                            allowed-statuscodes))))
                (if (nd/compare-statuscodes > new-status project-status allowed-statuscodes)
                    (setq project-status new-status)))))
        (setq previous-point (point))
        (org-forward-heading-same-level 1 t)))
    project-status))

(defun nd/get-project-status ()
  "Return project heading statuscode (assumes it is indeed a project)."
  (let ((keyword (nd/is-todoitem-p)))
    ;;
    ;; these first three are easy because they only require
    ;; testing the project headline and nothing underneath
    ;;
    (cond
     ;; it does not make sense for projects to be scheduled
     ((nd/is-scheduled-heading-p) :scheduled-project)

     ;; held projects do not care what is underneath them
     ;; only need to test if they are inert
     ((equal keyword "HOLD") (if (nd/is-inert-p) :inrt :held))

     ;; projects with invalid todostates are nonsense
     ((member keyword nd/project-invalid-todostates)
      :invalid-todostate)
     
     ;;
     ;; these require descending into the project subtasks
     ;;

     ;; canceled projects can either be archivable or complete
     ;; any errors or undone tasks are irrelevant
     ((equal keyword "CANC")
      (nd/descend-into-project
       '(:arch :comp)
       '((:stck . 1)
         (:inrt . 1)
         (:held . 1)
         (:wait . 1)
         (:actv . 1)
         (:sched-project . 1)
         (:invalid-todostate . 1)
         (:undone-complete . 1)
         (:done-incomplete . 1))
       (lambda (k)
         (if (and (member k org-done-keywords)
                  (nd/is-archivable-heading-p)) 0 1))
       #'nd/get-project-status))
     
     ;; done projects are like canceled projects but can also be incomplete
     ((equal keyword "DONE")
      (nd/descend-into-project
       '(:arch :comp :done-incomplete)
       '((:stck . 2)
         (:inrt . 2)
         (:held . 2)
         (:wait . 2)
         (:actv . 2)
         (:scheduled-project . 2)
         (:invalid-todostate . 2)
         (:undone-complete . 2))
       (lambda (k)
         (if (member k org-done-keywords)
             (if (nd/is-archivable-heading-p) 0 1)
           2))
       #'nd/get-project-status))
     
     ;; project with TODO states could be basically any status
     ((equal keyword "TODO")
      (nd/descend-into-project
       '(:undone-complete :stck :held :wait :actv :inrt)
       '((:comp . 0)
         (:arch . 0)
         (:scheduled-project . 1)
         (:invalid-todostate . 1)
         (:done-incomplete . 1))
       (lambda (k)
         (cond ((nd/is-inert-p) 5)
               ((equal k "TODO") (if (nd/is-scheduled-heading-p) 4 1))
               ((equal k "HOLD") 2)
               ((equal k "WAIT") 3)
               ((equal k "NEXT") 4)
               (t 0)))
       #'nd/get-project-status))
     
     (t (error (concat "invalid keyword detected: " keyword))))))

repeater testing

Iterators and periodicals are tested similarly to projects in that they have statuscodes.

(defun nd/get-iterator-project-status (kw)
  (cond
   ((or (nd/is-scheduled-heading-p)
        (member kw nd/project-invalid-todostates)) :project-error)

   ;; canceled tasks add nothing
   ((equal kw "CANC") :empt)
   
   ;;
   ;; these require descending into the project subtasks
   ;;

   ;; done projects either add nothing (empty) or are not actually
   ;; done (project error)
   ((equal kw "DONE")
    (nd/descend-into-project
     '(:empt :project-error)
     '((:unscheduled . 1)
       (:actv . 1))
     (lambda (k)
       (if (member k org-done-keywords) 0 1))
     #'nd/get-iterator-project-status))
   
   ;; project with TODO states could be basically any status
   ((equal kw "TODO")
    (nd/descend-into-project
     '(:unscheduled :empt :actv)
     '(:project-error . 0)
     (lambda (k)
       (let ((ts (nd/is-scheduled-heading-p)))
         (cond
          ((not ts) 1)
          ((< nd/iter-future-time (- ts (float-time))) 2)
          (t 0))))
     #'nd/get-iterator-project-status))
   
   (t (error (concat "invalid keyword detected: " kw)))))

(defun nd/get-iterator-status ()
  "Get the status of an iterator where allowed statuscodes are in list
 `nd/get-iter-statuscodes.' where latter codes in the list trump 
earlier ones."
  (let ((cur-status (first nd/iter-statuscodes))
        (breaker-status (-last-item nd/iter-statuscodes))
        (subtree-end (save-excursion (org-end-of-subtree t)))
        (prev-point (point)))
    (save-excursion
      (outline-next-heading)
      (while (and (not (eq cur-status breaker-status))
                  (< prev-point (point) subtree-end))
        (let ((kw (nd/is-todoitem-p))
              (new-status))
          (when kw
            ;; test if project of atomic task
            ;; assume that there are no todoitems above this headline
            ;; to make checking easier
            (setq
             new-status
             (if (nd/heading-has-children 'nd/is-todoitem-p)
                 (nd/get-iterator-project-status kw)
               (let ((ts (or (nd/is-scheduled-heading-p)
                             (nd/is-deadlined-heading-p))))
                 (cond
                  ((member kw org-done-keywords) :empt)
                  ((not ts) :unscheduled)
                  ((< nd/iter-future-time (- ts (float-time))) :actv)
                  (t :empt)))))
            (when (nd/compare-statuscodes > new-status cur-status nd/iter-statuscodes)
              (setq cur-status new-status))))
        (setq prev-point (point))
        (org-forward-heading-same-level 1 t)))
    cur-status))
    
(defun nd/get-periodical-status ()
  "Get the status of a periodical where allowed statuscodes are in list
 `nd/get-peri-statuscodes.' where latter codes in the list trump 
earlier ones."
  (letrec
      ((max-ts
        (lambda ()
          (-some-->
           (nd/org-element-parse-headline)
           (org-element-map it 'timestamp #'identity)
           (--filter
            (memq (org-element-property :type it) '(active active-range))
            it)
           (--map
            (--> it
                 (org-timestamp-split-range it t)
                 (org-element-property :raw-value it)
                 (org-2ft it))
            it)
           (-max it))))
       (compare
        (lambda (s1 s2)
          (if (nd/compare-statuscodes > s1 s2 nd/peri-statuscodes) s1 s2)))
       (new-status
        (lambda (ts)
          (-->
           ts
           (cond
            ((not it) :unscheduled)
            ((< nd/peri-future-time (- it (float-time))) :actv)
            (t :empt))
           (funcall compare it cur-status))))
       (cur-status (first nd/peri-statuscodes))
       (breaker-status (-last-item nd/peri-statuscodes))
       (subtree-end (save-excursion (org-end-of-subtree t)))
       (prev-point (point)))
    (save-excursion
      (outline-next-heading)
      (while (and (not (eq cur-status breaker-status))
                  (< prev-point (point) subtree-end))
        (setq cur-status (->> (funcall max-ts) (funcall new-status)))
        (setq prev-point (point))
        (org-forward-heading-same-level 1 t)))
    cur-status))

skip functions

These are the primary means used to sort through tasks and build agenda block views

helper skip functions and macros

Subunits for skip functions. Not meant to be used or called from the custom commands api

(defun nd/skip-heading ()
  "Skip forward to next heading."
  (save-excursion (or (outline-next-heading) (point-max))))

(defun nd/skip-subtree ()
  "Skip forward to next subtree."
  (save-excursion (or (org-end-of-subtree t) (point-max))))


(defmacro nd/skip-heading-without (heading-fun test-fun)
  "Skip headings accoring to certain characteristics. 

HEADING-FUN is a function that tests the heading and returns the 
todoitem keyword on success. TEST-FUN is a function that further tests 
the identity of the heading and may or may not use the keyword output 
supplied by the HEADING-FUN. This function will not skip if 
HEADING-FUN and TEST-FUN return true"
  `(save-restriction
     (widen)
     (let ((keyword (,heading-fun)))
       ;; (message keyword)
       (if (not (and keyword ,test-fun))
           (nd/skip-heading)))))

headings

Skip functions for headings which may or may not be todo-items.

(defun nd/skip-headings-with-tags (pos-tags-list &optional neg-tags-list)
  "Skip headings that have tags in POS-TAGS-LIST and not in NEG-TAGS-LIST."
  (save-restriction
    (widen)
    (let ((heading-tags (org-get-tags-at)))
      (if (and (or (not pos-tags-list)
                   (intersection pos-tags-list heading-tags :test 'equal))
               (not (intersection neg-tags-list heading-tags :test 'equal)))
          (nd/skip-heading)))))

(defun nd/skip-non-stale-headings ()
  "Skip headings that do not have stale timestamps and are not part of projects."
  (save-restriction
    (widen)
    (let ((keyword (nd/is-todoitem-p)))
      (if (not
           (and (nd/is-stale-heading-p)
                (not (member keyword org-done-keywords))
                (not (nd/heading-has-children 'nd/is-todoitem-p))
                (not (nd/heading-has-parent 'nd/is-todoitem-p))))
          (nd/skip-heading)))))

tasks

A few functions apply to both atomic tasks and project tasks the same.

(defun nd/skip-non-tasks ()
  "Skip headlines that are not tasks."
  (save-restriction
    (widen)
    (let ((keyword (nd/is-todoitem-p)))
      (if keyword
          (when (nd/heading-has-children 'nd/is-todoitem-p)
              (if (member keyword nd/project-skip-todostates)
                  (nd/skip-subtree)
                (nd/skip-heading)))
        (nd/skip-heading)))))

(defun nd/skip-non-created-tasks ()
  "Skip tasks that do not have CREATED timestamp properties."
  (save-excursion
    (widen)
    (if (not (and (nd/is-task-p)
                  (not (nd/is-created-heading-p))))
        (nd/skip-heading))))

atomic tasks

By definition these have no parents, so I don't need to worry about skipping over projects. Any todo state is valid and we only sort by done/canc

(defun nd/skip-non-atomic-tasks ()
  "Skip headings that are not atomic tasks."
  (save-excursion
    (widen)
    (if (not (nd/is-atomic-task-p))
        (nd/skip-heading))))

(defun nd/skip-non-closed-atomic-tasks ()
  "Skip headings that are not complete (but not archivable) atomic tasks."
  (nd/skip-heading-without
   nd/is-atomic-task-p
   (and (member keyword org-done-keywords)
        (not (nd/is-archivable-heading-p)))))

(defun nd/skip-non-archivable-atomic-tasks ()
  "Skip headings that are not archivable atomic tasks."
  (nd/skip-heading-without
   nd/is-atomic-task-p
   (nd/is-archivable-heading-p)))

repeaters

These are headings marked with PARENT_TYPE property that have timestamped headings as children. They are to be refilled when all children are stale. Note that I only care about the parent headings as the children should always show up in the agenda simply because they have timestamps. Parents can be either fresh (at least one child in the future) or stale (all children in the past).

(defun nd/skip-non-iterator-parent-headings ()
  "Skip headings that are not toplevel iterator headings."
  (save-restriction
    (widen)
    (if (not (and (nd/is-iterator-heading-p)
                  (not (nd/heading-has-parent 'nd/is-iterator-heading-p))))
        (nd/skip-heading))))

;; (defun nd/skip-non-iterator-unscheduled ()
;;   "Skip all headings that are not unscheduled iterator children."
;;   (nd/skip-heading-without
;;    nd/is-atomic-task-p
;;    (not (or (nd/is-scheduled-heading-p)
;;             (nd/is-deadlined-heading-p)))))
            
(defun nd/skip-non-periodical-parent-headings ()
  "Skip headings that are not toplevel periodical headings."
  (save-restriction
    (widen)
    (unless (and
             (nd/is-periodical-heading-p)
             (not (nd/heading-has-parent 'nd/is-periodical-heading-p)))
      (nd/skip-heading))))

;; (defun nd/skip-non-periodical-untimestamped ()
;;   "Skip all headings that are not periodical children without a timestamp."
;;   (save-restriction
;;     (widen)
;;     (if (not (and (nd/is-periodical-heading-p)
;;                   (not (nd/is-timestamped-heading-p))
;;                   (not (nd/heading-has-children 'nd/is-periodical-heading-p))))
;;         (nd/skip-heading))))

project tasks

Note that I don't care about the timestamp in these cases because I don't archive these; I archive their parent projects. The keywords I care about are NEXT, WAIT, and HOLD because these are definitive project tasks that require/inhibit futher action. (TODO = stuck which I take care of at the project level, and DONE/CANC = archivable which is dealt with similarly)

For performance, I need to assess if the parent project is skippable, in which case I jump to the next subtree.

(defun nd/skip-non-project-tasks ()
  "Skip headings that are not project tasks."
  (save-restriction
    (widen)
    (let ((keyword (nd/is-todoitem-p)))
      (if keyword
          (if (nd/heading-has-children 'nd/is-todoitem-p)
              (if (member keyword nd/project-skip-todostates)
                  (nd/skip-subtree)
                (nd/skip-heading))
            (if (not (nd/heading-has-parent 'nd/is-todoitem-p))
                (nd/skip-heading)))
        (nd/skip-heading)))))

heading-level errors

Some headings are invalid under certain conditions; these are tested here.

(defun nd/skip-non-discontinuous-project-tasks ()
  "Skip headings that are not discontinuous within projects."
  (nd/skip-heading-without
   nd/is-todoitem-p
   (nd/has-discontinuous-parent)))

(defun nd/skip-non-done-unclosed-todoitems ()
  "Skip headings that are not completed without a closed timestamp."
  (nd/skip-heading-without
   nd/is-todoitem-p
   (and (member keyword org-done-keywords)
        (not (nd/is-closed-heading-p)))))

(defun nd/skip-non-undone-closed-todoitems ()
  "Skip headings that are not incomplete with a closed timestamp."
  (nd/skip-heading-without
   nd/is-todoitem-p
   (and (not (member keyword org-done-keywords))
        (nd/is-closed-heading-p))))

projects

Projects are handled quite simply. They have statuscodes for which I test, and this can all be handled by one function. Note that this is used for "normal" projects as well as repeaters.

(defun nd/skip-non-projects (&optional ignore-toplevel)
  "Skip headings that are not projects (toplevel-only if IGNORE-TOPLEVEL is t)."
  (save-restriction
    (widen)
    (let ((keyword (nd/is-project-p)))
      (if keyword
          (if (and nd/agenda-limit-project-toplevel
                   (not ignore-toplevel)
                   (nd/heading-has-parent 'nd/is-todoitem-p))
              (nd/skip-subtree))
        (nd/skip-heading)))))

sorting and filtering

These are used to filter and sort within block agendas (note this is different from the other filtering functions above as these are non-interactive).

(defun nd/org-agenda-filter-prop (a-line filter prop-fun
                                           &optional prop-key)
  "Filter for `org-agenda-before-sorting-filter-function' where
A-LINE is a line from the agenda view, FILTER is an ordered list
of property values to be filtered/sorted, and PROP-FUN is a function
that determines a property value based on the org content of the
original buffer. If PROP-KEY is supplied, assign the return value of
PROP-FUN to PROP-KEY in A-LINE's text properties. Returns either nil
if return value of PROP-FUN not in FILTER or A-LINE (modified or not)."
  (let* ((m (get-text-property 1 'org-marker a-line))
         (s (with-current-buffer (marker-buffer m)
              (goto-char m)
              (funcall prop-fun))))
    (when (find s filter)
        (if (not prop-key) a-line
          (org-add-props a-line nil prop-key s)))))
               
(defun nd/org-agenda-regexp-replace-props (props)
  (letrec
      ((replace-prop
        (lambda (p)
          (let ((prop-val (->> (thing-at-point 'line)
                               (get-text-property 1 (cdr p))
                               symbol-name))
                (re (format "$%s$" (car p))))
            (when prop-val
              (save-excursion
                (when (search-forward re (line-end-position) t 1)
                  (replace-match prop-val))))))))
   (save-excursion
     (goto-char (point-min))
     (while (< (point) (point-max))
       (--each props (funcall replace-prop it))
       (forward-line)))))
       
(add-hook
 'org-agenda-finalize-hook
 (lambda ()
   (nd/org-agenda-regexp-replace-props '(("y" . atomic)
                                         ("xxxx" . statuscode)))))

(defun nd/org-agenda-sort-prop (prop order a b)
  "Sort a block agenda view by text property PROP given a list ORDER
of said text properties in the desired order and lines A and B as 
inputs. To be used with `org-agenda-cmp-user-defined'."
  (let* ((ta (get-text-property 1 prop a))
         (tb (get-text-property 1 prop b))
         (pa (position ta order :test (if (stringp ta) #'equal)))
         (pb (position tb order :test (if (stringp tb) #'equal))))
    (cond ((or (null pa) (null pb)) nil)
          ((< pa pb) +1)
          ((> pa pb) -1))))

(defun nd/org-agenda-sort-multi (a b &rest funs)
  "Sort lines A and B from block agenda view given functions FUNS.
Functions in FUNS must take either A or B as their arguments and
should return a positive integer indicating their rank. The FUNS
list is traversed in order, where the front is the outermost sorting
order."
  (let* ((fun (car funs))
         (pa (funcall fun a))
         (pb (funcall fun b)))
    (cond
     ((< pa pb) +1)
     ((> pa pb) -1)
     (t (-some->> funs cdr (apply #'nd/org-agenda-sort-multi a b))))))

(defun nd/org-agenda-sort-task-todo (line)
  (or
   (-some-> (get-text-property 1 'todo-state line)
            (position nd/org-agenda-todo-sort-order :test #'equal))
   (length nd/org-agenda-todo-sort-order)))
  
(defun nd/org-agenda-sort-status (line order)
  (or
   (-some-> (get-text-property 1 'statuscode line) (position order))
   (length order)))

(defun nd/org-agenda-sort-task-atomic (line)
  (if (eq '-!- (get-text-property 1 'atomic line)) 1 0))

block view building macros

Some useful shorthands to create block agenda views

(defun nd/agenda-base-heading-cmd (match header skip-fun)
  "Make a tags agenda view that matches tags in string MATCH with
header given as string HEADER and with skip function SKIP-FUN."
  `(tags
    ,match
    ((org-agenda-overriding-header ,header)
     (org-agenda-skip-function ,skip-fun)
     (org-agenda-sorting-strategy '(category-keep)))))

(defun nd/agenda-base-task-cmd (match header skip-fun &optional sort)
  "Make a tags-todo agenda view that matches tags in string MATCH with
header given as string HEADER and with skip function SKIP-FUN. Also
takes a sorting structure SORT which is passed to 
`org-agenda-sorting-strategy'"
  (or sort (setq sort ''(category-keep)))
  `(tags-todo
    ,match
    ((org-agenda-overriding-header ,header)
     (org-agenda-skip-function ,skip-fun)
     (org-agenda-todo-ignore-with-date t)
     (org-agenda-sorting-strategy ,sort))))

(defun nd/agenda-base-task-cmd* (match header skip-fun kw-list status-fun
                                         &optional status-px)
  (let ((prefix (if status-px
                    ''((tags . "  %-12:c $xxxx$: $y$ %-5:e "))
                  ''((tags . "  %-12:c         %-5:e")))))
    `(tags-todo
      ,match
      ((org-agenda-overriding-header ,header)
       (org-agenda-skip-function ,skip-fun)
       (org-agenda-todo-ignore-with-date t)
       (org-agenda-before-sorting-filter-function
        (lambda (l)
          (-some->
           l
           (nd/org-agenda-filter-prop ,kw-list ,status-fun 'statuscode)
           (nd/org-agenda-filter-prop
            '(-*- -!-) (lambda () (if (nd/is-atomic-task-p) '-!- '-*-)) 'atomic))))
       (org-agenda-cmp-user-defined
        (lambda (a b)
          (nd/org-agenda-sort-multi
           a b
           (lambda (l) (nd/org-agenda-sort-status l ,kw-list))
           #'nd/org-agenda-sort-task-atomic
           #'nd/org-agenda-sort-task-todo)))
       (org-agenda-prefix-format ,prefix)
       (org-agenda-sorting-strategy '(user-defined-down category-keep))))))

(defun nd/agenda-base-project-cmd (match header skip-fun kw-list status-fun
                                         &optional todo status-px)
  "Make a tags-todo agenda view that matches tags in string MATCH with
header given as string HEADER and with skip function SKIP-FUN. KW-LIST
is a list of keywords to be used in filtering and sorting (the order
in the list defines the sort order). STATUS-FUN is a function used to
get the statuscode of the current line in the agenda. Optional arg
TODO determines if this is a tags-todo (t) or tags (nil) block, and
STATUS-PX as t enables the statuscode to be formatted into the prefix
string."
  (let ((prefix (if status-px
                    ''((tags . "  %-12:c $xxxx$: "))
                  ''((tags . "  %-12:c        ")))))
    `(,(if 'tags-todo 'tags)
      ,match
      ((org-agenda-overriding-header ,header)
       (org-agenda-skip-function ,skip-fun)
       (org-agenda-before-sorting-filter-function
        (lambda (l) (nd/org-agenda-filter-prop l ,kw-list ,status-fun 'statuscode)))
       (org-agenda-cmp-user-defined
        (lambda (a b) (nd/org-agenda-sort-prop 'statuscode ,kw-list a b)))
       (org-agenda-prefix-format ,prefix)
       (org-agenda-sorting-strategy '(user-defined-down category-keep))))))

interactive functions

This is basically a filter but since it is implemented through skip functions it makes more sense to include it here. It allows distinguishing between toplevel projects and projects that are subprojects of the toplevel project (I usually only care about the former).

(defun nd/toggle-project-toplevel-display ()
  "Toggle all project headings and toplevel only headings in project blocks."
  (interactive)
  (setq nd/agenda-limit-project-toplevel (not nd/agenda-limit-project-toplevel))
  (when (equal major-mode 'org-agenda-mode)
    (org-agenda-redo))
  (message "Showing %s project view in agenda"
           (if nd/agenda-limit-project-toplevel "toplevel" "complete")))

advising

Some org functions don't do exactly what I want. Re-educate them here

org-tags-view done keywords

The org-tags-view can filter tags for only headings with TODO keywords (with type tags-todo), but this automatically excludes keywords in org-done-keywords. Therefore, if I want to include these in any custom agenda blocks, I need to use type tags instead and skip the unwanted TODO keywords with a skip function. This is far slower as it applies the skip function to EVERY heading.

Fix that here by nullifying org--matcher-tags-todo-only which controls how the matcher is created for tags and tags-todo. Now I can select done keywords using a match string like "+tag/DONE|CANC" (also much clearer in my opinion).

While this is usually more efficient, it may be counterproductive in cases where skip functions can be used to ignore huge sections of an org file (which is rarely for me; most only skip ahead to the next heading).

(defun nd/org-tags-view-advice (orig-fn &optional todo-only match)
  "Advice to include done states in `org-tags-view' for tags-todo agenda types."
  (nd/with-advice
      ((#'org-make-tags-matcher
        :around (lambda (f m)
                  (let ((org--matcher-tags-todo-only nil))
                    (funcall f m)))))
    (funcall orig-fn todo-only match)))

(advice-add #'org-tags-view :around #'nd/org-tags-view-advice)

block agenda views

default sorting

This gives more flexibility in ignoring items with timestamps

(setq org-agenda-tags-todo-honor-ignore-options t)

By default I want block agendas to sort based on the todo keyword (with NEXT being up top as these have priority).

(setq org-agenda-cmp-user-defined
      '(lambda (a b)
         (let ((pa (- (length (member
                               (get-text-property 1 'todo-state a)
                               nd/org-agenda-todo-sort-order))))
               (pb (- (length (member
                               (get-text-property 1 'todo-state b)
                               nd/org-agenda-todo-sort-order)))))
           (cond ((or (null pa) (null pb)) nil)
                 ((> pa pb) +1)
                 ((< pa pb) -1)))))

custom commands

These agenda commands are the center of the gtd workflow. Some are slower than dirt but that's ok becuase the load times are far less than the that I would waste rifling through each org file trying to find a task.

(let* ((actionable "-NA-REFILE-%inc")
       (periodical "PARENT_TYPE=\"periodical\"")
       (iterator "PARENT_TYPE=\"iterator\"")
       (habit "STYLE=\"habit\"")
       (task-match (concat actionable "-" periodical "-" habit "/!"))
       (act-no-rep-match (concat actionable "-" periodical "-" iterator "-" habit "/!"))
       (peri-match (concat actionable "+" periodical "-" iterator "-" habit))
       (iter-match (concat actionable "-" periodical "+" iterator "-" habit "/!")))

  (setq
   org-agenda-custom-commands
   `(("a"
      "Calendar View"
      ((agenda "" ((org-agenda-skip-function '(nd/skip-headings-with-tags '("%inc" "REFILE")))
                        (org-agenda-include-diary t)))))

     ("t"
      "Task View"
      (,(nd/agenda-base-task-cmd*
         ;; TODO, this can be better optimized if this view is split,
         ;; right now need to include DONE because may have
         ;; done-unclosed
         (concat actionable "-" periodical "-" habit "-" iterator)
         ;; act-no-rep-match
         "Tasks"
         ''nd/skip-non-tasks
         ''(:undone-closed :done-unclosed :actv :inrt)
         ''nd/task-status t)))

     ("p"
      "Project View"
      (,(nd/agenda-base-project-cmd
         act-no-rep-match
         '(concat (and nd/agenda-limit-project-toplevel "Toplevel ") "Projects")
         ''nd/skip-non-projects
         ''(:scheduled-project :invalid-todostate :undone-complete :done-incomplete
                               :stck :wait :held :actv :inrt)
         ''nd/get-project-status t t)))
          
     ("i"
      "Incubator View"
      ((agenda "" ((org-agenda-skip-function '(nd/skip-headings-with-tags nil '("%inc")))
                   (org-agenda-span 7)
                   (org-agenda-time-grid nil)
                   (org-agenda-entry-types '(:deadline :timestamp :scheduled))))
       ,(nd/agenda-base-heading-cmd "-NA-REFILE+%inc"
                                   "Stale Incubated Timestamps"
                                   ''nd/skip-non-stale-headings)
       ,(nd/agenda-base-task-cmd "-NA-REFILE+%inc/!"
                                 "Incubated Tasks"
                                 ''nd/skip-non-atomic-tasks)
       ,(nd/agenda-base-project-cmd
         "-NA-REFILE+%inc/!"
         '(concat (and nd/agenda-limit-project-toplevel "Toplevel ") "Incubated Projects")
         ''nd/skip-non-projects
         ''(:scheduled-project :invalid-todostate :undone-complete :done-incomplete
                               :stck :wait :held :actv)
         ''nd/get-project-status
         t t)))
          
     ("P"
      "Periodical View"
      (,(nd/agenda-base-project-cmd
         (concat actionable "-" iterator "+" periodical "-" habit)
         "Periodical Status"
         ''nd/skip-non-periodical-parent-headings
         'nd/peri-statuscodes ''nd/get-periodical-status nil t)))

     ("I"
      "Iterator View"
      (,(nd/agenda-base-project-cmd
         "-NA-REFILE+PARENT_TYPE=\"iterator\""
         "Iterator Status"
         ''nd/skip-non-iterator-parent-headings
         'nd/iter-statuscodes ''nd/get-iterator-status nil t)))

     ("r" "Refile" ((tags "REFILE" ((org-agenda-overriding-header "Tasks to Refile"))
                          (org-tags-match-list-sublevels nil))))
     
     ("f" "Flagged" ((tags "%flag" ((org-agenda-overriding-header "Flagged Tasks")))))
     
     ("e"
      "Critical Errors"
      (,(nd/agenda-base-task-cmd task-match
                                 "Discontinous Project"
                                 ''nd/skip-non-discontinuous-project-tasks)
       ,(nd/agenda-base-heading-cmd task-match
                                    "Undone Closed"
                                    ''nd/skip-non-undone-closed-todoitems)
       ,(nd/agenda-base-heading-cmd (concat actionable "-" periodical)
                                    "Done Unclosed"
                                    ''nd/skip-non-done-unclosed-todoitems)
       ,(nd/agenda-base-task-cmd (concat task-match)
                                 "Missing Creation Timestamp"
                                 ''nd/skip-non-created-tasks)))
       

     ("A"
      "Archivable Tasks and Projects"
      ((tags-todo ,(concat actionable "-" periodical "-" habit "/DONE|CANC")
                  ((org-agenda-overriding-header "Archivable Atomic Tasks and Iterators")
                   (org-agenda-sorting-strategy '(category-keep))
                   (org-agenda-skip-function 'nd/skip-non-archivable-atomic-tasks)))
       ,(nd/agenda-base-heading-cmd (concat actionable "-" habit)
                                   "Stale Tasks and Periodicals"
                                   ''nd/skip-non-stale-headings)
       ,(nd/agenda-base-project-cmd
         (concat actionable "-" periodical "-" iterator "-" habit)
         '(concat (and nd/agenda-limit-project-toplevel "Toplevel ") "Archivable Projects")
         ''nd/skip-non-projects ''(:arch) ''nd/get-project-status))))))

gtd next generation

GTD is great but has many limitations…mostly due to the fact that it was originally made on paper. This is meant to extend the GTD workflow into a comprehensive tracking engine that can be used and analyze and project long-term plans and goals.

logging

drawer

I prefer all logging to go in a seperate drawer (aptly named) which allows easier navigation and parsing for data analytics.

(setq org-log-into-drawer "LOGBOOK")

events

Events are nice to record because it enables tracking of my behavior (eg how often I reschedule, which may indicate how well I can predict when things should happen).

(setq org-log-done 'time
      org-log-redeadline 'time
      org-log-reschedule 'time)

repeated tasks

In these cases, it is nice to know what happened during each cycle, so force notes.

(setq org-log-repeat 'note)

creation time

org-mode has no good way out of the box to add creation time to todo entries or headings. This is nice to have as I can use them to see which tasks are bein ignored or neglected.

And yes, there is org-expiry, but it does more than I need and I don't feel like installing the extra contrib libraries.

This function adds the CREATED property. Note that I only really care about TODO entries, as anything else is either not worth tracking or an appointment which already have timestamps.

(defun nd/org-set-creation-time (&optional always)
  "Set the creation time property of the current heading.
Applies only to todo entries unless ALWAYS is t."
  (when (or always (nd/is-todoitem-p))
    (let* ((ts (format-time-string (cdr org-time-stamp-formats)))
           (ts-ia (concat "[" (substring ts 1 -1) "]")))
      (funcall-interactively 'org-set-property "CREATED" ts-ia))))

Advise the org-insert-todo-entry function. Advice here is necessary as there is only a hook for org-insert-heading and it fires before the TODO info is added.

(advice-add 'org-insert-todo-heading :after #'nd/org-set-creation-time)

Add hook for org-capture.

(add-hook 'org-capture-before-finalize-hook #'nd/org-set-creation-time)

sqlite backend

Org mode is great and all, but in many cases, text files just won't cut it. Hardcore data analysis is one of them, so make functions to shove org files (specifically archive files) into a sqlite database

load path

(add-to-list 'load-path "~/.emacs.d/dvl/org-sql/")
(require 'org-sql)

customized variables

These are variables that I set for my use but will not go into the eventual package

(setq org-sql-use-tag-inheritance t
      org-sql-ignored-properties '("CREATED")
      org-sql-files '("~/Org/general.org_archive" 
                      "~/Org/projects/"))

tools

printing

For some reason there is no default way to get a "print prompt." Instead one needs to either install some third-party helper or make a function like this.

(defun nd/helm-set-printer-name ()
  "Set the printer name using helm-completion to select printer."
  (interactive)
  (let ((pl (or helm-ff-printer-list (helm-ff-find-printers))))
    (if pl (setq printer-name (helm-comp-read "Printer: " pl)))))

magit

(use-package magit
  :ensure t
  :config
  :delight auto-revert-mode
  (setq magit-push-always-verify nil
        git-commit-summary-max-length 50))

dired

no confirm

Keeping confirmation enabled does weird stuff with helm. Not ideal at the moment but we shall see if I find something better.

(setq dired-no-confirm '(move copy))

compression

Only supports tar.gz, tar.bz2, tar.xz, and .zip by default. Add support for more fun algos such as lzo and zpaq

(if (file-exists-p "/usr/bin/7z")
    (add-to-list 'dired-compress-files-alist
                    '("\\.7z\\'" . "7z a %o %i")))

(if (file-exists-p "/usr/bin/lrzip")
    (progn
      (add-to-list 'dired-compress-files-alist
                   '("\\.lrz\\'" . "lrzip -L 9 -o %o %i &"))
      (add-to-list 'dired-compress-files-alist
                   '("\\.lzo\\'" . "lrzip -l -L 9 -o %o %i &"))
      (add-to-list 'dired-compress-files-alist
                   '("\\.zpaq\\'" . "lrzip -z -L 9 -o %o %i &"))))

;; NOTE: this must be after the shorter lrz algos otherwise it will
;; always default to .lrz and not .tar.lrz
(if (file-exists-p "/usr/bin/lrztar")
    (progn
      (add-to-list 'dired-compress-files-alist
                   '("\\.tar\\.lrz\\'" . "lrztar -L 9 -o %o %i &"))
      (add-to-list 'dired-compress-files-alist
                   '("\\.tar\\.lzo\\'" . "lrztar -l -L 9 -o %o %i &"))
      (add-to-list 'dired-compress-files-alist
                   '("\\.tar\\.zpaq\\'" . "lrztar -z -L 9 -o %o %i &"))))

formatting for humans

make sizes human readable

(setq dired-listing-switches "-Alh")

mu4e attachments

By default the included gnus-dired package does not understan mu4e, so override the existing gnus-dired-mail-buffers function to fix. This allows going to a dired buffer, marking files, and attaching them interactively to mu4e draft buffers.

(nd/when-bin
 "mu"
 ;; from here:
 ;; https://www.djcbsoftware.nl/code/mu/mu4e/Dired.html#Dired
 (require 'gnus-dired)

 (eval-after-load 'gnus-dired
   '(defun gnus-dired-mail-buffers ()
      "Return a list of active mu4e message buffers."
      (let (buffers)
        (save-current-buffer
          (dolist (buffer (buffer-list t))
            (set-buffer buffer)
            (when (and (derived-mode-p 'message-mode)
                       (null message-sent-message-via))
              (push (buffer-name buffer) buffers))))
        (nreverse buffers))))

 (setq gnus-dired-mail-mode 'mu4e-user-agent)
 (add-hook 'dired-mode-hook 'turn-on-gnus-dired-mode))

directory sized

By default dired uses ls -whatever to get its output. This does not have recursive directory contents by default. This nitfy package solves this. This is not on default because navigation is much slower and the du output adds very little in many situations (toggle when needed).

(use-package dired-du
  :ensure t
  :config
  (setq dired-du-size-format t))

mounted devices

If dired is to replace all other file managers it must handle devices. This function assumes all my devices are mounted on /media/$USER and that udevil is installed. It provides mount and mount/follow ops for all usb removable media and follow/unmount for all mounted devices (note the latter includes things that are not mounted here such as samba drives, which I normally hotkey to my window manager). This almost replicates the functionality of gvfs that I actually use without the bloat; the only missing piece is MPT for android (which will come later).

(defun nd/helm-devices ()
  "Mount, unmount, and navigate to removable media using helm."
  (interactive)
  (let* ((mounted (mapcar
                   (lambda (d)
                     `(,(file-name-base d) . ,d))
                   (nd/get-mounted-directories)))
         (mountable (seq-filter
                     (lambda (d) (not (member (car d) (mapcar #'car mounted))))
                     (nd/get-mountable-devices))))
    (helm
     :sources
     (list
      (helm-build-sync-source "Mounted Devices"
        :candidates mounted
        :action
        '(("Open" . (lambda (s) (find-file s)))
          ("Unmount" . (lambda (s) (start-process "unmount" nil "udevil" "unmount" s)))))
      (helm-build-sync-source "Mountable Devices"
        :candidates mountable
        :action
        '(("Mount and Follow" . (lambda (s)
                                  (nd/mount-device s)
                                  (find-file (nd/get-mountpoint s))))
          ("Mount" . (lambda (s) (nd/mount-device s))))))
     :buffer "*helm device buffer*"
     :prompt "Device: ")))

filtering

Filtering is useful for obvious reasons

(use-package dired-narrow
  :ensure t)

mu4e

Since mu4e is an external program, need to check that it is installed before loading anything.

(nd/when-bin
 "mu"
 (require 'mu4e)
 (defun nd/make-mu4e-context (name dir addr smtp-srv sent-behavior)
   (let ((trash (format "/%s/trash" dir))
         (drafts (format "/%s/drafts" dir))
         (sent (format "/%s/sent" dir))
         (archive (format "/%s/archive" dir))
         (inbox (format "/%s/inbox" dir))
         (mf (lambda (d msg)
               (-some-->
                msg
                (mu4e-message-field it :maildir)
                (string-prefix-p (concat "/" d) it)))))
     (make-mu4e-context
      :name name
      :match-func (-partial mf dir) ; use lexical scope here
      :vars `((mu4e-trash-folder . ,trash)
              (mu4e-drafts-folder . ,drafts)
              (mu4e-sent-folder . ,sent)
              (mu4e-refile-folder . ,archive)
              (mu4e-sent-messages-behavior . ,sent-behavior)
              (smtpmail-stream-type . starttls)
              (smtpmail-smtp-server . ,smtp-srv)
              (smtpmail-smtp-service . 587)
              (smtpmail-smtp-user . ,addr)
              (user-mail-address . ,addr)
              (mu4e-maildir-shortcuts . ((,inbox . ?i)
                                         (,sent . ?s)
                                         (,trash . ?t)
                                         (,drafts . ?d)
                                         (,archive . ?a)))))))
 (setq mail-user-agent 'mu4e-user-agent
       message-kill-buffer-on-exit t

       ;; misc
       mu4e-change-filenames-when-moving t
       mu4e-confirm-quit nil
       mu4e-compose-dont-reply-to-self t
       mu4e-get-mail-command "systemctl --user start mbsync"
       mu4e-use-fancy-chars nil

       ;; directories
       mu4e-maildir "/mnt/data/Mail"
       mu4e-attachment-dir "~/Downloads"
       
       ;; headers
       mu4e-headers-show-target nil
       mu4e-headers-fields '((:human-date . 11)
                             (:flags . 5)
                             (:from . 22)
                             (:thread-subject))
       mu4e-headers-date-format "%F"
       mu4e-headers-time-format "%R"

       ;; view
       mu4e-view-show-images t
       mu4e-view-show-addresses t
       mu4e-view-prefer-html t

       ;; compose
       mu4e-compose-signature-auto-include nil ;; sigs are annoying by default
       mu4e-compose-signature
       (string-join
        '("Nathan Dwarshuis"
          ""
          "PhD Student - Biomedical Engineering - Krish Roy Lab"
          "Georgia Institute of Technology and Emory University"
          "ndwarshuis3@gatech.edu")
        "\n")

       ;; yanking (aka citing)
       message-yank-prefix "" ;; the ">" characters are annoying
       message-yank-cited-prefix ""
       message-yank-empty-prefix ""

       ;; contexts (multiple inboxes)
       mu4e-context-policy 'pick-first
       mu4e-compose-context-policy 'ask-if-none
       mu4e-user-mail-address-list '("natedwarshuis@gmail.com"
                                     "ndwarshuis3@gatech.edu"
                                     "ndwarsh@emory.edu")
       mu4e-contexts
       (list
        (nd/make-mu4e-context "personal"
                              "gmail"
                              "natedwarshuis@gmail.com"
                              "smtp.gmail.com"
                              'delete)
        (nd/make-mu4e-context "gatech"
                              "gatech"
                              "ndwarshuis3@gatech.edu"
                              "smtp.office365.com"
                              'sent)
        (nd/make-mu4e-context "emory"
                              "emory"
                              "ndwarsh@emory.edu"
                              "smtp.office365.com"
                              'sent)))
 
 ;; enable visual line mode and spell checking
 (add-hook 'mu4e-compose-mode-hook 'turn-off-auto-fill)
 (add-hook 'mu4e-compose-mode-hook 'visual-line-mode)
 (add-hook 'mu4e-view-mode-hook 'turn-off-auto-fill)
 (add-hook 'mu4e-view-mode-hook 'visual-line-mode)
 (add-hook 'mu4e-compose-mode-hook (lambda () (flyspell-mode 1)))
 
 ;; Outlook doesn't know how to fold mu4e messages by default
 ;; This is enabled by using 32 underscores followed by the addressing
 ;; info of the previou message(s).
 (require 'nnheader) ; necessary for the header macros below

 (defun nd/message-insert-citation-header ()
   "Insert the header of the reply message."
   (let* ((h message-reply-headers)
          (sep "________________________________")
          (from (concat "From: " (mail-header-from h)))
          (date (concat "Sent: " (mail-header-date h)))
          (to (concat "To: " user-full-name))
          (subj (concat "Subject: " (message-strip-subject-re (mail-header-subject h)))))
     (insert (string-join `("" ,sep ,from ,date ,to ,subj "") "\n"))))
 
 (setq message-citation-line-function 'nd/message-insert-citation-header)

 ;; prevent html to text conversion from destroying links
 (setq
  mu4e-compose-pre-hook
  (lambda ()
    (let* ((msg mu4e-compose-parent-message)
           (html (and msg (plist-get msg :body-html)))
           ;; oops, mu4e screwed up
           (mu4e-html2text-command
            (if (executable-find "html2text")
                "html2text --ignore-emphasis --images-to-alt --body-width=0"
              'mu4e-shr2text)))
      (when (and html mu4e-view-prefer-html (member compose-type '(reply forward)))
        ;; hackity hack, since the normal mu4e-message-body-text function
        ;; does not render the desired html, do it here and force the
        ;; aforementioned function to only look at text by removing
        ;; the html
        (plist-put msg :body-txt (mu4e~html2text-shell msg mu4e-html2text-command))
        (plist-put msg :body-html nil)))))

 (require 'smtpmail)
 ;; (require 'smtpmail-async)
 ;; (require 'secrets)
 ;; (setq secrets-enabled t)
 ;; (add-to-list 'auth-sources "secrets:default")
 (setq send-mail-function 'smtpmail-send-it
       message-send-mail-function 'smtpmail-send-it)
 (add-to-list 'auth-sources (expand-file-name "~/.emacs.d/.authinfo_mu4e.gpg"))

 (use-package org-mu4e
   :after (org mu4e)
   :config
   (setq
    ;; for using mu4e in org-capture templates
    org-mu4e-link-query-in-headers-mode nil
    ;; for composing rich-text emails using org mode
    org-mu4e-convert-to-html t)))

shell

(defadvice ansi-term (before force-bash)
  (interactive (list "/bin/zsh")))
(ad-activate 'ansi-term)

(defun nd/term-send-raw-escape ()
  "Send a raw escape character to the running terminal."
  (interactive)
  (term-send-raw-string "\e"))

ediff

(setq ediff-window-setup-function 'ediff-setup-windows-plain)

jupyter

(use-package ein
  :ensure t)

keybindings

For the sake of my sanity, all bindings go here. Note this means I don't use :bind in use-package forms.

setup

Most of my modifiers are reloacted using xkb and xcape. Below is a summary where each item is in the form <original key> -> <new key action> (<key release action if used>)

• tab -> l_super (tab)
• backslash -> r_super (backslash)
• caps -> l_ctrl (escape)
• return -> r_ctrl (return)
• l_ctrl -> l_hyper
• l_super -> iso_l3_shift (xf86search)
• space -> r_alt (space)
• r_alt -> r_hyper
• r_ctrl -> caps

whichkey

Everyone forgets keybindings. When typing a key chord, this will display a window with all possible completions and their commands.

(use-package which-key
  :ensure t
  :delight
  :init
  (which-key-mode))

evil

I like being evil. All package and custom bindings go here.

base

(use-package evil
  :ensure t
  :init
  ;; this is required to make evil collection work
  (setq evil-want-integration nil
        evil-want-keybinding nil)
  :config
  (evil-mode 1))

search

By default search uses the default emacs built-in search module. Not evil enough (which really means vim search has features that I like)

(evil-select-search-module 'evil-search-module 'evil-search)

motion

By default, emacs counts a sentence as having at least 2 spaces after punctuation. Make this behave more like vim.

(setq sentence-end-double-space nil)

evil state defaults

Some modes use primitive emacs bindings by default. Educate them.

(add-to-list 'evil-motion-state-modes 'ess-help-mode)
(add-to-list 'evil-insert-state-modes 'inferior-ess-mode)

enhancements

delightfully ripped off from vim plugins

surround

(use-package evil-surround
  :ensure t
  :after evil
  :config
  (global-evil-surround-mode 1))

commentary

(use-package evil-commentary
  :ensure t
  :after evil
  :delight
  :config
  (evil-commentary-mode))

replace with register

(use-package evil-replace-with-register
  :ensure t
  :after evil
  :config
  (evil-replace-with-register-install))

unbind emacs keys

Some of these commands just get in the way of being evil (which really means that I keep pressing them on accident). Rather than nullifying them completely, tuck them away in the emacs state map in case I actually want them.

(mapc (lambda (k) (nd/move-key global-map evil-emacs-state-map (eval k)))
      '((kbd "C-s")
        (kbd "C-p")
        (kbd "C-n")
        (kbd "C-f")
        (kbd "C-b")
        (kbd "C-a")
        (kbd "C-e")
        (kbd "C-<SPC>")
        
        (kbd "C-x C-;")
        (kbd "C-x C-l")
        (kbd "C-x C-u")
        (kbd "C-x C-z")
        (kbd "C-x C-c")

        (kbd "M-c")
        (kbd "M-d")
        (kbd "M-e")
        (kbd "M-r")
        (kbd "M-f")
        (kbd "M-h")
        (kbd "M-j")
        (kbd "C-M-j")
        (kbd "M-k")
        (kbd "M-l")
        (kbd "M-m")
        (kbd "M-q")
        (kbd "M-w")
        (kbd "M-t")
        (kbd "M-u")
        (kbd "M-i")
        (kbd "M-z")
        (kbd "M-v")
        (kbd "M-/")
        (kbd "M-;")
        (kbd "M-DEL")))

evil-org

(use-package evil-org
  :ensure t
  :after (evil org)
  :delight
  :config
  (add-hook 'org-mode-hook 'evil-org-mode)
  (add-hook 'evil-org-mode-hook 'evil-org-set-key-theme)

  (require 'evil-org-agenda)
  (evil-org-agenda-set-keys)
  ;; some of the defaults bug me...
  (evil-define-key 'motion org-agenda-mode-map
    "t" 'nd/toggle-project-toplevel-display
    "D" 'org-agenda-day-view
    "W" 'org-agenda-week-view
    "M" 'org-agenda-month-view
    "Y" 'org-agenda-year-view
    "ct" nil
    "sC" 'nd/org-agenda-filter-non-context
    "sE" 'nd/org-agenda-filter-non-effort
    "sD" 'nd/org-agenda-filter-delegate
    "sP" 'nd/org-agenda-filter-non-peripheral
    "e" 'org-agenda-set-effort
    "ce" nil))

evil-magit

(use-package evil-magit
  :ensure t
  :after (evil magit))

visual line mode

This is somewhat strange because all I really care about is moving between lines and to the beginning and end as normal. However, I like the idea of thinking of paragraphs as one line (eg df. deletes a sentence even if on multiple lines). Opinion subject to change.

(evil-define-key '(normal visual) 'visual-line-mode
  "j" 'evil-next-visual-line
  "k" 'evil-previous-visual-line
  "0" 'beginning-of-visual-line
  "$" 'end-of-visual-line)

comint

Comint-based inferior modes often are not evil (eg intero and ESS). Configure this similarly to term (see below) where C-j/k navigate cmd history and insert mode goes to cmd input line.

interactive functions

Some common interactive functions for comint-based modes

;; (defun nd/comint-char-mode-evil-insert ()
;;   "If not at the last line, go to the end of the buffer and enter insert mode.  Else just enter insert mode."
;;   (interactive)
;;   (if (/= (line-number-at-pos (point)) (line-number-at-pos (point-max)))
;;         (goto-char (point-max))))
        
(defun nd/comint-send-input-evil-insert (&optional send-input-cmd)
  "Go into insert mode after calling SEND-INPUT-CMD which is usually
the function that send the command to the interactive process in the
REPL. If no SEND-INPUT-CMD then `comint-send-input' is used."
  (interactive)
  (if send-input-cmd (funcall send-input-cmd) (comint-send-input))
  (evil-insert 1))
        
(evil-define-key '(normal insert) comint-mode-map
  (kbd "C-k") 'comint-previous-input
  (kbd "C-j") 'comint-next-input)

ess

(evil-define-key 'normal inferior-ess-mode-map
  (kbd "RET") (lambda () nd/comint-send-input-evil-insert
                'inferior-ess-send-input))

;; (add-hook 'inferior-ess-mode-hook
;;           (lambda ()
;;             (add-hook 'evil-insert-state-entry-hook
;;                       'nd/comint-char-mode-evil-insert nil t)))

intero

(evil-define-key 'normal intero-repl-mode-map
  (kbd "RET") 'nd/comint-send-input-evil-insert)
  
;; (add-hook 'intero-repl-mode-hook
;;           (lambda ()
;;             (add-hook 'evil-insert-state-entry-hook
;;                       'nd/comint-char-mode-evil-insert nil t)))

collection

Most packages that don't have an evil version are in this one. I don't like surprises so I set evil-collection-modes-list with the modes I actually want. Some of these are further configured below.

(use-package evil-collection
  :ensure t
  :after evil
  :init
  (setq evil-collection-mode-list
        '(company dired ediff flycheck helm minibuffer mu4e
                  package-menu term which-key))
  (setq evil-collection-setup-minibuffer t)
  :config
  (evil-collection-init))

dired

Dired makes new buffers by default. Use find-alternate-file to avoid this.

(defun nd/dired-move-to-parent-directory ()
  "Move buffer to parent directory (like 'cd ..')."
  (interactive)
  (find-alternate-file ".."))

(defun nd/dired-xdg-open ()
  "Open all non-text files in external app using xdg-open.
Only regular files are considered."
  (interactive)
  (let* ((file-list (seq-filter #'file-regular-p (dired-get-marked-files)))
         (do-it (if (<= (length file-list) 5)
                    t
                  (y-or-n-p "Open more then 5 files? "))))
    (when do-it
      (mapc
       (lambda (f) (let ((process-connection-type nil))
                (start-process "" nil "xdg-open" f)))
       file-list))))

(defun nd/dired-open-with ()
  "Open marked non-text files in external app via open-with dialog
according to mime types as listed in all available desktop files."
  (interactive)
  (let* ((mf (seq-filter #'file-regular-p (dired-get-marked-files)))
         (qmf (mapcar #'shell-quote-argument mf))
         (file-mime-list (mapcar (lambda (f) (list f (nd/get-mime-type f))) qmf)))

    (if (= (length file-mime-list) 0)
        (message "No files selected")
      
      (let* ((first-pair (car file-mime-list))
             (last-pairs (cdr file-mime-list))
             mime-alist file-list)
        (setq file-list (nth 0 first-pair)
              mime-alist (nd/get-apps-from-mime (nth 1 first-pair)))
        ;; if multiple files selected, add to the selection list
        (if last-pairs
            (progn
              (setq file-list (string-join (mapcar #'car file-mime-list) " "))
              (dolist (mime (mapcar (lambda (f) (nth 1 f)) last-pairs))
                (setq mime-alist (intersection mime-alist
                                               (nd/get-apps-from-mime mime)
                                               :test #'equal)))))
        (if (= (length mime-alist) 0)
            (let* ((ml (delete-dups (mapcan #'cdr file-mime-list)))
                   (mls (string-join ml ", ")))
              (if (= (length ml) 1)
                  (message (concat "No apps found for mime type: "  mls))
                (message (concat "No common apps found for mime types: " mls))))
          (helm
           :sources (helm-build-sync-source "Apps"
                      :candidates mime-alist
                      :action '(("Open" . (lambda (f) (nd/execute-desktop-command f file-list)))))
           :buffer "*helm open with*"))))))

(defun nd/dired-sort-by ()
  "Sort current dired buffer by a list of choices presented in helm menu.
Note this assumes there are no sorting switches on `dired-ls'"
  (interactive)
  (let ((sort-alist '(("Name" . "")
                      ("Date" . "-t")
                      ("Size" . "-S")
                      ("Extension" . "-X")
                      ("Dirs First" . "--group-directories-first"))))
    (helm
     :sources
     (helm-build-sync-source "Switches"
       :candidates sort-alist
       :action
       '(("Sort" . (lambda (s) (dired-sort-other (concat dired-listing-switches " " s))))))
     :buffer "*helm sort buffer*")))

(put 'dired-find-alternate-file 'disabled nil)

(evil-define-key #'normal dired-mode-map
  "a" #'dired-find-file
  "za" #'gnus-dired-attach
  "gs" #'nd/dired-sort-by
  "gg" #'evil-goto-first-line
  "G" #'evil-goto-line
  "^" #'nd/dired-move-to-parent-directory
  "q" #'nd/kill-current-buffer
  "n" #'dired-narrow
  (kbd "<return>") #'dired-find-alternate-file
  (kbd "C-<return>") #'nd/dired-xdg-open
  (kbd "M-<return>") #'nd/dired-open-with)

helm

I like tab completion…regardless of what the helm zealots say. This is actually easier and faster because I can just scroll through the source list with j/k and mash TAB when I find the right directory.

(evil-define-key '(normal insert) helm-map
  (kbd "<tab>") 'helm-execute-persistent-action
  (kbd "C-<tab>") 'helm-select-action)

term

Since I use vi mode in my terminal emulator, need to preserve the escape key's raw behavior

(evil-define-key 'insert term-raw-map
  (kbd "<escape>") 'nd/term-send-raw-escape
  (kbd "C-<escape>") 'evil-normal-state)

local

These are for mode-specific bindings that can/should be outside of the evil maps above (there are not many, and these may be merged with their evil bretheren in the future).

org-mode

(add-hook 'org-mode-hook
          (lambda ()
            ;; use the hyper keys/vim arrows with the shifters instead of shift/arrows
            (local-set-key (kbd "H-k") 'org-shiftup)
            (local-set-key (kbd "H-l") 'org-shiftright)
            (local-set-key (kbd "H-j") 'org-shiftdown)
            (local-set-key (kbd "H-h") 'org-shiftleft)

            ;; this is just a useful function I made (actually I think I stole)
            (local-set-key (kbd "C-c C-x x") 'nd/mark-subtree-done)

            ;; this actually overrides org-clock-report (which I never use)
            ;; with a function to insert full clock entries for those times
            ;; I forget to clock in (often)
            (local-set-key (kbd "C-c C-x C-r") 'nd/org-clock-range)

            ;; override default org subtree cloning with something that clones and resets
            (local-set-key (kbd "C-c C-x c") 'nd/org-clone-subtree-with-time-shift)))
            
(add-hook 'org-agenda-mode-hook
          (lambda ()
            (local-set-key (kbd "C-c C-c") 'org-agenda-set-tags)
            (local-set-key (kbd "C-c C-x c") 'nd/org-agenda-clone-subtree-with-time-shift)
            (local-set-key (kbd "C-c C-x C-b") 'nd/org-agenda-toggle-checkbox)
            (local-set-key (kbd "C-c C-x C-r") 'nd/org-agenda-clock-range)))

mu4e

(nd/when-bin
 "mu"
 (define-key mu4e-headers-mode-map (kbd "C-c C-l") 'org-store-link)
 (define-key mu4e-view-mode-map (kbd "C-c C-l") 'org-store-link))

dired

(define-key dired-mode-map (kbd "C-x g") 'magit)

helm-prefix

Some of these are useful enough that I make give them a direct binding without requiring a prefix. For now this is fine.

(define-key helm-command-prefix (kbd "b") 'helm-bibtex)
(define-key helm-command-prefix (kbd "B") 'helm-bibtex-with-local-bibliography)
(define-key helm-command-prefix (kbd "S") 'helm-swoop)
(define-key helm-command-prefix (kbd "<f8>") 'helm-resume)

Give f to nd/helm-flyspell-correct instead of helm-multi-files and give the latter F (used much less).

(define-key helm-command-prefix (kbd "f") 'helm-flyspell-correct)
(define-key helm-command-prefix (kbd "F") 'helm-multi-files)

outline-magic

(define-key outline-minor-mode-map (kbd "<tab>") 'outline-cycle)

ess

They removed the underscore-inserts-arrow feature. Bring it back.

(define-key ess-r-mode-map "_" #'ess-insert-assign)
(define-key inferior-ess-r-mode-map "_" #'ess-insert-assign)

global

function

The function keys are nice because they are almost (not always) free in every mode. Therefore I use these for functions that I need to access anywhere, but not necessary extremely often (because they are out of the way and harder to reach).

(global-set-key (kbd "<f1>") 'org-agenda)
(global-set-key (kbd "<f2>") 'org-capture)
(global-set-key (kbd "<f3>") 'cfw:open-org-calendar)
(global-set-key (kbd "C-<f3>") 'nd/org-cluster-show-conflicts)
(global-set-key (kbd "C-S-<f3>") 'nd/org-cluster-show-overloads)
(global-set-key (kbd "<f4>") 'org-clock-goto)
(global-set-key (kbd "C-<f4>") 'nd/org-tomato-user-get-summary)
(global-set-key (kbd "<f5>") 'ansi-term)
(global-set-key (kbd "<f8>") 'helm-command-prefix)
(global-set-key (kbd "C-<f5>") 'nd/open-urxvt)
(nd/when-bin "mu" (global-set-key (kbd "<f12>") 'mu4e))
(global-set-key (kbd "C-<f12>") 'global-hl-line-mode)
(global-set-key (kbd "S-<f12>") 'display-line-numbers-mode)

control/meta

(global-set-key (kbd "C-<SPC>") 'company-complete)

(global-set-key (kbd "C-c e") 'nd/config-visit)
(global-set-key (kbd "C-c r") 'nd/config-reload)
(global-set-key (kbd "C-c s") 'sudo-edit)

(global-set-key (kbd "C-x 2") 'nd/split-and-follow-horizontally)
(global-set-key (kbd "C-x 3") 'nd/split-and-follow-vertically)
(global-unset-key (kbd "C-x c"))
(global-set-key (kbd "C-x k") 'nd/kill-current-buffer)
(global-set-key (kbd "C-x C-d") 'helm-bookmarks)
(global-set-key (kbd "C-x C-c C-d") 'nd/helm-devices)
(global-set-key (kbd "C-x C-f") 'helm-find-files)
(global-set-key (kbd "C-x C-b") 'helm-buffers-list)

(global-set-key (kbd "C-M-S-k") 'nd/close-all-buffers)
(global-set-key (kbd "C-M-S-o") 'nd/org-close-all-buffers)
(global-set-key (kbd "C-M-S-a") 'org-agenda-kill-all-agenda-buffers)

(global-set-key (kbd "M-b") 'nd/switch-to-previous-buffer)
(global-set-key (kbd "M-o") 'ace-window)
(global-set-key (kbd "M-s") 'avy-goto-char)
(global-set-key (kbd "M-x") 'helm-M-x)