Implement recurring events and todos (#308)

doc/org-caldav.org

@@ -308,6 +308,44 @@ to prevent the CATEGORY from being exported to iCalendar. This problem
 only seems to affect some CalDav servers: in particular, NextCloud
 is affected, but Radicale does not seem to experience this problem.
 
+** Behavior of recurring TODO deadlines without a start time
+:PROPERTIES:
+:CUSTOM_ID: recur-deadline
+:END:
+
+Technically, the [[https://datatracker.ietf.org/doc/html/rfc5545#section-3.8.2.4][iCalendar spec]] requires repeating events and todos
+(i.e. having an ~RRULE~ property) to have a starting time (~DTSTART~
+in iCalendar, equivalent to ~SCHEDULED~ in Org TODOs).  This means
+that, a TODO with a repeating ~DEADLINE~ but without a ~SCHEDULED~
+property, such as below, is not allowed by the iCalendar spec:
+
+#+begin_src org
+  ,* TODO An example todo with a repeating deadline and no start time
+  DEADLINE: <2024-09-15 Sun +1w>
+#+end_src
+
+This is a clear shortcoming of the iCalendar spec, because it /does/
+allow tasks to have a standalone deadline without a starting time, but
+/doesn't/ allow such tasks to repeat.
+
+By default, ~ox-icalendar~ follows the iCalendar spec, and when
+exporting a TODO with a repeating ~DEADLINE~ but no ~SCHEDULED~
+timestamp, will add a start time based on
+~org-deadline-warning-days~. On future syncs, this start time will be
+inserted into Org as a ~SCHEDULED~ timestamp.
+
+However, in practice, many iCalendar implementations ignore this
+limitation, and allow Todos with ~DEADLINE~ (~DUE~) times to have
+repeaters (~RRULE~), even if they are missing ~SCHEDULED~ (~DTSTART~)
+times. If your CalDav server allows this, then you may set the
+variable ~org-icalendar-todo-unscheduled-start~ to ~nil~.  This will
+prevent ~ox-icalendar~ from adding a start time to such TODOs, thus
+preventing the ~SCHEDULED~ timestamp from being inserted on future
+syncs.
+
+See [[#recur-event-todo][Repeating events and todos]] for more details about how org-caldav
+handles repeating events and todos.
+
 * Compatible CalDav servers
 :PROPERTIES:
 :CUSTOM_ID:       caldav-servers
@@ -455,6 +493,26 @@ entry in Org /and/ in the calendar, the changes in the calendar will
 be lost. I might implement proper conflict handling some day, but
 don't hold your breath (patches are welcome, of course).
 
+** Repeating events and todos
+:PROPERTIES:
+:CUSTOM_ID: recur-event-todo
+:END:
+
+Org-caldav has basic support for repeating events and todos. In
+particular, simple Org timestamp repeaters such as ~+3d~ or ~+1m~ can
+be succesfully sync'd bidirectionally.
+
+However, complex iCalender recurrences, such as "repeat on the 2nd
+Tuesday of each month until X date", are not supported.
+
+For Org TODOs with both ~SCHEDULED~ and ~DEADLINE~ timestamps, each of
+the timestamps must have the same repeater, otherwise the behavior is
+undefined.
+
+Furthermore, the behavior of Org TODOs with a repeating ~DEADLINE~
+timestamp, but no ~SCHEDULED~ timestamp, has some subtleties; see the
+configuration section: [[#recur-deadline][Behavior of recurring TODO deadlines without a start time]].
+
 ** How syncing happens (a.k.a. David's little CalDAV rant)
 
 (This is probably not interesting, so you can skip this.)
@@ -590,10 +648,6 @@ from those events.
 
 * Known Bugs
 
-- Recurring events created or changed on the calendar side cannot be
-  synced (they will work fine as long as you manage them in Org,
-  though).
-
 - Syncing is currently pretty slow since everything is done
   synchronously.
 

org-caldav-tests.el

@@ -1117,3 +1117,88 @@ https://orgmode.org
   ;; Test if second sync can find the ID we created. If not, the test
   ;; will exit with org-caldav error "Could not find UID"
   (org-caldav-sync))
+
+(defun org-caldav-test-input-output-entry (input output)
+  "Helper function to test Org->Cal->Org preserves an entry.
+Clear the Org files and iCalendar. Then add INPUT to
+`org-caldav-test-inbox'.  Then sync it to iCalendar. Then reset
+the Org files and database, pull from iCalendar, and check that
+the result matches the regular expression OUTPUT."
+  (message "Setting up temporary files")
+  (org-caldav-test-setup-temp-files)
+  (setq org-caldav-calendar-id (car org-caldav-test-calendar-names))
+  ;; Set up data for org-caldav.
+  (setq org-caldav-files (list org-caldav-test-orgfile))
+  (setq org-caldav-inbox org-caldav-test-inbox)
+
+  (message "Cleaning up upstream calendars")
+  (org-caldav-test-set-up)
+
+  ;; Set up orgfile.
+  (with-current-buffer (find-file-noselect org-caldav-test-orgfile)
+    (insert input)
+    (save-buffer))
+
+  (message "Sync")
+  ;; Sync event to iCal
+  (org-caldav-sync)
+
+  ;; Reset org-caldav sync state
+  (delete-file (org-caldav-sync-state-filename org-caldav-calendar-id))
+  (setq org-caldav-event-list nil)
+  (setq org-caldav-sync-result nil)
+  ;; Also delete the event in org
+  ;;(delete-file org-caldav-test-orgfile)
+  (with-current-buffer (find-file-noselect org-caldav-test-orgfile)
+    (erase-buffer)
+    (save-buffer))
+
+  ;; Sync event back to inbox
+  (org-caldav-sync)
+  (with-current-buffer (find-file-noselect org-caldav-test-inbox)
+    (goto-char (point-min))
+    (should (re-search-forward
+             output))))
+
+(ert-deftest org-caldav-13a-test-simple-repeating-event ()
+  (org-caldav-test-input-output-entry
+   "* Simple repeating event
+:PROPERTIES:
+:ID:       test-repeating-event
+:END:
+<2024-05-25 Sat +7d>"
+   "* Simple repeating event
+:PROPERTIES:
+:ID:\\s-+test-repeating-event
+:END:
+<2024-05-25 Sat \\+7d>"))
+
+(ert-deftest org-caldav-13b-test-simple-repeating-todo-dtstart ()
+  (let ((org-caldav-sync-todo t)
+        (org-icalendar-include-todo 'all))
+    (org-caldav-test-input-output-entry
+     "* TODO Simple repeating scheduled todo
+SCHEDULED: <2024-06-08 Sat +3d>
+:PROPERTIES:
+:ID:       test-simple-repeating-todo-dtstart
+:END:"
+     "* TODO Simple repeating scheduled todo
+SCHEDULED: <2024-06-08 Sat \\+3d>
+:PROPERTIES:
+:ID:\\s-+test-simple-repeating-todo-dtstart
+:END:")))
+
+(ert-deftest org-caldav-13c-test-simple-repeating-todo-dtstart-due ()
+  (let ((org-caldav-sync-todo t)
+        (org-icalendar-include-todo 'all))
+    (org-caldav-test-input-output-entry
+     "* TODO Simple repeating scheduled todo with deadline
+SCHEDULED: <2024-06-08 Sat +3d> DEADLINE: <2024-06-10 Mon +3d>
+:PROPERTIES:
+:ID:       test-simple-repeating-todo-dtstart-due
+:END:"
+     "* TODO Simple repeating scheduled todo with deadline
+\\(SCHEDULED: <2024-06-08 Sat \\+3d> DEADLINE: <2024-06-10 Mon \\+3d>\\|DEADLINE: <2024-06-10 Mon \\+3d> SCHEDULED: <2024-06-08 Sat \\+3d>\\)
+:PROPERTIES:
+:ID:\\s-+test-simple-repeating-todo-dtstart-due
+:END:")))

org-caldav.el

@@ -1119,6 +1119,7 @@ ICSBUF is the buffer containing the exported iCalendar file."
           (org-caldav-fix-todo-status-percent-state)
           (org-caldav-fix-categories)
           (org-caldav-fix-todo-dtstart)
+          (org-caldav-fix-todo-remove-until)
           (message "Putting event %d of %d Org --> Cal" counter (length events))
           (if (org-caldav-put-event icsbuf)
               (org-caldav-event-set-etag cur 'put)
@@ -1334,6 +1335,35 @@ also look if there is a deadline."
               (org-get-scheduled-time nil)))
           (delete-region (line-beginning-position) (+ 1 (line-end-position))))))))
 
+(defun org-caldav-fix-todo-remove-until ()
+  "Remove RRULE UNTIL from VTODOs in ox-icalendar export.
+ox-icalendar has confusing behavior around RRULE UNTIL in todos:
+Todos with repeating SCHEDULED timestamp and non-repeating
+DEADLINE timestamp treat the latter as RRULE UNTIL.  This makes
+sense for users who set
+`org-agenda-skip-scheduled-repeats-after-deadline' but that
+option is not well known and nil by default.  A better and more
+flexible approach (that applies to both Todos and non-Todos)
+would be to use diary-style sexp timestamps for UNTIL and other
+complex RRULEs, but this requires implementing sexp timestamps
+during import, as well as upstream work to fix sexp timestamps on
+export (ox-icalendar export of sexp timestamps appears broken as
+of current writing 2024-08-26).  For now, we simply ignore UNTIL
+during Cal->Org, and strip out UNTIL during Org->Cal, until we
+figure out a better path forward -- ideally in collaboration with
+upstream ox-icalendar."
+  (save-excursion
+    (goto-char (point-min))
+    (when (search-forward "BEGIN:VTODO" nil t)
+      (when (re-search-forward "^RRULE:\\(.*UNTIL=[^\^M\n]+\\)" nil t)
+        (replace-match
+         (save-match-data
+           (string-join (cl-loop for x in (string-split (match-string 1) ";")
+                                 unless (string-prefix-p "UNTIL=" x)
+                                 collect x)
+                        ";"))
+         nil nil nil 1)))))
+
 (defun org-caldav-inbox-file (inbox)
   "Return file name associated with INBOX.
 For format of INBOX, see `org-caldav-inbox'."
@@ -1517,7 +1547,8 @@ which can only be synced to calendar. Ignoring." uid))
                               ;; before calling re-search-forward
                               (let ((tr (org-caldav-create-time-range
                                          .start-d .start-t
-                                         .end-d .end-t .end-type)))
+                                         .end-d .end-t
+                                         .e-type .rrule-props)))
                                 (when (re-search-forward org-tsr-regexp nil t)
                                   (replace-match tr nil t)))))
                       (widen))
@@ -1869,19 +1900,28 @@ Returns MD5 from entry."
           (org-caldav--insert-description .description)
           (forward-line -1)
           (when .start-d
+            ;; FIXME use `org-schedule' or `org-deadline' instead
+            ;; (here and elsewhere)
             (org--deadline-or-schedule
-             nil 'scheduled (org-caldav-convert-to-org-time .start-d .start-t)))
+             nil 'scheduled (org-caldav--convert-to-org-time-with-brackets
+                             .start-d .start-t
+                             .rrule-props)))
           (when .due-d
             (org--deadline-or-schedule
-             nil 'deadline (org-caldav-convert-to-org-time .due-d .due-t)))
+             nil 'deadline (org-caldav--convert-to-org-time-with-brackets
+                            .due-d .due-t
+                            .rrule-props)))
           (when .completed-d
             (org-add-planning-info 'closed (org-caldav-convert-to-org-time .completed-d .completed-t)))
           (org-caldav-set-org-tags .categories)
           (when .uid (org-set-property "ID" (url-unhex-string .uid)))
           (org-caldav-insert-org-entry--wrapup))
       (insert (make-string (or .level 1) ?*) " " .summary "\n")
       (insert (if org-adapt-indentation "  " "")
-              (org-caldav-create-time-range .start-d .start-t .end-d .end-t .e-type) "\n")
+              (org-caldav-create-time-range .start-d .start-t
+                                            .end-d .end-t
+                                            .e-type .rrule-props)
+               "\n")
       (org-caldav--insert-description .description)
       (forward-line -1)
       (when .uid
@@ -1946,7 +1986,8 @@ Sets the block's tags, and return its MD5."
         (org-caldav--org-set-tags-to (reverse cleantags)))
       (org-caldav--org-set-tags-to nil))))
 
-(defun org-caldav-create-time-range (start-d start-t end-d end-t e-type)
+(defun org-caldav-create-time-range (start-d start-t end-d end-t
+                                             e-type &optional rrule-props)
   "Create an Org timestamp range from START-D/T, END-D/T."
   (with-temp-buffer
     (cond
@@ -1957,20 +1998,30 @@ Sets the block's tags, and return its MD5."
 	     (not (equal end-d start-d)))
 	(progn
 	  (insert "--")
-	  (org-caldav-insert-org-time-stamp end-d end-t))
+	  (org-caldav-insert-org-time-stamp end-d end-t)
+          (when rrule-props
+            ;; TODO Implement repeating multiday events upstrean in ox-icalendar
+            (org-caldav-debug-print 1 "Skipping repeater for multiday timestamp (ox-icalendar does not yet support repeating multiday time ranges)")))
+      (backward-char 1)
       (when end-t
-	;; Same day, different time.
-	(backward-char 1)
-	(insert "-" end-t)))
+        (insert "-" end-t))
+      (when rrule-props
+        (insert (format " +%d%s"
+                        (read (or (cadr (assoc 'INTERVAL rrule-props)) "1"))
+                        (downcase (substring (cadr (assoc 'FREQ rrule-props)) 0 1))))))
     (buffer-string)))
 
 (defun org-caldav-insert-org-time-stamp (date &optional time)
   "Insert org time stamp using DATE and TIME at point.
 DATE is given as european date (DD MM YYYY)."
   (insert
-   "<" (org-caldav-convert-to-org-time date time) ">"))
+   (org-caldav--convert-to-org-time-with-brackets date time)))
+
+(defun org-caldav--convert-to-org-time-with-brackets (&rest args)
+  "Wraps `org-caldav-convert-to-org-time' results with angle brackets."
+  (concat "<" (apply 'org-caldav-convert-to-org-time args) ">"))
 
-(defun org-caldav-convert-to-org-time (date &optional time)
+(defun org-caldav-convert-to-org-time (date &optional time rrule-props)
   "Convert to org time stamp using DATE and TIME.
 DATE is given as european date \"DD MM YYYY\"."
   (let* ((stime (when time (mapcar 'string-to-number
@@ -1982,9 +2033,13 @@ DATE is given as european date \"DD MM YYYY\"."
                                      (calendar-extract-day sdate)
                                      (calendar-extract-month sdate)
                                      (calendar-extract-year sdate))))
-    (if time
-        (format-time-string "%Y-%m-%d %a %H:%M" internaltime)
-      (format-time-string "%Y-%m-%d %a" internaltime))))
+    (concat
+     (if time
+         (format-time-string "%Y-%m-%d %a %H:%M" internaltime)
+       (format-time-string "%Y-%m-%d %a" internaltime))
+     (when rrule-props
+       (format " +%d%s" (read (or (cadr (assoc 'INTERVAL rrule-props)) "1"))
+               (downcase (substring (cadr (assoc 'FREQ rrule-props)) 0 1)))))))
 
 (defun org-caldav--convert-to-calendar (date)
   "Convert DATE to calendar.el-style list (month day year).
@@ -2207,7 +2262,9 @@ which can be fed into `org-caldav-insert-org-event-or-todo'."
 		             "No Title")))
             (description . ,(icalendar--convert-string-for-import
 		             (or (icalendar--get-event-property e 'DESCRIPTION)
-			         ""))))))
+			         "")))
+            (rrule-props . ,(icalendar--split-value
+                             (icalendar--get-event-property e 'RRULE))))))
     (if is-todo
         (org-caldav-convert-event-or-todo--todo e zone-map eventdata-alist)
       (org-caldav-convert-event-or-todo--event e zone-map eventdata-alist))))
@@ -2257,7 +2314,7 @@ which can be fed into `org-caldav-insert-org-event-or-todo'."
 		(location
 		 . ,(icalendar--convert-string-for-import
 		     (or (icalendar--get-event-property e 'LOCATION) "")))
-		(end-type . ,e-type))
+		(e-type . ,e-type))
 	      eventdata-alist))))
 
 (defun org-caldav-convert-event-or-todo--todo (e zone-map eventdata-alist)