Blockr modules

[nbenn]

This discussion topic should clarify what the server modules for fields, blocks, stacks and the workspace do, expect as inputs and produce as outputs. I will start with 4 separate posts on each of the object types.

[nbenn]

Fields

I don't think fields need to be implemented as modules. We might need a hook to be able to register some observables (i.e. for ace editor or for file browser fields), but communication with the user happens exclusively via input entries.

An (optional) server function for fields was added in #146, but I'm planning on reverting this change medium term. Can be further discussed. A question to be answered before doing this is how well we feel that "compound" fields behave.

[nbenn]

Basic requirements for fields

• An (atomic) field corresponds to a shiny UI element.
• The current field value (loosely meaning "the information that is user-provided") can be retrieved using value() and set using `value<-`().
• Field components are either static values or functions and as such require no setter, only an updater function.

[nbenn]

What kind of updates need to be possible?

• The field value has to be updated whenever the user chooses a new value.
• Field components have to be updated whenever the values of other fields change (and a given component is "functional").

[nbenn]

Blocks

The block server function instantiates a reactive representation of the block object which reacts to relevant changes in input, as well as a reactive version of the output produced by the previous block, by updating the block object whenever needed.

• Relevant entries in input are identified by input_ids() and this set op input values together with the input data in_dat (in case of non starter blocks) trigger updates.
• The block is updated via update_fields() (at least this was the case prior to Key value field / streamlined mutate block #146), which keeps all fields up to date and pushes UI updates back to the browser (note to self: naming was not great here).
• Triggered by the reactive input data and the reactive representation of the block, evaluate_block() is called which produces a reactive representation of the output data. We need a manual trigger as well, current solution with the submit_block class should be removed) and we might want to have a global option that disables the auto-updates and forces manual mode?
• The return value is the reactive representation of the block as block and a reactive representation of the output data as data.

Further tasks/actions of the block server function include the following:

• Rendering of the block output (plot, table, etc.) via server_output().
• Display of block code via server_code(). We might want to remove this for display of stack-level code, which is more self-contained at the expense of being a bit more crowded for longer stacks. Fine for now and to be discussed further.
• We output some metadata (currently number of rows, cols). I'm not sure about this as it is very data.frame-centric. How can we make this more flexible/configurable? Or do we remove this altogether?
• Currently we have a "code copy" action at the block level. I'm not sure how useful this is, as this code cannot really be run standalone. I feel that code export should only happen on stack level.
• Not implemented at the moment, but at some point we discussed that for longer-lived sessions, cleanup of inputs, UI elements and observers could become performance relevant and potentially more importantly, lack of proper cleanup could lead to name-clashes.

[nbenn]

Stacks (WIP)

The stack server function (as for the block) creates a reactive representation of the stack which is updated if

• any of the blocks change,
• the block order changes,
• a block is added,
• a block is removed.

It has available the return values of the corresponding block server modules, i.e. reactive representations of blocks and output data.

[nbenn]

Workspace (WIP)

Currently implemented as singleton object, the workspace also (unlike other objects) comes with reference semantics (implemented as environment).

[DivadNojnarg]

@nbenn, @JohnCoene

I am currently testing some possible simplifications of the plot server methods. Below is a WIP test for a dataset block, without the stack wrapper.

The core "experiment" here is the creation of 1 observer per field so that each time the related input change, we only update this field.
The consequence, is we don't need update_fields but only update_field, which avoids to call update_ui (that's currently what happens in main and leads to too many reactivity circuits).

That works well for a simple block like this, I did not check yet how to handle dynamic fields like with the filter block.

generate_server.dataset_block <- function(x, id, ...) {
  moduleServer(
    id,
    function(input, output, session) {
      ns <- session$ns
      blk <- reactiveVal(x)
      obs <- list()
      is_valid <- reactiveValues(block = FALSE)
      
      # For each block field, we listen to its related input
      # so that we can update the block accordingly and keep shiny/blockr in sync

      # I don't call update_fields but update_field as I listen to each field individually
      # which avoids to update the block when not necessary. The benefit is to avoid
      # triggering unecessary reactive issues.
      lapply(names(x), \(field) {
        # Don't process hidden fields (as they correspond to no shiny input)
        if (!inherits(x[[field]], "hidden_field")) {
          obs[[sprintf("update_%s_field", field)]] <- observeEvent(input[[field]], {
            tmp <- blk()
            tmp[[field]] <- update_field(tmp[[field]], input[[field]], list())
            blk(tmp)
  
            is_valid$block <- validate_block(blk())
          })
        }
      })
      
      out_dat <- reactive({
        validate(need(is_valid$block, "Block is not valid"))
        evaluate_block(blk())
      })

      output$res <- renderTable(out_dat())

      return(
        list(
          block = blk,
          data = out_dat,
          # Needed by the stack to block
          # computations for the next block
          is_valid = reactive(is_valid$block)
        )
      )
    }
  )   
}

# Minimal block UI for testing
generate_ui.dataset_block <- function(x, id, ...) {
  ns <- shiny::NS(id)
  tagList(
    ui_fields(x, ns, ""),
    tableOutput(ns("res"))
  )
}

library(shiny)
library(bslib)

blk <- initialize_block(new_dataset_block())

shinyApp(
  ui = page_fluid(
    generate_ui(blk, "test")
  ),
  server = function(input, output, session) {
    plop <- generate_server(blk, "test")
    observe(print(plop$data()))
  }
)

[JohnCoene]

I remember trying that to fix the infinite reactivity but unless I'm mistaken it breaks list fields and others, the problem is that we have latent dependencies between fields so we need to update some fields when others change but there's no way of knowing which currently.

[DivadNojnarg]

If we add a parent attribute to new_field, we could for instance indicate that values depends on columns (or maybe the other way around with children):

fields <- list(
    columns = new_select_field(columns, col_choices, multiple = TRUE, title = "Columns"),
    values = new_list_field(sub_fields, title = "Value", parent = "columns")
  )

Then, on the server side, the difference with my above code, is that, we'd need extra observers to call ui_update.list_field whenever input$columns changes as well as update the list of observers related to the children fields (since obs is a list, we can call obs[[my_obs]]$destroy()).