Template Compilation Error

[osktyn]

`
Template Compilation Error

Unhandled memory fault at #x0..
In #<COMPILED-TEMPLATE /var/username/.roswell/lisp/quicklisp/dists/quicklisp/software/djula-20170227-git/templates/error-template.djhtml {20629C3B}>.
Date/time: 2017-08-08-08:00An unhandled error condition has been signalled:
Unhandled memory fault at #x0.

Backtrace for: #
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #)
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE # :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL)
2: #
`

Error is thrown when a webserver (cavemanbased, just like in #32, related?) calls one of several templates that loops and plots some javascript. The template doesn't have any {%include%} or {%if%}, but some {%for%} loops.

I have severe problems to reproduce the error, but once it shows up, calling that template will produce the same error until the whole process is killed.

The error have been present for several months, but since I have had such problems pinpointing it have felt hesitant to report it until now.

[mmontone]

Oh. Sorry to here this. I think this is likely related to #32 . Will have to see if I discover something.

[mmontone]

Some times it works, and some times it doesn't, right? (your template)

[mmontone]

And the error occurs always in the same template?

[mmontone]

One thing to try if you are running production, is to do:

(push :djula-prod *features*)

before asdf:require your system. That tells Djula to not look at the status of templates on disk and never recompile. Perhaps that helps.

[osktyn]

Some times it works, and some times it doesn't, right? (your template)

Precisely.

And the error occurs always in the same template?

In anyone of four different templates, but those are the only ones I use and they all have some {%} elements.

(push :djula-prod features)

I'll try that and report back.

[osktyn]

I think that the djula push have helped (I'm usually running with a connected repl, so sometimes one is not 100% sure of the state of all variables when an error occured).

I'm happy with what seems like quick-fix, but if there is something more I can do I'd be glad to help.

[mmontone]

I'm glad you are glad to help :)

Well, for a start, if the :djula-prod feature has helped, then there's a problem with template recompilation, as that :djula-prod feature should not be needed. It means that templates are being recompiled even when they don't change on disk. So I think looking at template compilation, and when it is trigged could be a good idea:

djula/src/compiler.lisp

Line 49 in 594837c

(defmethod initialize-instance :after ((compiled-template compiled-template) &rest initargs)

. Perhaps making sure that no recompilation is triggered when templates haven't changed on disk.

[mmontone]

Anyway, it's good to run with :djula-prod on production, because there's no checking of template files changes.

[mmontone]

And I should document that feature, btw.

[osktyn]

I think I have triggered the error even with :djula-prod on. At least 90% sure it was on (it is set when the program launches but I couldn't confirm the value once the error showed up).

I don't know if this clears or obscures the picture. My hunch was that the error has occurred with much lower frequency after :djula-prod. But again, not being able to reliantly trigger the error is annoying.

[mmontone]

Mm. Ok. This is tricky. I think one thing we could do is try to automate this.

Take a template and call render-template* thousands of times, until it breaks, or revise the memory somehow if we have the chance.

[mmontone]

I'm trying this myself but nothing happens. I can't reproduce.

CL-USER> (djula:add-template-directory #p"/home/marian/")
(#P"/home/marian/" #P"/home/marian/src/lisp/djula/test/templates/")
CL-USER> (defparameter +template+ (djula:compile-template* "test.html"))
+TEMPLATE+
CL-USER> +template+
#<DJULA::COMPILED-TEMPLATE /home/marian/test.html {10082CE76B}>
CL-USER> (loop repeat 1000 do
              (djula:render-template* +template+ nil :flag t))
NIL
CL-USER> (room)
Dynamic space usage is:   158,569,856 bytes.
Read-only space usage is:      4,944 bytes.
Static space usage is:         3,168 bytes.
Control stack usage is:       10,840 bytes.
Binding stack usage is:          976 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.

Breakdown for dynamic space:
  38,391,824 bytes for 2,399,489 cons objects.
  30,076,288 bytes for   248,257 simple-vector objects.
  22,217,648 bytes for   260,056 instance objects.
  20,476,944 bytes for    24,572 code objects.
  17,606,496 bytes for   147,583 simple-character-string objects.
   8,386,128 bytes for     7,046 simple-array-unsigned-byte-64 objects.
  21,447,152 bytes for   446,145 other objects.
  158,602,480 bytes for 3,533,148 dynamic objects (space total.)
; No value

Perhaps you can try something similar with you templates?

[osktyn]

Using the same code as you, but with all four templates where I have confirmed errors. Neither I can reproduce the error with or without (push :djula-prod features). The (room) output looks similar. But I have still seen the error in the last two weeks when running the server.

I don't know what to do here. I have moved my "looping logic", I now have new templates without the "{%for%}", so I can avoid the error. I am open to:

• Closing the issue (after all, I can now make this not an issue and nobody else have seems to have it)
• Leave it open for the future (I think the issue is still there, somewhere)
• Trying other things to reproduce the error.

Is it conceivable that the error is not in djula, but djula happens to catch the error?

[mmontone]

I think we should leave this open as it is not solved.

I don't know if it's possible that Djula catches and error that is not in Djula; to me this sounds like a problem in Djula. I have a suspicion on the expression parser used for ifs and fors, that is implemented using parser combinators, that it may be leaking memory, but I would have to test.

Not sure how else to test this, apart from repeating the tests with bigger numbers, like loop repeat 10000000, etc, and with changing conditions on the templates (giving templates different data each time). And also, it would be cool if you could observe the memory consumption of your production application, I guess that could give us a clue if you verify that memory consumption increases along the week.

Btw, are you using SBCL --dynamic-space-size option?

[mmontone]

Actually, now that I look, it may be that you don't have a heap exhaustion problem, but some other problem, and Djula is trying to catch it.

Template Compilation Error

Unhandled memory fault at #x0..
In #<COMPILED-TEMPLATE /var/username/.roswell/lisp/quicklisp/dists/quicklisp/software/djula-20170227-git/templates/error-template.djhtml {20629C3B}>.
Date/time: 2017-08-08-08:00An unhandled error condition has been signalled:
Unhandled memory fault at #x0.

Djula is compiling the error-template.djhtml there, to show the error.

Can you paste a more complete backtrace of the errors here in the future?

[osktyn]

(loop repeat 1000000 do (djula:render-template* +template+ nil :flag t))

doesn't trigger it either. Anyway, I'm under the impression that it has occured so often that already 1000 loops certainly should have caught it, if it would ever happen.

I have observed the error with

(setq inferior-lisp-program "ros dynamic-space-size=4096 -Q run")

Let's decide that I run one instance of the server without djula-prod (just for the sake of it) and with the old templates until I get the error again. Then I'll try to document the hell out of it and post it here.

[osktyn]

Pastebin

Complete backtrace. There might be a tendency that the problem occurs when there is much/more data in the call to the template.

[mmontone]

Aha. It's a memory fault error on render-template, but I'm still clueless, sorry. :/