High ram utilization

[marcellov7]

Hi, thanks for your contribution!

When I set the cache to 0, the system keeps keeping the memory full. What I want to achieve, considering that we use Cloudflare as a cache in front of the service, is to avoid storing images or otherwise always free up the available RAM memory. Once processed, these images will be available through the Cloudflare cache, so we do not need to keep storing. Our goal is to optimize and reduce these spikes in deployment.

Is there a solution to this?

Thank you

[dooman87]

Hi there,

Unfortunately, the cache option wouldn't help here as it only controls HTTP header in the response, so you probably wouldn't want to set it to 0, so Cloudflare can still recognize response as cacheable.

We don't have any cache in the application, so the memory spikes would usually mean that that your server can't keep up with the load. That's happening due to when request is coming to the server the application will download a source image into the memory and sends it to the processor queue. There are several processors that are working in parallel (proc option that defaults to number of processors).

Couple of questions I have:

• What version of the application are you using?
• Do you see the response time is growing when spikes are happening?
• Does the process get killed by the OS cause there is no memory?

Couple of things I would recommend:

• Check that cache parameter is not 0 and CloudFlare is caching the images
• Try to use more powerful machine or with more cores - image processing is CPU hungry
• Setting MAGICK_TIME_LIMIT to something sensible to make sure there is no transformations that take long and blocking the queues. Could be 5 or 10 seconds?

Something we could do on our side is probably limit the amount of downloaded images we store and reject further requests (send 429 Too Many Requests) when it reaches threshold.

Let me know how you go and we can look into this further.

[marcellov7]

Hi,
Thanks for your suggestions!

• What version of the application are you using?
  latest
• Do you see the response time is growing when spikes are happening?
  No
• Does the process get killed by the OS cause there is no memory?
  Yes

the only really important thing is that it's actually true with Cloudflare that if it gets 0 it won't cach the content,
but we've created a custom rule that allows Cloudflare to cache the contents by ignoring the source cacheTTL.

So we were wondering if it would be possible to override the caching of images, so that we could leave the dirty work of caching to Cloudflare and keep the image process server pretty light at all times
(or implement an automatic cache clearing system every so much memory)

We are also testing another library (Flyimg), but the Transformimgs optimise option is really great!
We get really perfect compressions and at minimum weight.

The situation with Flyimg though is more stable, and the machine never restarts, I assume kind of cleaning.

The problem is that if the machine restarts, Cloudflare for some reason intends origin changed and will reprocess all the images even though they have already been cached

In Transformimgs graph, never even in the 1-minute range the memory drops...

Small update:
Using Flyimg, as you can see the system is stable, it is still a production system as tested for Transformimgs: the discrepancy between the two libraries is a few gb of ram memory, which is not small.

You may notice an initial spike, which occurred because there is a heavy amount of operations to do only the first time before they go into cache, but this is normal, then the situation stabilized.

In the case of transformimgs, the screenshot initially posted in the first post should correspond to the “quiet” phase.

The Transforming situation on the other hand is a bit ambiguous, we left the system on, with no load, really none, but it keeps going up for no logical reason, at least for us, the ram consumption is really excessive for no operation.

Thanks

[dooman87]

Thanks a lot for all the info and background!

The one thing I worry about is growing memory footprint when nothing is happening.

Let me run performance tests and see if there is no memory leaks around. I will also run remove in-memory cache of source images and run the tests again to see how memory footprint is doing.

We do use the same version of this library to run SaaS offering and using the latest version and its been quite stable from memory prospective. But we horizontally scale it base on the load.

One last question, do you use the docker image to run the service?

[marcellov7]

You're welcome, actually thank you for the fantastic work.

I used both this template
https://railway.app/template/a_Rgxz
It seems to me that I'm fishing from the latest version of Docker Hub.

I also always tried with Railway taking the project from git, but it seems to me that it does the same thing using docker.

If I can help further let me know,

[dooman87]

Thanks for the prompt reply!

Prob one last question that will help me to get on the right track. Are you source images are mostly PNGs or JPEGs? Or mix of both?

[marcellov7]

Really mixed, but they're also already pre-compressed a little bit. I don't know if it can be useful, I can invite you to Railway (if it can be done) on the instance that is currently without traffic.
Anything to be helpful. Thank you

A small update:
maybe some traffic was present until this morning, but still the ram shots to process a single image seem too high to me still.

For example for 2 images:

[dooman87]

I was able to reproduce the problem and got something going for the fix here: #52. The PR is almost ready codewise, but we'd need to do it a bit more testing to release it. I could push the image to Docker hub if you'd like to test it as well and see if it's fixing the problem?

The problem is imagick library which is caching things and eats as much memory as it needs. For instance, on my local machine when I run performance test the memory grows to 2.5Gb then just stays at that level no matter how many times i run the test. That scenario above is working well when you have dedicated instance for transformimgs but obviously is not working well when it's shared or constrained environment.

[marcellov7]

Ok I'll try it directly from the branch, I'll let you know. Thanks in the meantime.

[dooman87]

Alright, I tested that one and the memory consumption never crossed 1Gb for me. There is a slight (+3%) increase in latency but gain in the memory usage worth it. Closing this one for now, but happy to re-open if the problem still exist.