forked from Unidata/LDM
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGE_LOG
3229 lines (2403 loc) · 132 KB
/
CHANGE_LOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
6.13.5 2016-10-06T13:53:22-0600
ldmd(1):
Added logging of received signals. In particular, SIGINT and SIGTERM
are logged at the notice level.
pqact(1):
Corrected handling of blank but non-empty lines in configuration-file.
Before, they would terminate parsing of the configuration-file.
noaaportIngester(1):
Merged some changes by Raytheon.
fileIngester(1):
Added this utility created by Raytheon in support of NOAAPort. Don't ask
me what it does.
LDM library:
Improved log message for when shmget() fails.
noaaportIngester(1):
Added missing basename(1) declaration.
Ported to FreeBSD (thanks Eric).
HTML Documentation:
Added activity diagrams for ldmd(1), upstream LDM, downstream LDM, and
pqact(1).
Added product-queue graphic.
Added troubleshooting page.
Improved LDM tips.
Multicast (not ready yet):
Added FMTP-LDM7 as a submodule.
6.13.4 2016-08-05T10:39:58-0600
Logging:
log.c:
Ensured that the "refresh_logging" flag is cleared if and only if
the destination is refreshed to eliminate a potential race
condition.
On a system built without configure(1)'s "--with-ulog" option:
refresh_logging(1):
Corrected the signal from SIGHUP to SIGUSR1.
Prefixed kill(1) command with command(1) to avoid non-standard
kill(1)-s (e.g., those that expect "%job").
log.c:
Changed the signal that the system uses to re-open log files
from SIGHUP to SIGUSR1 to follow more closely the semantics of
these signals. (Re-opening a log file is necessary to support
log file rotation.) SIGUSR1 is also used by some LDM utilities
to log performance statistics; consequently, such statistics
will appear at the start of a rotated log file. The extant LDM
utilities that log such statistics are: dvbs_multicast(1),
noaaportIngester(1), readnoaaport(1), pqcat(1), pqcopy(1),
pqing(1), pqsend(1), pqsurf(1), and rtstats(1).
ldmd(1):
Changed "obsolete" to "redundant" in the log message about terminating a
previous upstream LDM whose subscription is a subset of the
currently-subscribing downstream LDM.
Blocked delivery of the SIGUSR1 signal for EXEC-ed programs so that it
won't terminate programs that don't handle it (e.g., the McIDAS
product-decoding programs). Programs that do handle it must and do
explicitly unblock this signal.
pqcat(1):
Added printing of product-class to starting-up message.
pqcopy(1):
Corrected usage-message and manual-page.
ldmadmin(1):
Fixed computation of the number of LDM connections on a Solaris system
by making the algorithm more robust to variations in netstat(1) output
and by using uldbutil(1).
Multicast:
Ensured removal of test product-queues in up7_down7_test.c.
6.13.3 2016-07-20T14:56:07-0600
logging:
Enhanced performance by making "log_level" volatile and having LOG_LOG()
test it directly.
Improved robustness by moving log_refresh() action from logl_lock() to
the static I/O function flush().
ldmd(1):
Changed the exit status of the top-level LDM server from 0 to 3 when it
terminates due to the crashing of a child process.
pqact(1):
Corrected "pq_sequence()" to "pq_next()" in log message in pqact.c.
notifyme(1):
Corrected handling of "-l ..." option.
6.13.2 2016-07-13T10:32:17-0600
Logging:
log.c:
Added check of return-value of logi_set_destination() in log_init().
Added performance test and profiling target.
ulogger(1):
Enabled makefile "make ctags" target.
Product-Queue:
pq.c:
Added function pq_next() as a more performant replacement for
pq_sequence() and pq_sequenceLock().
Squashed latent bug that would eventually cause an assertion failure
or a segmentation violation if a new skip-list node couldn't be
obtained.
Restored caching to fb_sz(), tq_sz(), fl_sz(), and ix_sz() to
improve performance. Added log_assert(nelems) to same and to
ix_ptrs().
Inlined some static functions to obtain small performance
improvements.
test_pq.c:
Created test_pq_sequence().
pq_test_insert(1):
Added ability to periodically insert synthetic data products.
pqact(1):
pqact.c:
Refactored to use pq_next().
Moved setting of insertion-time of last successfully-processed
product from main() to processProduct() to increase performance.
Corrected time-value in first "Processed oldest product" message.
filel.c:
Improved performance by
Converting the function entry_isFlagSet() into a macro
(profiling revealed that pqact(1) spent about 15% of its time in
this function).
Executing the fl_sync() function only when the head of the
product- queue is reached instead of after every matched
data-product (profiling revealed that pqact(1) spent about 1/3
of its time in this function).
Squashed latent bug in format argument of log_add_syserr() in
unio_put().
palt.c:
Rewrote processProduct(), which is called by pq_next().
Rewrote dummyprod() to use new processProduct().
Added test and profiling targets.
noaaportIngester(1) (productMaker.c):
Reverted the computation of the MD5 checksum in productMaker.c for
compressed GINI data-blocks when neither the "-c" nor "-f" options are
specified from Raytheon's algorithm (which always decompresses and
recompresses the data-block) to the original algorithm (which uses the
data-block "as is"). This was done because Raytheon's algorithm results
in MD5 checksums that differ from previous versions of
noaaportIngester(1) -- resulting in failure of the duplicate product
rejection mechanism of the product-queue.
Incorporated Raytheon's bug-fix that eliminates unnecessary
retransmission requests for GINI products from NOAA's NCF. (Only
requests from NOAA offices are honored).
ldmadmin(1):
Added check for the pqsurf(1) queue for any pqsurf(1) EXEC-ed in the LDM
configuration-file.
refresh_logging(1):
Modified makefile to create logging-specific script at build-time to
accomodate Solaris' broken sh(1) (`test -z "#"` fails).
Changed "test -f <pid_file>" to "test -r <pid_file>" to accomodate the
non-standard test(1) built-in of Solaris' shell.
Added quotes around variable substitutions.
Multicast(1):
Improved up7_down7_test(1).
6.13.1 2016-06-02T15:29:28-0600
Installation:
Improved examination of system logging daemon by configure(1) script:
Removed check for running daemon to accomodate Docker
Simplified tests for configuration and PID files.
Modified the "make clean" rule so that it doesn't delete the (generated)
LDM tutorial files so that a subsequent "make" won't fail.
Added "StrBuf.h" to BUILT_SOURCES macro in "misc/Makefile.am".
Logging:
log(3):
Created log_is_stderr_useful(), which indicates if the standard
error stream is open on something other than /dev/null.
Eliminated log_reinit() because it's no longer needed.
log2slog(3):
Made initialization more robust.
Corrected algorithm that vets the facility code.
Eliminated printing of trailing newline in messages.
Made the setting of the destination for log messages more atomic.
ulogger(1):
Corrected interpretation of priority levels by making the map from
name to logging level distinct from the one in
</usr/include/sys/syslog.h>.
Product-Queue (pq.c):
Made product-queue file descriptor close-on-exec.
Added property that is the count of outstanding reservations from
uncompleted pqe_new() and pqe_newDirect() calls and the function
pqe_get_count() to read it.
ldmd(1):
Moved configuring of standard I/O file descriptors from EXEC action of
the configuration-file module to near where ldmd(1) daemonizes itself.
pqact(1):
Made the configuration of the standard output and standard error streams
more robust in preparation for execution of child processes.
pqing(1):
Corrected bug introduced in 6.13.0 that didn't honor the argument of the
"-q" option.
NOAAPort:
noaaportIngester(1):
Merged Raytheon's correction to the use of zlib(3). When option
"-c" wasn't specified, the OCONUS imagery received over the
NOAAPORT_OPT channel wasn't processed correctly: the zlib stream
was not reset before the beginning of the new product resulting
in Z_STREAM_ERROR.
nbs_ingest(1) (not yet ready for use):
Use "Option Field Length" parameter in product-definition header for
length of product-specific header instead of remaining bytes in
frame.
Added debugging statements.
Misc:
Replaced the use of clock_gettime() with gettimeofday() because the
former doesn't exist on one client's OS X system (at least) and it
appears to be supported on fewer platforms than the latter. Also, while
clock_gettime() can have nanosecond resolution, the microsecond
resolution of gettimeofday() is sufficient for the LDM.
Created open_on_dev_null_if_closed().
Added unit-test of timestamp module.
Corrected timeval_normalize() (which is currently unused).
Added "#include <stddef.h>" to "noaaport/fifo.hin" for `size_t`.
Removed minor lint discovered by Coverity.
Testing:
Shortened run-time of pq_test(1).
Documentation:
Updated version-independent HTML files.
Multicast (not yet ready for use):
mcast_lib(3):
Changed use of SendAppNotifier and RecvAppNotifier to SendProxy and
RecvProxy, respectively.
FMTP:
Changed name from "vcmtp" to "FMTP" in LDM codebase.
Ported TcpSend::updatePathMTU() to SunOS 5.10.
6.13.0 2016-05-03T16:03:52-0600
Logging:
Re-wrote LDM logging to eliminate the dependency on the system logging
daemon because:
1) It was the major impediment to correctly using the LDM (different
logging daemons, different configuration-file formats, different
security considerations, etc.);
2) It was the major impediment to putting the LDM in a Docker
container;
3) Only 7 "local" syslog(8) facilities are guaranteed and NOAAPort
ingestion could use more; and
4) Some system logging daemons would drop LDM log messages depending
on how heavily they were loaded.
For backward-compatibility, the original logging component, "ulog",
still exists and is unconditionally included in the LDM library,
"libldm". Indeed, the original behavior of the LDM can be obtained by
specifying the option "--with-ulog" to the configure(1) script. By
default, however, the new logging component, named "slog" (for "simple
logging"), will be built and used. While "slog" can be told to write to
the system logging daemon, by default it will write directly to the LDM
log file if the process doesn't have a controlling terminal (i.e., is a
daemon) or to the standard error stream if the process does have a
controlling terminal (i.e., is not a daemon). Like the original "ulog"
component, "slog" also supports log file rotation via a
crontab(1)-executed "ldmadmin newlog" command. See the installed HTML
documentation "$HOME/share/doc/ldm/basics/logging-format.html" or the
web page
<http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/logging-format.html>
for more information.
A particular consequence is that logging, by default, will no longer
depend on SELINUX.
NOAAPort Ingestion (noaaportIngester(1), readnoaaport(1)):
Incorporated Raytheon's modifications to noaaportIngester(1) to:
1) Optionally create blank scan lines for missing GINI data; and
2) Optionally always de-compress a compressed data block in a NOAAPort
frame.
Eliminated memory-leaks and SIGSEGV violations from components that read
GEMPAK tables.
Added entries to GEMPAK tables (indicated column positions aren't correct):
File "g2varswmo.tbl":
000 001 008 010 Total precipitation kg m**-2 APCP 0 -0.01
000 001 029 009 Total Snowfall m SNOW 0 -9999.00
000 001 029 010 Total Snowfall m SNOW 0 -9999.00
000 019 028 000 Mtn Wave Turb (eddy disp rate) m**2/3 s**-1 MWTURB 0 -9999.00
File "g2varsncep.tbl":
000 001 225 009 Freezing Rain kg m**-2 FRZL 0 -9999.00
000 001 225 010 Freezing Rain kg m**-2 FRZL 0 -9999.00
000 019 219 000 Turbulence Potential Fcst Index NA TPFI 0 -9999.00
010 000 193 000 Wave Length - WLENG 0 -9999.00
ldmadmin(1):
Reduced overestimation of new queue capacities (i.e., data capacity and
product capacity) by "ldmadmin vetqueuesize" and eliminated computing
new capacities if the data is insufficient.
Modified the "printmetrics" command to use free(1) to obtain
memory-usage metrics if they aren't obtained from top(1).
Corrected "break" in loop to "last".
LDM (ldmd(1)):
Top-level Server:
Added immediate, unsuccessful termination of the top-level LDM
server -- before it forks a daemon version of itself -- if its LDM
configuration-file doesn't specify anything to do (i.e., is
effectively empty).
Removed misleading "Couldn't get command of EXEC process ..." error
message because it was mistakenly applied to upstream LDM processes
and not just EXEC processes.
Upstream LDM Child Process:
Reduced the delivery latency of products with relatively large
inter-product arrival gaps by modifying the RPC calls that LDM-6
makes to send data products in primary mode by switching from
"batched" RPC calls to asynchronous "message-passing" RPC calls.
Specifically, the XDR function to decode the (non-existent) response
was changed from `NULL` to `xdr_void()` in the functions
`notification_6()`, `hereis_6()`, and `blkdata_6()`. Because the
results are highly dependent on the characteristics of the data
flow, you might not notice any difference.
pqinsert(1):
Corrected test of mmap() return-value to eliminate crashing if a
directory was specified on the command-line.
pqact(1):
Corrected the setting of the product-queue cursor based on the last
successfully-processed data-product from the previous session. Before
this, pqact(1) might miss a product due to a restart of the LDM. This
was most apparent in processing NEXRAD-2 data-products. Thanks Ryan.
Demoted uerror() message in spipe_prodput() to log_debug().
pqing(1):
Accepted Bill Gerry's additions of higher baud rates.
Added support for decoding the length of GRIB-2 data-products and
improved algorithm for decoding the length of GRIB-1 messages.
Product-Queue (pq(3)):
Added mutex destruction to freeing.
Modified fb_get() to log `fblk` diagnostics when it fails.
Improved assertion of relationship between `fblk` arena size and actual
used space.
Modified tq_add() to error-return if it can't get an `fblk` because
there are too many products in the queue.
Modified rpqe_new() to return the product-queue to a consistent state
if an error occurs.
Used correct product size in pqe_insert() and pq_insertNoSig().
Refactored pqe_insert() for better error-handling and return status.
Harmonized return code names by adding PQ_DUP and PQ_BIG and making
PQUEUE_DUP and PQUEUE_BIG aliases.
Hardened the product-queue module against concurrent access by multiple
threads by:
1) Adding a new (undocumented) option (`PQ_THREADSAFE`) to the
`pq_create()` and `pq_open()` functions;
2) Replacing the use of the thread-unsafe function `random()` with
the thread-safe function `nrand48()`; and
3) Removing the (undocumented) ability to reset the
pseudo-random-number generator used in the skip-list
implementation.
inetutil(3):
Modified sa_getInetSockAddr() to try AF_UNSPEC first, then AF_INET.
ldmprint(3):
Added missing space between the timestamp and feedtype.
Installation:
Stopped "make distclean" from removing "*.log" files so that the prior
invocation of the configure(1)-script can be seen in the file
"config.log".
Improved test of netstat(1) options by the configure-script.
Replaced UD_SEARCH_HEADER and UD_SEARCH_LIB with AC_ macros, improved
library searching, and modified makefiles accordingly.
Improved the setting of header file and library references by the
configure script -- in particular, `/opt/csw` is favored over `/usr`
because the Solaris XML2 library appears to have a bug.
Modified script ensureSyslogEntry(1) to accommodate Solaris's grep(1),
which doesn't support the standard "-F" or "-q" options.
Documentation:
Improved documentation based on latest workshop.
Testing:
Added "check-local" make(1)-target for ldmd(1).
Continuous-Delivery:
Corrected setting of configure(1) options in "delivery/deploy.sh".
6.12.14 2015-06-09T09:45:43-0600
Logging:
Corrected bug dealing with variadic arguments that could cause a crash
under certain circumstances.
pq_test_insert(1):
Created test utility for inserting synthetic data-products into the
product-queue.
6.12.13 2015-05-29T15:42:39-0600
Logging:
Rewrote many of the formatting functions in "ldmprint.c" in an attempt
to eliminate the cause of a crash experienced by Art Person when a
downstream's LDM request was reduced due to applicable ALLOW entries.
6.12.12 2015-05-27T08:45:53-0600
scour(1):
Corrected setting of PATH.
6.12.11 2015-05-21T15:32:07-0600
pqact(1):
Corrected a bug introduced in version 6.12.10 in the logic regarding the
setting of the last successfully-processed data-product and the return
value of processProduct(). The bug manifested itself in version 6.12.10
by not writing products to disk under certain circumstances. Thanks
Gilbert.
6.12.10 2015-05-16T08:42:51-0600
Product-Queue (pq.c):
Corrected a bug that dealt with finding the last data-product that a
downstream LDM successfully received given the product's signature. This
bug manifested itself as either a segmentation violation (i.e., SIGSEGV)
or 100% CPU usage -- both in the function `tqe_find()`. Thanks Gilbert.
pqact(1):
Modified pqact(1) to correct the setting of the last
successfully-processed data-product when the LDM exits. Before, the last
data-product that matched the `-f feed` and `-p pattern` options would
be used regardless of whether or not it matched any entries in the
pqact(1) configuration-file. Now, a data-product will be used if and
only if it's successfully processed by an entry in the pqact(1)
configuration-file. Thanks Daryl.
Logging (ulog.c, log.c):
Added signal blocking to the <log.h> implementation because that module
calls async-unsafe functions and could be called by a signal handler.
Misc:
Removed considerable lint identified by Coverity Scan. Some of it was
significant. See <https://scan.coverity.com/projects/4645>.
6.12.9 2015-03-16T11:11:30-0600
Added to the configure(1) script, support for Solaris' non-standard grep(1),
which doesn't have the "-q" (quiet) option.
6.12.8 2015-03-13T15:45:05-0600
Installation:
Modified to hopefully prevent installation subdirectories/links from
being created in the home directory of an LDM developer executing the
command `make distcheck`. This only affects LDM developers.
NOAAPORT (noaaportIngester):
Corrected code in the AWIPS-II retransmission-request layer that relied
on undefined behavior in formatting log messages. This only affects NWS
offices.
6.12.7 2015-03-06T09:39:26-0700
LDM Server (ldmd):
If the LDM configuration-file has no ALLOW or ACCEPT entries, then the
top-level LDM process will no longer listen for incoming connections.
This change makes it easier to run downstream-only LDM instances that
don't interfere with an LDM server on the same computer.
pqact(1) (filel.c):
Refactored and generally improved the code in an attempt to stop
pqact(1) from receiving a SIGSEGV when executing regexec(3). This has
only occurred on one host and it's not clear if it's due to the code or
the host.
Product Queue (pq.c):
Corrected bug introduced in LDM 6.12.4 that caused the command "ldmadmin
vetqueuesize" to work incorrectly: the queue size parameter or maximum
acceptible latency parameter would *not* be adjusted if the registry
parameter "/reconciliation-mode" was set to "increase queue" or
"decrease maximum latency", respectively.
NOAAPORT (noaaportIngester):
Corrected a bug in getting GRIB2 vertical coordinate table in GEMPAK
library. The routines to get the WMO and local tables cleared the extant
table if a new table needed to be read. They then forgot that they did
this if reading-in the new table failed.
Added GEMPAK GRIB2 table "g2varsnssl.tbl" and installation of the
symbolic link "g2varsnssl1.tbl".
Improved the performance of finding GRIB2 parameter information by
caching the GEMPAK parameter tables.
Improved the error-message logged when information on a GRIB2 parameter
or vertical coordinate can not be found in the relevant GEMPAK table.
Added entries to the GEMPAK GRIB2 NCEP parameter table.
Got the option "-I <ifaceIpAddr>" working. A process can now be made to
receive from one interface only.
Added printing of WMO header of GRIB2 data-products with unknown
parameters.
Slightly improved the performance of reading multicast input.
Added attempt to lock process in physical memory to increase
performance. The program will continue if this fails.
Added to the statistics report, instances when the data input thread
had to wait until sufficient space was available in the internal FIFO
queue. Ideally, no such instances should occur when reading NOAAPORT
multicast input. The statistics report is logged when the process
terminates or receives a SIGUSR1. Reporting statistics might interfere
with realtime data ingest.
Merged changes to the retransmit-request capability from Raytheon for
the AWIPS-II system. This capability can only be used by NOAA's weather
forecast offices.
Added `make check` using multi-radar/multi-sensor (MRMS) data from NSSL.
Installation:
Reduced the likelyhood that the script that modifies the system logging
daemon's configuration-file would add an effectively duplicate entry
for the LDM.
Added support for running rsyslogd(8) on a Solaris system with
configuration-file "/etc/opt/csw/rsyslog.conf" and PID-file
"/var/run/rsyslogd.pid".
Misc:
Improved internal documentation.
Added finding doxygen(1) to configure script.
6.12.6 2014-09-04T13:56:41-0600
pqcreate(1):
Changed the assumed mean product size (which is used to compute the
default number of slots in the product-queue) from 4096 to 51000
bytes to better match current reality.
NOAAPORT:
Enhanced GRIB2 decoding by adding "ESTOFS" and "HRRR" model codes for
the NCEP center.
6.12.5 2014-08-21T13:57:50-0600
Installation:
Added (untested) support for syslog-ng(8) as the system logging daemon
rather than rsyslog(8) or syslog(8).
NOAAPORT:
Added port numbers for the two new channels that are part of the
NOAAPORT SBN upgrade.
Documentation:
Removed from the HTML documentation, links to the web pages that listed
successful and unsuccessful build environments because the information
was hopelessly out of date, difficult to maintain, and labor intensive.
6.12.4 2014-07-29T17:04:22-0600
Product-queue library (pq.c):
Modified the product-insertion logic to continue rather than
error-return (and, consequently, terminate the downstream LDM process)
if the signature of a product that's being deleted to make room for the
new product can't be found in the signature-list. An error-message will
be logged instead. This was done because we've seen this occur at least
three times over the past couple of years -- resulting in the LDM
ceasing to receive data. Our working hypothesis is that a bit (either
on-disk or in-memory) was somehow corrupted.
6.12.3 2014-06-27T16:12:47-0600
Fixed a bug in the upstream LDM database that prevented disabling of the
anti-DOS feature from taking effect and telling the user why.
6.12.2 2014-06-26T17:49:53-0600
Fixed a bug in the extraction of function declarations from C source files
into header files that caused the function `getSysConfDirPath()` to be
undeclared and, consequently, to have its `char*` return value interpreted
as an `int` -- with possibly catastrophic consequences on 64-bit systems.
6.12.1 2014-06-24T09:50:11-0600
LDM registry:
Fixed a bug in the decoding of boolean values that was introduced in
the previous version.
LDM server, ldmd(1):
The ability to disable the anti-denial-of-service (anti-DOS) feature
should now work.
6.12.0 2014-06-19T16:11:23-0600
Installation:
Improved robustness:
* Check for CUnit package by configure(1) script;
* Creation of "runtime" symbolic link;
* Non-creation of LDM tutorial.
Now requires that the C compiler supports the C99 standard (GNU's
gcc(1) does, as should any c99(1) program).
LDM server, ldmd(1):
Added the ability to disable the anti-denial-of-service (anti-DOS)
feature of upstream LDM-s. The anti-DOS feature ensures non-overlapping
feeds to every downstream host by reducing subscription requests from
downstream LDM-s running on the same host and terminating duplicate
upstream LDM proceses. This action can cause problems if multiple
downstream hosts are behind a NAT device or are connecting via ssh(1)
tunnels because the downstream host will appear to be the same. The
anti-DOS feature is enabled by default. To disable it, set the value of
the registry parameter "/server/enable-anti-DOS" to "FALSE" via the
command
regutil -b FALSE /server/enable-anti-DOS
The anti-DOS feature should only be disabled by sites that know and
trust *all* downstream LDM-s that can feed from it. Note that accidental
denial-of-service attacks have occurred from trusted downstream sites
due to incorrect network configuration.
Here's your rope, now go hang yourself. :-)
NOAAPORT Ingestion:
Incorporated Raytheon's support for extracting NOAA Weather Wire
Service (NWWS) products from the broadcast (basically a new numbered
datastream).
6.11.7 2014-04-04T15:37:47-0600
Downstream LDM:
Squashed a bug that could result in an assertion failure if a remote
site's portmapper was successfully used to connect to a remote LDM
server (thanks Gilbert).
NOAAPORT Ingestion:
noaaportIngester(1):
Incorporated Ratheon's latest modifications to support
retransmission requests. This feature can only be used by NWS sites
-- so don't even think about it.
Improved the code in the NOAAPORT, GRIB-2, and GEMPAK libraries to
reduce the chance a segmentation violation (SIGSEGV).
notifyme(1):
Squashed a bug in the LDM 5 client library (which is used by
notifyme(1)) that would cause a segmentation violation (SIGSEGV) if the
upstream hostname couldn't be resolved into an IP address.
Documentation:
Corrected URL for the IDD.
Added missing manual-page precursor files to the source distribution.
Misc:
Implemented a continuous-delivery pipeline for the project.
6.11.6 2013-06-24 17:04:55-06:00
SUMMARY OF SIGNIFICANT CHANGES:
* Eliminated the bundled PNG library;
* Squashed bugs;
* Improved the portability, release-engineering, and installation
process;
* Created a binary RPM for fc14.x86_64.
Installation:
Configure(1) Script:
Made the search for the PNG library dependent on the
"--with-noaaport" option.
Improved the search for the XML2 and ZLIB header-files and
libraries. In particular, added "/usr/local/lib", "/opt/lib", and
"/user/sfw/include" for general searches.
Adapted to older tr(1)-s that don't extend the second character
set.
Mac OS/X Support:
Added definition of _DARWIN_C_SOURCE to have "struct ip_mreq"
defined on that system (that definition is needed by the NOAAPORT
module).
Don't define "union semun" on that system because it's defined by a
system header-file.
Added "opt/X11" to library search-path.
SunOS Support:
Removed SunOS 5.6 and 5.9 from portability-testing platforms
because they're end-of-life.
Added testing on other SunOS environments.
Misc:
Removed the bundled ZLIB package. This package's library was in the
LDM shared-library and, on some systems, it conflicted with the
system ZLIB library referenced by the system XML2 library (which was
removed from the LDM in version 6.11.5).
Created a relocatable binary RPM for fc14.x86_64.
Made the creation of the contents of the ~/var directory more
robust when relevant symbolic links already exist.
Changed the default installation-directory from "/usr/local/ldm"
to "/opt/ldm" following the Filesystem Hierarchy Standards. This
is the default for the RPM installation. Use the "--prefix=..."
option of rpm(1) if you want a different home-directory for the
LDM user.
Eliminated the creation of the "data" and "logs" symbolic links in
the LDM user's home-directory. If they exist, however, then they
won't be removed.
Removed the bundled "libpng" package.
ldmadmin(1):
Added verbose logging to the check of the LDM configuration-file by
the "start" command.
LDM server (ldmd):
Squashed a bug that caused a segmentation violation (SIGSEGV) on 64-bit
Darwin.
Eliminated a minor memory leak in the parsing of LDM configuration-file
when the "-n" (do nothing) option is specified.
Documented the "-n" (do nothing) option.
Logging:
Added setting of the SELINUX security context for the LDM log
directory. This should allow the LDM to log when SELINUX is enabled.
Squashed a bug that caused the log(3) module to drop a long new message.
pqinsert(1):
Squashed a bug that caused a segmentation violation (SIGSEGV) when given
an unknown option.
regutil(1):
Made the "usage" message use the LDM logging facility.
Added the "-v|-x" options and some debug-level messages.
NOAAPORT:
Enhanced the verification of the structure of a GRIB2 message.
Adapted code to the opaque "struct png_struct" of PNG 1.5 and later.
Corrected the inclusion of the "png.h" header-file.
Documentation:
HTML:
Documented the RPM installation.
Documented the dependency on the PNG package if the NOAAPORT
system is built.
Documented that the XML2 development and PNG development packages
(as opposed to the runtime packages) are necessary.
Misc:
Replaced the license with the BSD 3-Clause license enhanced with
patent protection.
6.11.5 2013-05-10 14:42:19-06:00
Summary:
Elimination of the bundled "libxml2" package and improvements due to 1)
vetting by valgrind(1); and 2) investigating and testing the Upstream
LDM Database (ULDB).
libxml2:
Eliminated this package from the LDM distribution. This was done because
1) the libxml2 package has become ubiquitous -- it's on all platforms to
which I have access; 2) bundling it with the LDM distribution means that
an upgraded, more secure version can't be used; and 3) eliminating it
approximately halves the size of the distribution and the time-to-build.
Upstream LDM:
Added printing of the subscription (the request by the downstream LDM)
when filtering by the upstream LDM disallows it.
Corrected the error-message when unable to open the product-queue.
Access Control List (src/protocol/acl.c):
Improved layering by abstracting the linked-list of requests.
ULDB (Upstream LDM Database in src/uldb/):
Added blocking of signals to the public functions of the ULDB module
that modify the database.
Improved internal layering.
Improved valgrind(1)-using unit-test.
These changes were made because a 6.11.4 LDM appeared to have received
a SIGSEGV in the ULDB module (support ticket JVF-237504). Consequently,
the unit-test of that module was improved and a robustness test was
added (about 3000 upstream processes in 5 seconds).
Unfortunately, none of these changes appear to have been necessary: the
ULDB module passes the improved unit-tests even without the changes.
Semaphore-based Read/Write Lock (src/misc/semRWLock.c):
Corrected the order of invalidating and freeing the lock structure as
part of the ULDB investigation. Didn't make a difference.
LDM Registry (src/registry/):
Ensured that free() isn't called twice on the same file object, which
could result in a SIGSEGV.
Improved some internal documentation.
NOAAPORT:
Corrected error-message when unable to open the product-queue.
Documentation:
Added "libxml2" as a platform prerequisite.
Release Engineering:
Improved the "release" target in the top-level makefile.
Re-enabled "make distcheck".
Improved the "valgrind" target for ldmd(8). Tried to add an
"installcheck-local" target, but valgrind(1) won't execute setuid
programs that are owned by root -- even adding a wrapper script doesn't
do what's needed.
Removed "libdict" subdirectory because it wasn't used.
6.11.4 2013-04-03 14:22:34-06:00
Installation:
Added $(DESTDIR) to "ensure-ldmhome-links" rule in the makefile.
ldmd(1):
Made the "Exiting" log message an unconditional NOTICE.
Eliminated redundant and dangerous free_prod_class() before
set_prod_class() in the LDM-5 module.
Improved places where an LDM process might terminate.
Made establishing a connection more efficient.
Decreased timeout if a forked server doesn't receive anything from 12
minutes to 1 minute.
Upstream LDM:
Changed policy on duplicate incoming requests from 6.11.3 to
terminate the earlier upstream LDM rather than current upstream
LDM.
Corrected PID in OK reply to version 6 FEEDME or NOTIFYME request.
Reduced resource usage by eliminating duplication of upstream socket.
Corrected time amount in "silent client" log message.
Documented log message about terminating an obsolete upstream LDM
process.
Eliminated some closings of the connection by the upstream LDM to
favor closing by the downstream LDM in order to avoid the upstream
LDM going into the TCP TIME_WAIT state.
Improved handling of errors.
Demoted logging-level of transmission failure to INFO because this
is expected when the downstream LDM switches its transfer-mode
between PRIMARY and ALTERNATE.
Documented log messages associated with subscription reduction by
upstream LDM.
Simplified waiting for next data-product (sleep either 0 or 60
seconds).
Downstream LDM:
Assume upstream LDM is alive unless explicitly stated otherwise by
the upstream LDM server.
Convert status of a time-out due to death of upstream LDM from a
disconnect into a time-out.
Added PID of upstream LDM to error-messages.
Enabled logging of LDM-6 request errors.
Removed duplication of downstream socket.
Made the autoshifting default "don't switch" in order to correctly
handle out-of-the-blue HIYA messages.
Improved decision on when a downstream LDM would change its
transfer-mode.
ldmsend(1):
Added flushing of connection when all files are sent.
protocol/child_process_set.c:
Simplified some functions and added internal documentation.
NOAAPORT Ingester:
Allowed tabs in textual products.
pqact(1):
Improved clarity of buffer-toggling code in processProduct().
rtstats(1):
Added "-H <hostname>" option to rtstats(1) program.
Added test for disallowed operands.
Added gethostbyname(3) to "See Also" section of manual-page.
RPM generation (still don't know if it works):
Improved support:
requtil(1) makefile:
Made rules involving ETC_DIR honor DESTDIR.
Corrected definition of ETC_DIR.
ldm.spec.in:
Added DESTDIR=%{buildroot} to "make" invocations.
Misc:
Improved some log messages.
Improved some internal documentation.
Made some cosmetic changes.
6.11.3 2013-02-12 13:11:20-07:00