1.4

What's Changed

• Bump to version 1.1 by @JulienPeloton in #429
• Add knscore in HBase by @JulienPeloton in #431
• Enable kilonova filter! by @JulienPeloton in #433
• Update final classification with KN candidates by @JulienPeloton in #435
• Fix namespace problem by @JulienPeloton in #437
• Update index table with new classification by @JulienPeloton in #439
• push TNS confirmed data by @JulienPeloton in #440
• Update classification for SSO and microlensing by @JulienPeloton in #442
• Add c column before distribution by @JulienPeloton in #445
• Update the script to push candidates to TNS with the new header by @JulienPeloton in #449
• Remove incoming duplicate based on candid by @JulienPeloton in #450
• Built map at nside=4096 for the xmatch by @JulienPeloton in #452
• Reduce the time frame to classify alerts as SN by @JulienPeloton in #456
• Update HBase version to 2.2.7 by @JulienPeloton in #469
• Remove unused folder for deprecated web interface by @JulienPeloton in #467
• Test the new pixels table by @JulienPeloton in #466
• Create individual index tables for conesearch and xmatch by @JulienPeloton in #473
• Add tracklet identification by @JulienPeloton in #478
• Bump to 1.2 by @JulienPeloton in #492
• Release the magdiff cut by @JulienPeloton in #493
• Bump to 1.3 by @JulienPeloton in #495
• New binary to produce statistics by @JulienPeloton in #498
• Update Fink configuration by @JulienPeloton in #508
• Conf files for the cloud by @JulienPeloton in #512
• Replace classification by new location by @JulienPeloton in #513
• Update path for daily stats, and increase test coverage by @JulienPeloton in #515
• Renaming fields for Active Learning and kilonova modules by @JulienPeloton in #517
• Update the science module with recent updates from fink-science by @JulienPeloton in #520
• [Science module] Update with new mulens module by @JulienPeloton in #523

Full Changelog: 1.1...1.4

1.1

1.0 to 1.1

https://github.com/astrolabsoftware/fink-broker/milestone/3?closed=1

1.0

0.7.0 to 1.0

• [Dependencies] Install torch only with CPU support (#398)
• [HBase] Scan along both time and objectId axes (#402)
• [TNS] Push Early SN candidates to TNS nightly (#404)
• [Schema] Add Fink distribution schema 0.3 (#406)
• [Distribution] Add new script to automatically save schema (#407)
• [Schedule] Towards a new Broker scheduler: nightly jobs (#410)
• [CI] Switch to GH Actions (#415)

0.7.0

0.6.0 to 0.7.0

• [HBase] structure change (#388)
• Broadcast microlensing models to executors (#391)
• Allow easy data reprocessing (#392)
• [Science] Update science part according to fink-science 0.3.6 (SuperNNova, asteroids, nalerthist) (#394)

0.6.0

0.5.0 -> 0.6.0

Database

• New partitioning strategy (#363)
• [Minor] Add night date in the merge job name. (#371)
• [Merge] Use a fix basepath and add YYYY in the partition tree (#373)
• Remove topic column, and rename ts_jd into timestamp (#377)
• [Parquet] Add library versions to the data (#382)

Stream2raw

• Decode simulated data and ZTF data simultaneously (#367)
• Add support for kerberised Kafka (#369)

Science module & filters

• [Science module] Microlensing (#358)
• Update raw2science with new fink-science release (#375)
• Add the asteroid catcher module (#379)

Science portal

• [Science Portal] Push a schema row while updating the science portal (#356)
• Remove ra_dec from the rowkey, and add new added values from the broker (#384)

Misc

• Update the README with GSoC 2020 news (#359)
• Add Fink logo in the README (#361)

0.5.0

0.4.0 -> 0.5.0

General

• Add Dockerfile (#315 )

Science module & filters

• Add SN classification based on Random Forest (#319)
• Add new filter snialike (#324)
• Register new topic snialike on the CI (#330)

Science portal

• [Science portal] Ingest science data to HBase (#351)

Schema

• Update alert schemas: ZTF 3.3 and Fink 0.1 (#317)
• Push new Fink schema version in the documentation (#321)
• Note on live vs simulation mode for distribution schema (#338)

Misc

• Bump hbase version to 2.2.3 (#326)
• fix broken link and update news in the README (#332)
• Upgrade Spark versions in the travis configuration script (#335)
• add missing ${EXTRA_SPARK_CONFIG} for the distribution job (#337)
• [Clean] remove unused classification module & bump HBase version (#346)

0.4.0

0.3.0 -> 0.4.0

General

• Externalise the simulator part to fink-alert-simulator (#311).

0.3.0

0.2.1 -> 0.3.0

General

• New Fink command to stop all services (#238)
• Add python logger (#243)
• Push CI notification to the Fink slack channel (#257)
• pyarrow incompatibility (#285)
• Replace HBase by Parquet before redistribution (#298)
• Integrate fink-science and fink-filters and remove science modules from fink-broker (#301)

Alert redistribution

• Enable Authentication and Authorization in Alert Redistribution (#231)
• slack integration (#249 )
• kafka authorization: allow use of wildcard resource '*' (#255 )
• select all columns by default (#269)
• add option to view acl in fink_kafka (#282)
• Allow using custom configuration file in fink_kafka (#283)

Other minor fixes

• several typo and bug fixes:
  • #240, #252, #253, #259, #262, #266, #268, #272, #275, #287, #295, #305, #307

Thanks @cAbhi15, @marcol480, @tallamjr for all your contributions!

0.2.1

0.2.0 -> 0.2.1

General

• Transfer documentation outside of the repo (#223 )
• New service: Fink shell (#227 )

Raw database to science database

• Add capability to manipulate the data (level one) from generic modules (#209)
• Make optional the use of cross-match in raw2science (#222 )
• Add the path to the HBase configuration file when running raw2science (#195 )

Alert redistribution

• Add timestamp-based streaming to Alert distribution service (#201 )
• Filtering alerts before distribution (#206 )
• Add level two generic filter (#216 )

Other minor fixes

• several typo and bug fixes:
  • fix typo in checkstream (#202 )
  • Update gitignore with mgmt files (#211 )
  • Catch TimeoutError and ConnectionError when doing xmatch with CDS (#221 )
  • PEP8 rules (#214 )

Thanks @cAbhi15 for many contributions!

0.2.0

• Print version number from the binary (#132)
• Update the gitignore to stop tracking .coverage* files (#136)
• From raw database to science database (#140)
  • Connect to the raw database
  • Filter alerts based on instrumental or environmental criteria.
  • Classify alerts using the xMatch service at CDS.
• Add Raw database connector (#146)
• Hosting the data for the test suite (#152)
• Build the raw database before the unit test suite (#154)
• Flatten DataFrame columns before sending to the science database (#159)
• From stream to science (#162)
  • The data from the raw database is loaded (streaming), filtered, tagged and finally pushed to the HBase table (using a custom Spark Structured Streaming HBase connector).
  • API break (fink services renamed).
• Modified fink_test to allow any configuration file (#163)
• Install HBase for CI tests (#166)
• Changing the threshold for coverage (#169)
• Re-introduce the monitoring service (checkstream) (#172)
  • Was lost in #162
• New Alert Distribution module (#182)
  • read the science db (Hbase)
  • serialize into avro and publish to Kafka
  • a Kafka consumer to deserialize the avro message and do spark DF operations
• Updating Spark versions in the travis CI script (#183)
• Switch from custom HBase sink provider to foreachBatch mechanism (#184)
• Add a new xmatch tab to the dashboard (#186)
• Bump HBase version for the CI to 2.1.5 (#188)
• Repo management: setup and new structure for the python module (#191)
  • API break (python module path changed)
• Several documentation and README updates (#134, #138, #142, #149, #173, #177)

Thanks to @cAbhi15 and @tallamjr who contribute to this new release!