Skip to content

Commit

Permalink
Squashed 'deps/ccommon/' changes from bb298bc..727905b (#167)
Browse files Browse the repository at this point in the history
727905b adding a new API to duration timer (#152)
9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (#150)
a0aafdf Add missing semicolon to ring array example code in documentation. (#149)
9264bbb Zero byte (#147) (emergency fix needed for pelikan)
d4002d7 simplify cc_print_int64 (#146)
b164fcf Clean-up hash functions and introduce MurmurHash3 (#145)
b1babb2 change wheel's sleep timer to make it less flaky (#143)
ce0b9ea allow printing negative integers in cc_print (#141)
ab0edc8 add metrics to track buf_sock objects (#138)
ae02038 add travis ci (copied from pelikan) (#139)
964645a Merge pull request #135 from paegun/fix_cmake_install
70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion.
5b095bc Merge pull request #126 from kevyang/kevyang/120
426d56a return NULL when cc_alloc/cc_realloc is called with size == 0
ad271d4 Merge pull request #133 from kevyang/132
47dbdba suppress unused parameter warning in debug_log_flush
648d19e Merge pull request #127 from kevyang/56
780941a Merge pull request #130 from kevyang/129
b8af6c0 Merge pull request #131 from kevyang/128
6ecc318 fix duplicate symbols in cc_signal
080c41d cc_array - stop doing arithmetic on void *
d526f7a add debug oriented memory management
a4fb927 Update bool member rules in style guide
05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables

git-subtree-dir: deps/ccommon
git-subtree-split: 727905b
  • Loading branch information
Yao Yue authored May 19, 2018
1 parent 390dab1 commit 1196b58
Show file tree
Hide file tree
Showing 10 changed files with 119 additions and 22 deletions.
57 changes: 45 additions & 12 deletions deps/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
# deps

The deps directory contains all the dependencies shipped as source packages. The project level CMakeLists.txt contains additional dependencies.
The deps directory contains all the dependencies shipped as source packages. The
project level CMakeLists.txt contains additional dependencies.

All packages in this directory are tracked using git subtree. For more information, follow the original [git-subtree documentation](https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt)
All packages in this directory are tracked using git subtree. For more
information, follow the original [git-subtree documentation](https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt)

## List of dependencies
* ccommon(libccommon): a library providing generic, low-level functionalities useful for writing a RPC service.
* ccommon(libccommon): a library providing generic, low-level functionalities
useful for writing a RPC service.

## ccommon

Expand All @@ -17,16 +20,23 @@ git remote add ccommon_remote [email protected]:twitter/ccommon.git
The first time we merge ccommon into deps, the following command was executed.
```bash
git checkout master
git subtree add --prefix=deps/ccommon/ ccommon_remote master --squash
git subtree add --prefix=deps/ccommon ccommon_remote master --squash
```

To update ccommon with upstream/remote changes
```bash
git fetch ccommon_remote master
git subtree pull --prefix=deps/ccommon/ --squash ccommon_remote master
git subtree pull --prefix=deps/ccommon --squash ccommon_remote master
```

To update upstream/remote with local changes involves somewhat complicated commands. At a high level, this is done in two steps: first, the local history needs to be sifted to isolate changes that are relevant to the subtree (deps/ccommon in our case), and an alternative "timeline" or history suitable for committing to the remote is created; second, this alternative history is pushed back to the remote. See Notes and subtree's github repo for more information.
To update upstream/remote with local changes involves somewhat complicated
commands. At a high level, this is done in two steps: first, the local history
needs to be sifted to isolate changes that are relevant to the subtree
(`deps/ccommon` in our case), and an alternative "timeline" or history suitable
for committing to the remote is created; second, this alternative history is
pushed back to the remote. See Notes and subtree's github repo for more
information.

```bash
# first find out the last SHA of a merge from upstream, in this example it is a06437
git subtree split --prefix=deps/ccommon --annotate='ccommon: ' a064371781e7fa4be044b80353dde9014353d6a5^.. -b ccommon_update
Expand All @@ -38,14 +48,37 @@ git push ccommon_remote ccommon_update:master
# Notes

## Why subtree?
There are two goals we try to achieve: a) manage dependencies explicitly; b) make the repo easy to use for most people, not just main developers.
The dependency management went through three phases: no management at all (two free standing repos); use of git submodule; use of git subtree. It was very clear that no dependency management was absolutely unacceptable, build broke all the time. Git submodule is the first thing we tried, and it is fairly easy to make changes within the submodule and merge the changes back to upstream. However, since submodules require extra options to be fully checked out or updated, it becomes harder to use especially for people unfamiliar with the project.
There are two goals we try to achieve: a) manage dependencies explicitly;
b) make the repo easy to use for most people, not just main developers.
The dependency management went through three phases: no management at all (two
free standing repos); use of git submodule; use of git subtree. It was very
clear that no dependency management was absolutely unacceptable, build broke all
the time. Git submodule is the first thing we tried, and it is fairly easy to
make changes within the submodule and merge the changes back to upstream.
However, since submodules require extra options to be fully checked out or
updated, it becomes harder to use especially for people unfamiliar with the
project.

Git subtree actually means two things: the subtree mechanism that manages sub-repository using native git commands, and the git subtree extension, which is mostly a bash script. With the former, it is rather difficult to merge changes in the sub-repository upstream. This is handled by the `split` command in the git-subtree extension, which makes it feasible. There simply isn't an elegant solution out there for our purpose, and subtree seems to be the best compromise so far given our goals.
Git subtree actually means two things: the subtree mechanism that manages
sub-repository using native git commands, and the git subtree extension, which
is mostly a bash script. With the former, it is rather difficult to merge
changes in the sub-repository upstream. This is handled by the `split` command
in the git-subtree extension, which makes it feasible. There simply isn't an
elegant solution out there for our purpose, and subtree seems to be the best
compromise so far given our goals.

## Merge direction
As a result of the complexity of merging to upstream, we prefer working on dependencies in their own repo, and update the parent repo by pulling. Merging upstream is possible but not encouraged.
As a result of the complexity of merging to upstream, we prefer working on
dependencies in their own repo, and update the parent repo by pulling. Merging
upstream is possible but not encouraged.

## Handle git history
We decide to use `--squash` when merging to keep most of the dependency history out of the parent project. This cuts down noise in the parent repo.
We also choose not to use the `--rejoin` option of git-subtree, because this option creates an extra entry in git log to mark the current split location. This also introduces noise into git history that means little to those who don't use subtree. Without this option, `git subtree split` by default scans the entire history of the parent project, which can take a while if there has been a long history. To speed things up, one can provide a range of SHA to scan from, which usually means git only has to go through the last few changes.
We decide to use `--squash` when merging to keep most of the dependency history
out of the parent project. This cuts down noise in the parent repo.
We also choose not to use the `--rejoin` option of git-subtree, because this
option creates an extra entry in git log to mark the current split location.
This also introduces noise into git history that means little to those who don't
use subtree. Without this option, `git subtree split` by default scans the
entire history of the parent project, which can take a while if there has been a
long history. To speed things up, one can provide a range of SHA to scan from,
which usually means git only has to go through the last few changes.
2 changes: 1 addition & 1 deletion deps/ccommon/docs/modules/cc_ring_array.rst
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Hello World! with ccommon ``ring_array``:
if (status != CC_OK) {
printf("Could not pop entire message!");
exit(1)
exit(1);
}
printf("%c", c);
Expand Down
8 changes: 8 additions & 0 deletions deps/ccommon/include/cc_mm.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

#pragma once

#ifdef __cplusplus
extern "C" {
#endif

#include <cc_define.h>

#include <stddef.h>
Expand Down Expand Up @@ -70,3 +74,7 @@ void * _cc_realloc_move(void *ptr, size_t size, const char *name, int line);
void _cc_free(void *ptr, const char *name, int line);
void * _cc_mmap(size_t size, const char *name, int line);
int _cc_munmap(void *p, size_t size, const char *name, int line);

#ifdef __cplusplus
}
#endif
8 changes: 8 additions & 0 deletions deps/ccommon/include/cc_rbuf.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@

#pragma once

#ifdef __cplusplus
extern "C" {
#endif

#include <cc_metric.h>

#include <stdint.h>
Expand Down Expand Up @@ -86,3 +90,7 @@ size_t rbuf_wcap(struct rbuf *buf);
size_t rbuf_read(void *dst, struct rbuf *src, size_t n);
/* write from a buffer in memory to the rbuf */
size_t rbuf_write(struct rbuf *dst, void *src, size_t n);

#ifdef __cplusplus
}
#endif
9 changes: 9 additions & 0 deletions deps/ccommon/include/cc_ring_array.h
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@

#pragma once

#ifdef __cplusplus
extern "C" {
#endif

#include <cc_define.h>

#include <stddef.h>
Expand Down Expand Up @@ -53,4 +57,9 @@ rstatus_i ring_array_pop(void *elem, struct ring_array *arr);

/* creation/destruction */
struct ring_array *ring_array_create(size_t elem_size, uint32_t cap);

void ring_array_destroy(struct ring_array *arr);

#ifdef __cplusplus
}
#endif
18 changes: 13 additions & 5 deletions deps/ccommon/include/hash/cc_murmur3.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@
*/

/**
* The cc_murmur3.[ch] are adapated from the canonical implementation of
* The cc_murmur3.[ch] are adapted from the canonical implementation of
* MurmurHash3 by Austin Appleby, released as part of SMHasher:
* https://github.com/aappleby/smhasher
*
* Changes include renaming fuctions, removing MSVC-related code, adding "static"
* Changes include renaming functions, removing MSVC-related code, adding "static"
* keyword to local-scope functions according to C language spec (original code is
* in C++), to better fit them into the scope and style of ccommon
*
Expand All @@ -29,11 +29,19 @@

#pragma once

#ifdef __cplusplus
extern "C" {
#endif

#include <stdint.h>


void hash_murmur3_32 ( const void * key, int len, uint32_t seed, void * out );
void hash_murmur3_32(const void *key, int len, uint32_t seed, void *out);

void hash_murmur3_128_x86(const void *key, int len, uint32_t seed, void *out);

void hash_murmur3_128_x86 ( const void * key, int len, uint32_t seed, void * out );
void hash_murmur3_128_x64(const void *key, int len, uint32_t seed, void *out);

void hash_murmur3_128_x64 ( const void * key, int len, uint32_t seed, void * out );
#ifdef __cplusplus
}
#endif
2 changes: 2 additions & 0 deletions deps/ccommon/include/time/cc_timer.h
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ struct timeout {

/* update duration */
void duration_reset(struct duration *d);
/* get a reading of duration and copy it without stopping the original timer */
void duration_snapshot(struct duration *s, const struct duration *d);
void duration_start(struct duration *d);
void duration_stop(struct duration *d);
/* read duration */
Expand Down
11 changes: 11 additions & 0 deletions deps/ccommon/src/time/cc_timer_darwin.c
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,17 @@ duration_reset(struct duration *d)
d->stop = 0;
}

void
duration_snapshot(struct duration *s, const struct duration *d)
{
ASSERT(s != 0 && d != NULL);

s->started = true;
s->start = d->start;
s->stopped = true;
s->stop = mach_absolute_time();
}

void
duration_start(struct duration *d)
{
Expand Down
11 changes: 11 additions & 0 deletions deps/ccommon/src/time/cc_timer_linux.c
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,17 @@ duration_start(struct duration *d)
d->started = true;
}

void
duration_snapshot(struct duration *s, const struct duration *d)
{
ASSERT(s != 0 && d != NULL);

s->started = true;
s->start = d->start;
s->stopped = true;
_gettime(&s->stop);
}

void
duration_stop(struct duration *d)
{
Expand Down
15 changes: 11 additions & 4 deletions deps/ccommon/test/time/timer/check_timer.c
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,25 @@ START_TEST(test_duration)
{
#define DURATION_NS 100000

struct duration d;
double d_ns, d_us, d_ms, d_sec;
struct duration d, s;
double d_ns, d_us, d_ms, d_sec, s_ns;
struct timespec ts = (struct timespec){0, DURATION_NS};

duration_reset(&d);
duration_start(&d);
nanosleep(&ts, NULL);
duration_snapshot(&s, &d);

/* snapshot is as expected */
s_ns = duration_ns(&s);
ck_assert_uint_ge((unsigned int)s_ns, DURATION_NS);

nanosleep(&ts, NULL);
duration_stop(&d);

/* duration is as expected */
/* final duration is as expected */
d_ns = duration_ns(&d);
ck_assert_uint_ge((unsigned int)d_ns, DURATION_NS);
ck_assert_uint_ge((unsigned int)d_ns, 2 * DURATION_NS);

/* readings of different units are consistent */
d_us = duration_us(&d);
Expand Down

0 comments on commit 1196b58

Please sign in to comment.