Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add timestamps parameter #126

Merged
merged 47 commits into from
Nov 28, 2023
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
219337b
add incomplete --timestamp parameter
fahimahmedx Nov 13, 2023
e7d5b1d
create interface for timestamp_to_block function
fahimahmedx Nov 15, 2023
8c15167
add get_latest_block_number() function
fahimahmedx Nov 15, 2023
ab2ed6b
Add timestamp_to_block_number() function
fahimahmedx Nov 15, 2023
ab0150d
Use Strings for timestamp
fahimahmedx Nov 15, 2023
31841c0
change timestamp argument to Vec<String> part 2
fahimahmedx Nov 16, 2023
273d386
use reference, fix get_latest_block_number()
fahimahmedx Nov 16, 2023
aff9e4e
add parse_timestamp() functions
fahimahmedx Nov 16, 2023
1b73742
make some block processing functions public
fahimahmedx Nov 16, 2023
55031cc
cargo fmt
fahimahmedx Nov 16, 2023
c9f069d
add docs
fahimahmedx Nov 16, 2023
c37000e
add support for "every nth value" syntax
fahimahmedx Nov 16, 2023
c79dd22
implement parse_timestamp_range_to_block_number_range()
fahimahmedx Nov 18, 2023
4e5551e
remove unused array in timestamp_to_block_numbers()
fahimahmedx Nov 18, 2023
aa5ae9a
fix binary search in timestamp_to_block_number()
fahimahmedx Nov 18, 2023
cd06ec7
cargo fmt
fahimahmedx Nov 18, 2023
fcd21a8
initial test for timestamp_to_block_number()
fahimahmedx Nov 18, 2023
cad44de
cargo fmt
fahimahmedx Nov 18, 2023
9909cc0
Add additional timestamp_to_block_number tests
fahimahmedx Nov 21, 2023
9d0fa79
return lower block number if edge case
fahimahmedx Nov 21, 2023
2ca9602
cargo fmt
fahimahmedx Nov 21, 2023
87fb1bd
Test before genesis block and after latest block
fahimahmedx Nov 22, 2023
48fa735
update commenting
fahimahmedx Nov 22, 2023
88d6289
Test latest timestamp to latest block number
fahimahmedx Nov 22, 2023
fb8e7bb
refactor timestamp test cases
fahimahmedx Nov 22, 2023
d09449d
add rate limit to timestamp test setup
fahimahmedx Nov 22, 2023
cd18ddc
remove unneeded test case
fahimahmedx Nov 22, 2023
1e8119e
update testing rpc
fahimahmedx Nov 22, 2023
69ea7c4
cargo fmt
fahimahmedx Nov 22, 2023
32b17b0
remove redundant timestamp test
fahimahmedx Nov 22, 2023
f17114c
reduce value in timestamp extrema test case
fahimahmedx Nov 22, 2023
87b99a0
Add tests for timestamp ending with a unit
fahimahmedx Nov 26, 2023
7547ce1
Fix timestamp range to blocknumber range and add tests
fahimahmedx Nov 26, 2023
a5e118a
Refactor parse timestamp functions and fix tests
fahimahmedx Nov 27, 2023
16983c7
cargo fmt
fahimahmedx Nov 27, 2023
89c9019
Refactor get_latest_block_number() into blocks.rs
fahimahmedx Nov 27, 2023
52b9520
remove unneeded test case
fahimahmedx Nov 27, 2023
c1e8ef4
optimize latest timestamp testcase
fahimahmedx Nov 27, 2023
d92444e
cargo fmt
fahimahmedx Nov 27, 2023
6161a2f
use parse_timestamps, and rename file
fahimahmedx Nov 27, 2023
21810c5
update docs, use timestamps instead of timestamp
fahimahmedx Nov 27, 2023
96363ad
fix typo in timestamp specification syntax
fahimahmedx Nov 28, 2023
5f86336
update timestamp scaling units
fahimahmedx Nov 28, 2023
ec904cd
remove debug print
fahimahmedx Nov 28, 2023
58f0ec0
update docs to incorporate timestamp scales
fahimahmedx Nov 28, 2023
3fc06ae
update tests
fahimahmedx Nov 28, 2023
bfa0ece
Remove test case that overloads RPC
fahimahmedx Nov 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Options:

Content Options:
-b, --blocks <BLOCKS>... Block numbers, see syntax below
--timestamps <TIMESTAMPS>... Timestamp numbers in unix, overriden by blocks
-t, --txs <TXS>... Transaction hashes, see syntax below
-a, --align Align chunk boundaries to regular intervals,
e.g. (1000 2000 3000), not (1106 2106 3106)
Expand Down Expand Up @@ -243,6 +244,18 @@ Block specification syntax
- can use every nth value 2000:5000:1000 == 2000 3000 4000
- can use n values total 100:200/5 == 100 124 149 174 199

Timestamp specification syntax
- can use numbers --timestamp 5000 6000 7000
- can use ranges --timestamp 12M:13M 15M:16M
- can use a parquet file --timestamp ./path/to/file.parquet[:COLUMN_NAME]
- can use multiple parquet files --timestamp ./path/to/files/*.parquet[:COLUMN_NAME]
- numbers can contain { _ . K M } 5_000 5K 15M 15.5M
- omitting range end means latest 15.5M: == 15.5M:latest
- omitting range start means 0 :700 == 0:700
- minus on start means minus end -1000:7000 == 6000:7000
- plus sign on end means plus start 15M:+1000 == 15M:15.001K
- can use n values total 100:200/5 == 100 124 149 174 199

Transaction specification syntax
- can use transaction hashes --txs TX_HASH1 TX_HASH2 TX_HASH3
- can use a parquet file --txs ./path/to/file.parquet[:COLUMN_NAME]
Expand Down
4 changes: 4 additions & 0 deletions crates/cli/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ pub struct Args {
#[arg(short, long, allow_negative_numbers = true, help_heading = "Content Options", num_args(1..))]
pub blocks: Option<Vec<String>>,

/// Timestamps in unix, see syntax below
#[arg(long, allow_negative_numbers = true, help_heading = "Content Options", num_args(0..))]
pub timestamps: Option<Vec<String>>,

/// Transaction hashes, see syntax below
#[arg(
short,
Expand Down
14 changes: 12 additions & 2 deletions crates/cli/src/parse/blocks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ fn read_integer_column(path: &str, column: &str) -> Result<Vec<u64>, ParseError>
}
}

async fn postprocess_block_chunks<P: JsonRpcClient>(
pub(crate) async fn postprocess_block_chunks<P: JsonRpcClient>(
block_chunks: Vec<BlockChunk>,
args: &Args,
fetcher: Arc<Fetcher<P>>,
Expand Down Expand Up @@ -218,7 +218,7 @@ async fn parse_block_token<P: JsonRpcClient>(
}
}

fn block_range_to_block_chunk(
pub(crate) fn block_range_to_block_chunk(
start_block: u64,
end_block: u64,
as_range: bool,
Expand Down Expand Up @@ -389,6 +389,16 @@ async fn apply_reorg_buffer<P: JsonRpcClient>(
}
}

pub(crate) async fn get_latest_block_number<P: JsonRpcClient>(
fetcher: &Fetcher<P>,
) -> Result<u64, ParseError> {
return fetcher
.get_block_number()
.await
.map(|n| n.as_u64())
.map_err(|_e| ParseError::ParseError("Error retrieving latest block number".to_string()));
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down
1 change: 1 addition & 0 deletions crates/cli/src/parse/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ mod partitions;
mod query;
pub(crate) mod schemas;
mod source;
mod timestamps;

pub use args::*;
use schemas::*;
6 changes: 6 additions & 0 deletions crates/cli/src/parse/partitions.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
use super::{
blocks,
parse_utils::{hex_string_to_binary, hex_strings_to_binary, parse_binary_arg},
timestamps,
};
use crate::args::Args;
use cryo_freeze::{
Expand All @@ -23,6 +24,11 @@ pub(crate) async fn parse_partitions<P: JsonRpcClient>(

// parse chunk data
let (block_number_labels, block_numbers) = blocks::parse_blocks(args, fetcher.clone()).await?;
let (block_number_labels, block_numbers) = if block_numbers.is_none() {
timestamps::parse_timestamps(args, fetcher.clone()).await?
} else {
(block_number_labels, block_numbers)
};
let (transaction_hash_labels, transactions) =
parse_transaction_chunks(&args.txs, "transaction_hash")?;
let call_datas = parse_call_datas(&args.call_data, &args.function, &args.inputs)?;
Expand Down
Loading
Loading