Skip to content

Commit

Permalink
feat(filecoin-api): allow custom hashing function to be passed to agg…
Browse files Browse the repository at this point in the history
…regate builder (#1553)

This PR updates `@web3-storage/data-segment` and passes a custom sha256
hashing function through to the aggregate builder.
  • Loading branch information
Alan Shaw authored Sep 20, 2024
1 parent a97a78a commit e2653d4
Show file tree
Hide file tree
Showing 5 changed files with 39 additions and 11 deletions.
2 changes: 1 addition & 1 deletion packages/filecoin-api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@
"@ucanto/transport": "^9.1.1",
"@web3-storage/capabilities": "workspace:^",
"@web3-storage/content-claims": "^5.0.0",
"@web3-storage/data-segment": "^4.0.0",
"@web3-storage/data-segment": "^5.2.0",
"fr32-sha2-256-trunc254-padded-binary-tree-multihash": "^3.3.0",
"p-map": "^6.0.0"
},
Expand Down
3 changes: 2 additions & 1 deletion packages/filecoin-api/src/aggregator/api.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { Signer, Principal, Link } from '@ucanto/interface'
import { InclusionProof } from '@web3-storage/capabilities/types'
import { PieceLink } from '@web3-storage/data-segment'
import { PieceLink, API as DataSegmentAPI } from '@web3-storage/data-segment'
import {
AggregatorService,
DealerService,
Expand Down Expand Up @@ -327,6 +327,7 @@ export interface AggregateConfig {
minAggregateSize: number
minUtilizationFactor: number
prependBufferedPieces?: BufferedPiece[]
hasher?: DataSegmentAPI.SyncMultihashHasher<DataSegmentAPI.SHA256_CODE>
}

// Enums
Expand Down
7 changes: 2 additions & 5 deletions packages/filecoin-api/src/aggregator/buffer-reducing.js
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,7 @@ export async function handleBufferReducingWithoutAggregate({
* Attempt to build an aggregate with buffered pieces within ranges.
*
* @param {BufferedPiece[]} bufferedPieces
* @param {object} config
* @param {number} config.maxAggregateSize
* @param {number} config.minAggregateSize
* @param {number} config.minUtilizationFactor
* @param {BufferedPiece[]} [config.prependBufferedPieces]
* @param {import('./api.js').AggregateConfig} config
*/
export function aggregatePieces(bufferedPieces, config) {
// Guarantee buffered pieces total size is bigger than the minimum utilization
Expand All @@ -175,6 +171,7 @@ export function aggregatePieces(bufferedPieces, config) {
// Create builder with maximum size and try to fill it up
const builder = Aggregate.createBuilder({
size: Aggregate.Size.from(config.maxAggregateSize),
hasher: config.hasher,
})

// add pieces to an aggregate until there is no more space, or no more pieces
Expand Down
1 change: 1 addition & 0 deletions packages/filecoin-api/src/aggregator/events.js
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ export const handleBufferQueueMessage = async (context, records) => {
minAggregateSize: context.config.minAggregateSize,
minUtilizationFactor: context.config.minUtilizationFactor,
prependBufferedPieces: context.config.prependBufferedPieces,
hasher: context.config.hasher,
})

// Store buffered pieces if not enough to do aggregate and re-queue them
Expand Down
37 changes: 33 additions & 4 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit e2653d4

Please sign in to comment.