Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new URL encoding function to align with SEC API #40

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f5171a1
Update README examples w/ x-platform escape chars
jordan-gillard Jul 7, 2024
549273e
Use urllib.parse.quote to encode quote characters
jordan-gillard Jul 9, 2024
ec66721
Merge remote-tracking branch 'upstream/main' into clarify-escape-char…
jordan-gillard Jul 9, 2024
53576dc
Avoid double escaping keywords in url and handle single quotes
GalenReich Jul 10, 2024
d7f754c
Update exact search instructions in README
GalenReich Jul 10, 2024
164bbb2
feat: Add search param validator and basic tests
jordan-gillard Jul 14, 2024
49876a2
test: Add invalid arg tests
jordan-gillard Jul 15, 2024
c498941
test: Add invalid custom date tests
jordan-gillard Jul 16, 2024
0391bf6
test: Raises error for invalid date_range_select
jordan-gillard Jul 16, 2024
bb850ec
test: Add tests for all date options
jordan-gillard Jul 16, 2024
9b51728
Merge remote-tracking branch 'upstream/main' into clarify-escape-char…
jordan-gillard Jul 28, 2024
e30880b
Add filing categories to url_generator
jordan-gillard Jul 28, 2024
eb2583a
Remove duplicate filing category to form id code
jordan-gillard Jul 28, 2024
534b672
feat: Add support for single forms
jordan-gillard Aug 7, 2024
ca7add3
Amend me! Add TODO for remaining work
jordan-gillard Aug 7, 2024
d31de9a
Add ISO 3166-1/2 country codes to EDGAR codes
jordan-gillard Aug 17, 2024
4d4c1ba
Add urls for remaining countries
jordan-gillard Aug 18, 2024
41ee9bb
Add test for invalid peo_in
jordan-gillard Aug 18, 2024
f3204b8
Add tests for incorportated in
jordan-gillard Sep 7, 2024
c4e0163
Update url_generator to use EDGAR API URL
jordan-gillard Sep 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 32 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,26 +38,49 @@ This is a command line tool that takes a search query, queries a server, and dow

### Examples

```bash
# Display help message describing all supported arguments along with their usage, aliases and eventual default values (type q to exit)
Display help message describing all supported arguments along with their usage, aliases and eventual default values (type `q` to exit)

```shell
edgar-tool text_search --help
```

Basic usage (defaults to searching the last 5 years of records)

# Basic usage (defaults to searching the last 5 years of records)
```shell
edgar-tool text_search John Doe
```

You can wrap a phrase in quotes if you want an exact match.
This works in both POSIX-compliant shells (Linux/Bash) and Windows PowerShell environments.

# Basic usage with a combination of exact and partial search parameters
edgar-tool text_search \"John Doe\" Pharmaceuticals Chemicals
For example, the following usage will search for the exact phrase `"John Doe"` and treat `Pharmaceuticals` and
`Chemicals` as partial search parameters.

```shell
edgar-tool text_search "John Doe" Pharmaceuticals Chemicals
```

# Usage with date range and export to custom CSV file
Usage with date range and export to custom CSV file

```shell
edgar-tool text_search Tsunami Hazards --start_date "2021-01-01" --end_date "2021-12-31" --output "results.csv"
```

# Usage with a partial set of filing forms + single forms
### Usage with a partial set of filing forms + single forms

```
edgar-tool text_search Hurricane Damage --filing_form "registration_statements" --single_forms "['1-K', '1-SA']"
```

Usage specifying the location of incorporation

# Usage specifying the location of incorporation
```shell
edgar-tool text_search oil --inc_in "Egypt"
```

More advanced usage specifying more arguments, with export to JSON

# More advanced usage specifying more arguments, with export to JSON
```shell
edgar-tool text_search Volcano Monitoring --start_date "2021-01-01" --end_date "2021-12-31" --output "results.json"\
--filing_form "all_annual_quarterly_and_current_reports" --entity_id "0001030717" \
--min_wait 5.0 --max_wait 7.0 --retries 3
Expand Down
4 changes: 2 additions & 2 deletions edgar_tool/main.py → edgar_tool/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
import fire


def main_entrypoint():
def main():
fire.Fire(SecEdgarScraperCli)


if __name__ == "__main__":
main_entrypoint()
main()
19 changes: 11 additions & 8 deletions edgar_tool/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,16 @@ def _validate_text_search_args(
):
raise ValueError(
f"Filing form group must be one of: {'; '.join(TEXT_SEARCH_FILING_VS_MAPPING_CATEGORIES_MAPPING.keys())}"
)
)
if single_forms:
single_list = [item for sublist in TEXT_SEARCH_CATEGORY_FORM_GROUPINGS.values() for item in
sublist]
single_list = [
item
for sublist in TEXT_SEARCH_CATEGORY_FORM_GROUPINGS.values()
for item in sublist
]
invalid_forms = [form for form in single_forms if form not in single_list]
if invalid_forms:
raise ValueError(
f"Single forms must be one or more of: {single_list}"
)
raise ValueError(f"Single forms must be one or more of: {single_list}")


class SecEdgarScraperCli:
Expand Down Expand Up @@ -135,7 +136,9 @@ def text_search(
scraper.text_search(
keywords=keywords,
entity_id=entity_id,
filing_form=TEXT_SEARCH_FILING_VS_MAPPING_CATEGORIES_MAPPING.get(filing_form),
filing_form=TEXT_SEARCH_FILING_VS_MAPPING_CATEGORIES_MAPPING.get(
filing_form
),
single_forms=single_forms,
start_date=start_date,
end_date=end_date,
Expand All @@ -144,7 +147,7 @@ def text_search(
retries=retries,
destination=output,
peo_in=peo_in,
inc_in=inc_in
inc_in=inc_in,
)

@staticmethod
Expand Down
Loading
Loading