Skip to content

Commit

Permalink
Merge pull request #502 from maps-as-data/search_text
Browse files Browse the repository at this point in the history 
Search text
  • Loading branch information
@rwood-97
rwood-97 authored Sep 12, 2024
2 parents a87b5bc + 47680bd commit 45633b6
Show file tree
Hide file tree
Showing 9 changed files with 491 additions and 271 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ _ADD NEW CHANGES HERE_
- Loading of dataframes from GeoJSON files now supported in many file loading methods (e.g. `add_metadata`, `Annotator.__init__`, `AnnotationsLoader.load`, etc.) ([#495](https://github.com/maps-as-data/MapReader/pull/495))
- `load_frames.py` added to `mapreader.utils`. This has functions for loading from various file formats (e.g. CSV, Excel, GeoJSON, etc.) and converting to GeoDataFrames ([#495](https://github.com/maps-as-data/MapReader/pull/495))
- Added tests for text spotting code ([#500](https://github.com/maps-as-data/MapReader/pull/500))
- Added `search_preds`, `show_search_results` and `save_search_results_to_geojson` methods to text spotting code ([#502](https://github.com/maps-as-data/MapReader/pull/502))

### Changed

Expand Down
83 changes: 75 additions & 8 deletions docs/source/using-mapreader/step-by-step-guide/6-spot-text.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,15 +112,15 @@ e.g. for the ``DPTextDETRRunner``, if you choose the "ArT/R_50_poly.yaml", you s

e.g. for the ``DeepSoloRunner``, if you choose the "R_50/IC15/finetune_150k_tt_mlt_13_15_textocr.yaml", you should download the "ic15_res50_finetune_synth-tt-mlt-13-15-textocr.pth" model weights file from the DeepSolo repo.

e.g. for the ``MapTextPipeline``, if you choose the "ViTAEv2_S/rumsey/final_rumsey.yaml", you should download the "rumsey-finetune.pth" model weights file from the MapTextPipeline repo.
e.g. for the ``MapTextRunner``, if you choose the "ViTAEv2_S/rumsey/final_rumsey.yaml", you should download the "rumsey-finetune.pth" model weights file from the MapTextPipeline repo.

.. note:: We recommend using the "ViTAEv2_S/rumsey/final_rumsey.yaml" configuration and "rumsey-finetune.pth" weights from the ``MapTextPipeline``. But you should choose based on your own use case.

For the DPTextDETRRunner, use:

.. code-block:: python
from map_reader import DPTextDETRRunner
from mapreader import DPTextDETRRunner
#EXAMPLE
my_runner = DPTextDETR(
Expand All @@ -146,7 +146,7 @@ For the DeepSoloRunner, use:

.. code-block:: python
from map_reader import DeepSoloRunner
from mapreader import DeepSoloRunner
#EXAMPLE
my_runner = DeepSoloRunner(
Expand All @@ -158,14 +158,14 @@ For the DeepSoloRunner, use:
or, you can load your patch/parent dataframes from CSV/GeoJSON files as shown for the DPTextRunner (above).

For the MapTextPipeline, use:
For the MapTextRunner, use:

.. code-block:: python
from map_reader import MapTextPipeline
from mapreader import MapTextRunner
#EXAMPLE
my_runner = MapTextPipeline(
my_runner = MapTextRunner(
patch_df,
parent_df,
cfg_file = "MapTextPipeline/configs/ViTAEv2_S/rumsey/final_rumsey.yaml",
Expand All @@ -182,7 +182,7 @@ You can explicitly set this using the ``device`` argument:
.. code-block:: python
#EXAMPLE
my_runner = MapTextPipeline(
my_runner = MapTextRunner(
"./patch_df.csv",
"./parent_df.csv",
cfg_file = "MapTextPipeline/configs/ViTAEv2_S/rumsey/final_rumsey.yaml",
Expand Down Expand Up @@ -322,10 +322,77 @@ If you maps are georeferenced in your ``parent_df``, you can also convert the pi
geo_preds_df = my_runner.convert_to_coords(return_dataframe=True)
Again, you can save these to a csv file as above, or, you can save them to a geojson file for loading into GIS software:
Again, you can save these to a csv file (as shown above), or, you can save them to a geojson file for loading into GIS software:

.. code-block:: python
my_runner.save_to_geojson("text_preds.geojson")
This will save the predictions to a geojson file, with each text prediction as a separate feature.

Search predictions
------------------

If you are using the DeepSoloRunner or the MapTextRunner, you will have recognized text outputs.
You can search these predictions using the ``search_preds`` method:

.. code-block:: python
search_results = my_runner.search_preds("search term")
e.g To find all predictions containing the word "church" and ignoring the case:

.. code-block:: python
# EXAMPLE
search_results = my_runner.search_preds("church")
By default, this will return a dictionary containing the search results.
If you'd like to return a dataframe instead, use the ``return_dataframe`` argument:

.. code-block:: python
# EXAMPLE
search_results_df = my_runner.search_preds("church", return_dataframe=True)
You can also ignore the case of the search term by setting the ``ignore_case`` argument:

.. code-block:: python
# EXAMPLE
search_results_df = my_runner.search_preds("church", return_dataframe=True, ignore_case=True)
The search accepts regex patterns so you can use these to search for more complex patterns.

e.g. To search for all predictions containing the word "church" or "chapel", you could use the pattern "church|chapel":

.. code-block:: python
# EXAMPLE
search_results_df = my_runner.search_preds("church|chapel", return_dataframe=True, ignore_case=True)
Once you have your search results, you can view them on your map using the ``show_search_results`` method.

.. code-block:: python
my_runner.show_search_results("map_74488689.png")
This will show the map with the search results.

As with the ``show`` method, you can use the ``border_color``, ``text_color`` and ``figsize`` arguments to customize the appearance of the image.

Save search results
~~~~~~~~~~~~~~~~~~~

If your maps are georeferenced, you can also save your search results using the ``save_search_results_to_geojson`` method:

.. code-block:: python
my_runner.save_search_results_to_geojson("search_results.geojson")
This will save the search results to a geojson file, with each search result as a separate feature.

These can then be loaded into GIS software for further analysis/exploration.

If your maps are not georeferenced, you can save the search results to a csv file using the pandas ``to_csv`` method (as shown above).
132 changes: 3 additions & 129 deletions mapreader/spot_text/deepsolo_runner.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
)

import geopandas as gpd
import numpy as np
import pandas as pd
import torch
from adet.config import get_cfg
Expand All @@ -21,18 +20,16 @@
except ImportError:
raise ImportError("[ERROR] Please install Detectron2")

from shapely import LineString, MultiPolygon, Polygon

# first assert we are using the deep solo version of adet
if adet.__version__ != "0.2.0-deepsolo":
raise ImportError(
"[ERROR] Please install DeepSolo from the following link: https://github.com/rwood-97/DeepSolo"
)

from .runner_base import Runner
from .rec_runner_base import RecRunner


class DeepSoloRunner(Runner):
class DeepSoloRunner(RecRunner):
def __init__(
self,
patch_df: pd.DataFrame | gpd.GeoDataFrame | str | pathlib.Path,
Expand Down Expand Up @@ -68,6 +65,7 @@ def __init__(
self.patch_predictions = {}
self.parent_predictions = {}
self.geo_predictions = {}
self.search_results = {}

# setup the config
cfg = get_cfg() # get a fresh new config
Expand Down Expand Up @@ -231,50 +229,6 @@ def __init__(
# setup the predictor
self.predictor = DefaultPredictor(cfg)

def get_patch_predictions(
self,
outputs: dict,
return_dataframe: bool = False,
min_ioa: float = 0.7,
) -> dict | pd.DataFrame:
"""Post process the model outputs to get patch predictions.
Parameters
----------
outputs : dict
The outputs from the model.
return_dataframe : bool, optional
Whether to return the predictions as a pandas DataFrame, by default False
min_ioa : float, optional
The minimum intersection over area to consider two polygons the same, by default 0.7
Returns
-------
dict or pd.DataFrame
A dictionary containing the patch predictions or a DataFrame if `as_dataframe` is True.
"""
# key for predictions
image_id = outputs["image_id"]
self.patch_predictions[image_id] = []

# get instances
instances = outputs["instances"].to("cpu")
ctrl_pnts = instances.ctrl_points.numpy()
scores = instances.scores.tolist()
recs = instances.recs
bd_pts = np.asarray(instances.bd)

self._post_process(image_id, ctrl_pnts, scores, recs, bd_pts)
self._deduplicate(image_id, min_ioa=min_ioa)

if return_dataframe:
return self._dict_to_dataframe(self.patch_predictions, geo=False)
return self.patch_predictions

def _process_ctrl_pnt(self, pnt):
points = pnt.reshape(-1, 2)
return points

def _ctc_decode_recognition(self, rec):
last_char = "###"
s = ""
Expand All @@ -291,83 +245,3 @@ def _ctc_decode_recognition(self, rec):
else:
last_char = "###"
return s

def _post_process(self, image_id, ctrl_pnts, scores, recs, bd_pnts, alpha=0.4):
for ctrl_pnt, score, rec, bd in zip(ctrl_pnts, scores, recs, bd_pnts):
# draw polygons
if bd is not None:
bd = np.hsplit(bd, 2)
bd = np.vstack([bd[0], bd[1][::-1]])
polygon = Polygon(bd).buffer(0)

if isinstance(polygon, MultiPolygon):
polygon = polygon.convex_hull

# draw center lines
line = self._process_ctrl_pnt(ctrl_pnt)
line = LineString(line)

# draw text
text = self._ctc_decode_recognition(rec)
if self.voc_size == 37:
text = text.upper()
# text = "{:.2f}: {}".format(score, text)
text = f"{text}"
score = f"{score:.2f}"

self.patch_predictions[image_id].append([polygon, text, score])

@staticmethod
def _dict_to_dataframe(
preds: dict,
geo: bool = False,
parent: bool = False,
) -> pd.DataFrame:
"""Convert the predictions dictionary to a pandas DataFrame.
Parameters
----------
preds : dict
A dictionary of predictions.
geo : bool, optional
Whether the dictionary is georeferenced coords (or pixel bounds), by default True
parent : bool, optional
Whether the dictionary is at parent level, by default False
Returns
-------
pd.DataFrame
A pandas DataFrame containing the predictions.
"""
if geo:
columns = ["geometry", "crs", "text", "score"]
else:
columns = ["geometry", "text", "score"]

if parent:
columns.append("patch_id")

preds_df = pd.concat(
pd.DataFrame(
preds[k],
index=np.full(len(preds[k]), k),
columns=columns,
)
for k in preds.keys()
)

if geo:
# get the crs (should be the same for all)
if not preds_df["crs"].nunique() == 1:
raise ValueError("[ERROR] Multiple crs found in the predictions.")
crs = preds_df["crs"].unique()[0]

preds_df = gpd.GeoDataFrame(
preds_df,
geometry="geometry",
crs=crs,
)

preds_df.index.name = "image_id"
preds_df.reset_index(inplace=True) # reset index to get image_id as a column
return preds_df
2 changes: 1 addition & 1 deletion mapreader/spot_text/dptext_detr_runner.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def get_patch_predictions(
Returns
-------
dict or pd.DataFrame
A dictionary containing the patch predictions or a DataFrame if `as_dataframe` is True.
A dictionary containing the patch predictions or a DataFrame if `return_dataframe` is True.
"""
# key for predictions
image_id = outputs["image_id"]
Expand Down
Loading

0 comments on commit 45633b6

Please sign in to comment.