| description |
|---|
This is the WIP open for the community to work on it. |
This is the current state of the WIP document
Short link: https://bit.ly/vPA-Guidelines
- PA Guidelines (official)
- vPA Eligibility, Rewards and Reputation (official)
- vPA Tools & Productivity Tips (Community)
- F8/F9 Endorsed Changes (Community)
A Veteran Proposal Assessor (vPA) is an offered role to Proposal Assessor (PAs) who have proven their status as a veteran in the Cardano Project Catalyst Community. In order to become a vPA in Fund9:
- You must have been a PA or vPA in the (2) most recently completed funds (excluding current fund)
- OR been a vPA in at least (3) funds since inception (excluding current fund)
- AND you must have less than a 51% filter-out rate on assessments you have done as a PA in the (2) most recently completed funds (excluding current fund)
The purpose of this role is to review PA assessments of proposals for funding (scoring proposals against criteria, providing ratings and feedback) as a part of the Project Catalyst “Assess QA” - quality assessment phase. As a vPA, you play an important role in the Project Catalyst governance process by ensuring that the PAs' guidance is genuine, of value and in line with the (PA-)guidelines. The assessment guidelines should be respected in the interest of the community.
vPAs are asked to determine how well each PA’s assessment has applied the PA guidelines to score and evaluate the proposal against each of the three criteria for normal proposal assessments: impact, feasibility, and auditability. Note that the criteria are different for challenge setting proposals: alignment, feasibility, and verifiability.
vPAs determine whether the quality of the work that the assessor (PA) has provided is high enough or if the assessment should be excluded as it is not up to standards. vPA’s will review proposal flags and also decide whether a PA assessment should be considered as (please refer to the Definitions of a quality assessment section of the PA guidelines for more details):
-
Excellent: An excellent application of the PA guidelines has been applied. The PA has participated as intended and provided high quality information to the voter about the proposal and, if required, given feedback to the proposer.
-
Good: A reasonable application of the PA guidelines has been applied. The PA has understood what is required from an assessment and has given a good assessment of the proposal. Various aspects of the PAs assessment may have been more opinionated, may have been lacking some detail or otherwise not of an excellent execution.
-
Filtered Out: The PA guidelines have not been followed. The PA has not participated as intended. Quality Information to the voter has not been provided or is lacking in the PA’s assessment. The assessment is extremely short and/or lacks clear rationale - as defined by the PA guidelines. You are strongly advised to provide a short rationale why you are filtering out the assessment. This encourages positive feedback loop but also a reference point for everyone else on reasoning for such action.
The PA’s combined scores across the three criteria will give an overall score for the proposal being assessed. This aggregated score is used to rank the proposals in the voting user interface. As a vPA, your review of this score will form a part of the collective vPA review of the quality of the PAs’ assessment of each proposal. This will impact the rewards that PAs receive and importantly, it will impact the final rank of each proposal in the voting app and can influence final results of the current Catalyst fund since voters' decisions can be informed by this information.
Conflict of interest: vPAs can NOT review assessments in challenges in which they are participating as a proposer, as an implementer, or are actively working with a proposer (eg. rewarded mentorship).
Public record: As a vPA, your work will not be anonymous. You will have to give your full name which will be linked to your work and published online and thus forever visible (The Internet does not forget).
- To add a layer of quality assessment to the assessment process
- To filter out biased or substandard assessments
- To highlight and reward high quality assessments
- To improve our PA’s capabilities. Being a vPA will also likely improve your abilities as a PA.
- To identify and block attempts to game the system by selfish actors.
Please consider each of the following points as you undertake your reviews:
-
As a veteran Proposer Assessor, your role helps to ensure the success and integrity of the Project Catalyst governance process. Making sure the guidelines are respected in the interest of the community. You are expected to uphold the same guiding principles as the PAs, outlined in the PA Assessment Guidelines.
-
Your decisions should always benefit the voters who will read the assessments. If an assessment does not add value to voters to help them in their decision, then it should be filtered out. When considering the impact of an assessment, the voter’s and proposer’s interests must come first, the PA interest is secondary. Being lenient when rating a PA’s work does not make sense if it is detrimental to the voter or to the proposer.
-
If applying the guidelines to the letter forces you to flag an otherwise useful assessment then consider that this might be a case where the guidelines need improvements. The PA guidelines are not set in stone nor deemed to be perfect. It's your role as a vPA to interpret them and see the intent behind each rule and use your own judgment and do what is best for the community. You are paid to think on your own, use your sensitivity and make decisions. No rule should be applied blindly.
-
Your review should be made based on how thoroughly the assessment achieves the review criteria set forth in the PA Assessment Guide. Since the role of a vPA is to make sure that the PA guidelines were followed by each individual PA, you must know these guidelines intimately. This is of primary importance because it gives the vPA a criterion for decision making and aligns our collective expectations.
-
The Catalyst process is all about retro-feedback loops to improve the process, so when you rate an assessment, it is highly recommended that you leave very short feedback on why you choose excellent, good or filtered out.
-
While every assessment is required to be in English, be mindful that assessors are global. Sentence structure, use of words and other communication norms may be different than your own. Evaluate the value added (usefulness) rather than the linguistic perfection.
-
Do not blindly follow the similarity filter: Similar proposals in some cases may lead to similar assessments, not necessarily implying that the assessments are baseless copies. Remember to have the voters’ best interest in mind.
-
The way you review is up to you. You can choose which proposals and which assessments to look at. You can develop a system to deep dive or you can explore the surface. Be creative and have fun.
-
In relation to flags, it is up to you as a vPA to determine the PA assessment’s quality. Assessment will have been flagged (1) by proposers who wish to draw attention to specific issues and (2) for similarity to other PA assessments, possibly indicating copy/paste.
-
Proposers tend to flag negative assessments more often than positive ones. Be sure to be balanced in your review of each assessment and keep the community’s best interest in mind while agreeing or disagreeing with the proposer’s flag.
-
Remember that as a vPA, your role is to use your best thinking to discern whether and how well the assessments benefit the community, and then make the best decision so that the people involved are treated fairly and according to the expectations of the process.
Be free to share your thoughts and process with other vPAs. But do your reviews independently. DO NOT share your reviews or thoughts on PAs assessments directly. ALL vPA reviewing must be conducted separately from other vPAs. However, if you uncover a non-obvious attempt at defeating the system (such as a PA using the comments on Ideascale to write their assessments, or other covered complex schemes involving several PAs) you can share the assessor IDs in the proper channel with other vPAs to have a closer look.
When the vPA phase opens, if you are eligible, you will receive an email with instructions on submitting your reviews. The email will provide the necessary details and explanations on the tools and methods used so that you can work confidently.
Feel free to reach out to other vPAs to help you develop and share your process for being a successful vPA. Also, look to community groups such as the Catalyst School and community made vPA Tools & Productivity Tips documents for assistance and advice.
These are some questions that you could have in mind while reviewing an assessment:
-
Is it specific to this particular proposal or can it fit any context? (bot usage or general abuse of the system defeating the purpose of the PA process: providing genuine and independent reviews of the proposals)
-
Is each star rating justified accordingly or does it seem independent of the rationale? (only positive points but just 4 stars instead of 5 or only negative points but still 3 stars?)
-
Does it bring value? Did you learn something that is not obvious after reading it?
-
Is it backed with facts or is it just an opinion that could be held without actual critical thinking and thorough analysis of the proposal?
-
Is it truthful? (does it agree with the actual proposal?)
-
Is it bringing a new perspective on the proposal from an original view point? Such as a useful rephrasing of the proposal successfully capturing the quintessence of the proposer’s idea and purpose. - Excellent assessments are often made of this.
More generally the ‘Definitions of a quality assessment’ section from the PA guidelines should be your main guide, try to rephrase this table as a set of questions to weigh the value of each assessment.
Once vPA work is complete, all the work will be aggregated and cross-referenced for outstanding blockers (such as referrals, etc.). Following the vPA assessment period, an advanced statistical analysis will be performed. This process is detailed in the vPA Eligibility, Rewards and Reputation (MVP) working document. A final document with results will then be made public. Whether an assessment is included in the final result, proposal score and displayed in the Voting App depends on the following criteria:
-
If PA assessments flagged by proposers are filtered out by vPA consensus (equal or greater than 50% of participating vPAs in review of any assessment), those comments and associated scores will be removed from final aggregate score of proposals that will be displayed in the Voting App
-
If PA assessments flagged by proposers are not filtered out by vPA consensus, those assessments will not be removed and shall remain part of Proposal aggregate scores displayed in the Voting App
-
Above holds true for any additional assessments reviewed by vPAs even if these were not originally flagged by Proposers. vPAs are encouraged to review any assessments, not just direct Proposer flags
Currently, a deviation script analysis is run which identifies how far individuals have deviated from the consensus. The purpose of this analysis is to prevent vPAs from intentionally gaming the system, reducing the integrity of the process, and being compensated for inappropriate behavior.
Don’t worry! If you participate as intended you will be doing your part to ensure quality control of the proposal process.
For Fund9 - 1% of the fund is allocated to vPA rewards. There are two key thresholds currently in place - minimum threshold for eligibility (200) and maximum threshold (5,000). Please see the document below for details:
Fund9 - vPA Eligibility & Incentives Requirements (open for comments)
Under Community ideation/review - will be finalized before the end of vPA review stage is over. Follow documentation via this link.
The Project Catalyst team thanks all our (v)PAs for helping to build the future of Cardano and to co-author these guidelines!
{% embed url="https://docs.google.com/document/d/1-Uheb5cH6PgjURBod3i95lnjRt5G0Q6WU7n21sSdqvs/edit?usp=sharing" %}