- for folder and file names:
- don't use white space in either folder or filenames, use an underscore "_" instead
- (almost always) use lower case only
- always use relative paths in your code
- for example, to save a figure from an R script inside the
code/R/
folder the path should be "../../figures/figure_name.pdf"
- for example, to save a figure from an R script inside the
- keep your folder structure organized
- we recommend adhering to the folder structure in this repository
- more complex projects may have additional folders such as
videos/
,tables/
, ...
- note: some of the folders are empty except for a
.keep
file- the
.keep
file is just there to make sure that github includes the otherwise empty folder - feel free to delete the
.keep
file once you've added another file to that folder
- the
- each code subfolder has a readme file that should be updated with information about the code scripts
- use github issues to keep track of any larger decisions that we make along the way
- make sure to create a slack channel for each project, link up the github repository with the slack channel, and add the people working on the project to the github repo and slack channel
- see our lab wiki for more help: https://github.com/cicl-stanford/wiki/wiki
├── code
├── data
├── figures
├── papers
├── presentation
└── writeup
Put all your code here. Use a separate folder for scripts based on the programming language.
The experiments folder is for the online (or in lab) experiments. Each experiment should be in its own folder. When you run another experiment, make sure to create a new folder (so that we always know what an experiment looked like when it was run). In readme file for the experiments folder, provide a brief summary of each experiment. Also note down any additional information that may not be saved within each experiment (e.g. how much the payment was for MTurk participants).
Put your raw data files here. Any data wrangling to that file should happen in your code scripts.
There are subfolders for each experiment, containing raw data in table format.
In the box_drawing
subfolder, the data is split between two sheets, with the relevant differences being the date and the version of the script used. In both data sheets, the tables has the following columns:
ID
, the first column, refers to the ID number assigned to the subject in that row. This number is determined by the order in which subjects participate; it is arbitrary for anything other than identification purposes.Sbj_initials
, the second column, contains the initials of the subject for that row. The convention is first initial and last initial, so for subjects with multiple last names, more than two letters may appear.- The
Date
column contains the date on which the subject participated in the study, written in MM/DD/YYYY format. - The
Duration
column contains the length of the experiment for each subject, written in minutes:seconds format. - The
Script Version
column indicates what version of the task script the experiment used to run that trial of the experiment. Scripts are occassionally tweaked and updated, and this column clarifies which version was used. Links to existing scripts are as follows:- v1.0: https://docs.google.com/document/d/14PzlaPKJn_mRJrq7o3u1eqwYMGY2NOIRWqcJ5cjOJ8U/edit
- v1.1: https://docs.google.com/document/d/10qGCRhbsQU-iwgjTPJbJ2Eg5zTXfJX6R_n0u5fiiui4/edit
- v1.2: https://docs.google.com/document/d/1f_RExck9tISWX32G8PtJTduXlGGbzDidApR5Z3ln4Sc/edit
- v1.3: https://docs.google.com/document/d/1qxSR5ftRd7R62ksblsltdppru05BUbfuax7Vr4S9144/edit
- v2.0: https://docs.google.com/document/d/19REw-8zXpKNuFg0ei-GoAGrdkE-3SoUXdKY_kotuHaE/edit
- The
Experimenter_initials
column contains the initials of the experimenter who ran each subject in the experiment. - The
NOTES
column is a space for experimenters to write down any relevant information about that iteration of the experiment. For example, this section could include a note about equipment/experimenter error, or it could contain a note about the subject's behavior, if notable. - The
peeking
column is answered with eitherYes
orNo
, signalling whether or not the subject attempted to peek beneath the box while they were drawing. This experiment involves the subject attempting to draw a straight line while unable to see what they are drawing, so it is important to note whether or not they tried to look during the drawing phase. - The
Age_mos
column contains the subject's age in months. - The
Gender
column contains either anM
for male or anF
for female, signifying the subject's assigned sex. - The
Condition
column indicates whether the subject was in the control condition or the experimental condition. In the control condition, which is denoted by a 0, subjects are given a "drawing guide" made of two rulers to help them draw their line. In the experimental condition, indicated by a 1, subjects do not receive the help of a guide. - The
Surprise_verbal
category indicates if, when asked, the subject responded that they were surprised by the outcome of their attempt to draw a straight line.Yes
indicates that the participant was surprised by the result, whileNo
indicates that they were not. - If a subject says yes to the surprise question, they are asked a follow-up question to gauge how suprised they are. The
Suprise_verbal_graded
column contains the answers to this question. Experimenters give subjects three options (ex: "a little", "medium", "a lot"), and the subject's response is listed in this column. Options have varried slightly from script to script; scipt documents can be consulted for the options presented in each version. For subjects who said they were not surprised, this column is filled with a dash. - The
Suprise_why
column contains answers to a follow-up question asking subjects why they were or were not surprised by the results of their drawing. Cells left blank indicate that the researcher did not ask this question. - The
Actual_performance
column lists eitherSuccess
orFail
/Failure
, indicating whether or not the subject's actual drawing was a straight line connecting the dots, as instructed. This experiment involves swapping out the subject's actual drawing for a perfectly straight line, so all subjects are led to believe that they have succeeded. This column represents whether or not their true performance was a success. - The
Surprise_expression
column indicates with aYes
orNo
whether or not the subject's facial expression looked surprised when their drawing was revealed. This category is intended as a check on the accuracy/authority of verbal responses in theSurprise_verbal
column. - The
Exploration_drawing_game
andExploration_tracing_game
pertain to the exploration phase of the experiment. After subjects have drawn their line and been shown the decoy drawing, they are allowed to choose between trying the same task again, or trying a novel task -- a connect-the-dot sheet. If the subject chooses to engage with the same task again, there will be aYes
in theExploration_drawing_game
column. If they choose to try the new task, there is aYes
in theExploration_tracing_game
column. It is possible for both columns to have aYes
(if the subject experiments with both), and it is also possible for both columns to have aNo
(i.e., if the subject chose not to engage with either option). - The
Exploration_use_box
category signals whether a subject chose to use the box prop in the exploratory phase of the experiment. Empty cells in this column indicate that this column did not exist at the time of the experiment. - Lastly, the
Exploration_check_out_box
column indicates whether or not subjects chose to examine the box prop while left alone for the exploration phase. Similar to the previous column, blank cells in this column indicate that it was added after those experiments were run.
In the cards
subfolder, the data is contained in a sheet broken down into the following columns:
ID
: see box_drawing (above) for explanationSbj_initials
: see box_drawing (above) for explanationDate
: see box_drawing (above) for explanationDuration
: see box_drawing (above) for explanation- The
Script ver
column serves the same purpose as theScript Version
column from the box_drawing data sheets. The following are links to each existing task script for the card experiment:- v1.2: https://docs.google.com/document/d/1xNwy87b9ELjgRmofi06195TBFM0YVWIVJW_4Ohpa5F8/edit
- v1.3: https://docs.google.com/document/d/14805mEIHWAD4M8qlzkz6L8BKoaDJQfoCx4QHU8YSsA4/edit
- v1.4: https://docs.google.com/document/d/1k-dNcLDtSVnJhCL8VQKgrWXMgpjnnfpYXExy--Hvpgg/edit
- v2.0: https://docs.google.com/document/d/1g8riSaudx332AjaqfXe0TQBnB11R8RLt2P0jhikpHRI/edit
Exp_initials
: seeExperimenter_initials
decription from box_drawing (above) for explanationNOTES
: see box_drawing (above) for explanationAge
: seeAge_mos
category from box_drawing (above) for explanationGender
: see box_drawing (above) for explanation- The
Cond_deck
category indicates which deck of cards was used for that subject. Both card decks used have 16 cards, four of which have stars on their fronts. TheNo markers
option corresponds to a deck in which the backs of all 16 cards are identical, while theMarkers
answer indicates that the experiment used the control condition, a deck in which the four star cards are marked with star stickers on their backs. - The
No of star cards picked
indicates how many star cards the subject was able to correctly identify when given four tries to pick out a card, looking only at the backs of the cards. Subjects run in the experimental condition (No markers
) have a dash for this column, because in their condition, the results of the game are rigged to result in four stars cards as a result every time. Subjects in the control condition (Markers
) have an integer between 0 and 4 in this category, indicating the number of star cards they identified. Surprise_verbal
: see box_drawing (above) for explanationSurprise_verbal_graded
: see box_drawing (above) for explanationPicking_strategy
: subjects were asked to explain the strategy that they used to pick their cards. Their answers are summarized in this column.- In the exploration phase of the cards experiment, subjects could choose to continue playing with the cards or to instead play with blocks. For script version 1.1 and 1.2,the blocks were a mix of metal and wood. From version 1.3 onwards, the blocks were wood only. The subjects played alone in the exploration phase. A
Yes
in theExploration_cards
column indicates that the subject chose to continue playing with the cards. AYes
in theExploration_blocks
column indicates that the subject chose to play with the blocks. It is possible to have aYes
in both columns, aNo
in both columns, or, most commonly, one of each.
In the jumping
subfolder, the data sheet is divided into the following columns:
ID
: see box_drawing (above) for explanationSubject_initials
: the same asSbj_initials
from cards and box_drawingDate
: see box_drawing (above) for explanationExperimenter_initials
: see box_drawing (above) for explanationAge in months
: same asAge
andAge_mos
columns from cards and box_drawing, respectivelyGender
: see box_drawing (above) for explanation- In the
How high do you THINK you can reach when you jump?
column, subjects were asked the title question. The experimenter moved their hand progressively lower on a measuring tape afixed to the wall, and instructed subjects to tell them when to stop. The height, in inches, of the stopping point is recorded in this column. - In the
Reach with extended arm WITHOUT jumping
column, subjects were asked to stand next to the measuring tape and fully extend their arm upwards. The height, in inches, reached by the top of their hand is recorded in this column. - In the
Reach with extended arm WITH jumping
column, subjects were asked to repeat the procedure from the previous column but then, with their arm still extended, to jump vertically as high as they could. The height reach by their hand at the peak of their jump is recorded in this column. - The
Other notes
column is similar to theNOTES
columns in cards and box_drawing; it is a space for the experimenter to take note of details that, while relevant, do not belong in any of the other columns.
Save all your figures here. You may want to include additional subfolder here such as plots/
, diagrams/
etc.
Put research papers here that are relevant for your project.
Put your project presentation here (e.g. your keynote, powerpoint, google slides, or pdf file).
Put all your writing here. This folder structure is likely to expand for more complex projects. For example, you could add a subfolders like folders journal/cognition/submission/
, proceedings/cogsci/resubmission/
etc.
Each public repository should have a credit author statement
Term | Definition |
---|---|
Conceptualization | Ideas; formulation or evolution of overarching research goals and aims |
Methodology | Development or design of methodology; creation of models |
Software | Programming, software development; designing computer programs; implementation of the computer code and supporting algorithms; testing of existing code components |
Validation | Verification, whether as a part of the activity or separate, of the overall replication/ reproducibility of results/experiments and other research outputs |
Formal analysis | Application of statistical, mathematical, computational, or other formal techniques to analyze or synthesize study data |
Investigation | Conducting a research and investigation process, specifically performing the experiments, or data/evidence collection |
Resources | Provision of study materials, reagents, materials, patients, laboratory samples, animals, instrumentation, computing resources, or other analysis tools |
Data Curation | Management activities to annotate (produce metadata), scrub data and maintain research data (including software code, where it is necessary for interpreting the data itself) for initial use and later reuse |
Writing - Original Draft | Preparation, creation and/or presentation of the published work, specifically writing the initial draft (including substantive translation) |
Writing - Review & Editing | Preparation, creation and/or presentation of the published work by those from the original research group, specifically critical review, commentary or revision – including pre-or postpublication stages |
Visualization | Preparation, creation and/or presentation of the published work, specifically visualization/ data presentation |
Supervision | Oversight and leadership responsibility for the research activity planning and execution, including mentorship external to the core team |
Project administration | Management and coordination responsibility for the research activity planning and execution |
Funding acquisition | Acquisition of the financial support for the project leading to this publication |
Sample CRediT author statement
- Zhang San: Conceptualization, Methodology, Software
- Priya Singh: Data curation, Writing- Original draft preparation
- Wang Wu: Visualization, Investigation
- Jan Jansen: Supervision
- Ajay Kumar: Software, Validation
- Sun Qi: Writing-Reviewing and Editing