Skip to content

EMD file

keitaroyam edited this page Sep 23, 2019 · 4 revisions

EMD file is generated by Velox software. For example this Zenodo entry DOI MicroED datasets are saved as this format. Here I describe how to use it for crystallographic data processing.

EMD file is actually HDF5 file. The hierarchy looks like:

$ h5ls -r Camera_Nano_Ceta_2044_850_mm.emd
/                        Group
/Application             Group
/Application/Velox       Group
/Application/Velox/DisplayLayout Dataset {1}
/Data                    Group
/Data/Image              Group
/Data/Image/37698bfa8e374ec3897b24d45fe467d8 Group
/Data/Image/37698bfa8e374ec3897b24d45fe467d8/Data Dataset {4096, 4096, 44/Inf}
/Data/Image/37698bfa8e374ec3897b24d45fe467d8/FrameLookupTable Dataset {44/Inf}
/Data/Image/37698bfa8e374ec3897b24d45fe467d8/Metadata Dataset {60000, 44/Inf}
/Data/Text               Group
/Data/Text/ccb62378cef544caa7a187f0f16eead4 Dataset {1}
/Experiment              Dataset {1}
/Features                Group
/Features/CameraFeature  Group
/Features/CameraFeature/b88a737bc636458a96d0d18fce5117e6 Dataset {1}
/Features/Features       Dataset {1}
/Info                    Dataset {1}
/Operations              Group
/Operations/CameraInputOperation Group
/Operations/CameraInputOperation/39373aca84b243248a6188be46c4d767 Dataset {1}
/Operations/Operations   Dataset {1}
/Presentation            Group
/Presentation/DisplayIndex Dataset {1}
/Presentation/Displays   Group
/Presentation/Displays/HtmlDisplay Group
/Presentation/Displays/HtmlDisplay/a5c3b3993d6a47ed921e84fb09f5999d Dataset {1}
/Presentation/Displays/ImageDisplay Group
/Presentation/Displays/ImageDisplay/dbee57785624425f9585f5e8a9ad0446 Dataset {1}
/Thumbnail.jpg           Dataset {326658}
/Version                 Dataset {1}

Metadata is saved in /Data/Image/*/Metadata. Using python with h5py, it can be read:

def get_metadata(metadata):
    mds = []
    for i in xrange(metadata.shape[1]):
        metadata_array = metadata[:, i].T
        mdata_string = metadata_array.tostring().decode("utf-8")
        mds.append(json.loads(mdata_string.rstrip('\x00')))

    return mds
# get_metadata()

import h5py
h = h5py.File(f, "r")
image_path = h["/Data/Image"]
k = image_path.keys()[0]
metadata = get_metadata(h["/Data/Image/%s/Metadata" % k])

Original json data look like:

{
    "Core": {
        "guid": "00000000000000000000000000000000", 
        "MetadataSchemaVersion": "v1/2013/07", 
        "MetadataDefinitionVersion": "7.9"
    }, 
    "GasInjectionSystems": "", 
    "Sample": "", 
    "Scan": {
        "ScanSize": {
            "width": "0", 
            "height": "0"
        }
    }, 
    "Detectors": {
        "Detector-2": {
            "ReadOutArea": {
                "top": "0", 
                "right": "4096", 
                "bottom": "4096", 
                "left": "0"
            }, 
            "ExposureTime": "2", 
            "ExposureMode": "", 
            "DetectorName": "EF-CCD", 
            "Binning": {
                "width": "1", 
                "height": "1"
            }, 
            "DetectorType": "ImagingDetector", 
            "DarkGainCorrectionType": "3", 
            "Shutters": {
                "Shutter-0": {
                    "Position": "PostSpecimen", 
                    "Type": "Electrostatic"
                }
            }
        }, 
        "Detector-1": {
            "ReadOutArea": {
                "top": "0", 
                "right": "4096", 
                "bottom": "4096", 
                "left": "0"
            }, 
            "ExposureTime": "2", 
            "ExposureMode": "", 
            "DetectorName": "BM-Falcon", 
            "Binning": {
                "width": "1", 
                "height": "1"
            }, 
            "DetectorType": "ImagingDetector", 
            "DarkGainCorrectionType": "3", 
            "Shutters": {
                "Shutter-0": {
                    "Position": "PostSpecimen", 
                    "Type": "Electrostatic"
                }
            }
        }, 
        "Detector-0": {
            "ReadOutArea": {
                "top": "0", 
                "right": "4096", 
                "bottom": "4096", 
                "left": "0"
            }, 
            "ExposureTime": "2", 
            "ExposureMode": "", 
            "DetectorName": "BM-Ceta", 
            "Binning": {
                "width": "1", 
                "height": "1"
            }, 
            "DetectorType": "ImagingDetector", 
            "DarkGainCorrectionType": "3", 
            "Shutters": {
                "Shutter-0": {
                    "Position": "PostSpecimen", 
                    "Type": "Electrostatic"
                }
            }
        }
    }, 
    "BinaryResult": {
        "DetectorIndex": "0", 
        "PixelUnitX": "m", 
        "ImageSize": {
            "width": "4096", 
            "height": "4096"
        }, 
        "Encoding": "Signed", 
        "CompositionType": "", 
        "BitsPerPixel": "32", 
        "AcquisitionUnit": "CameraImage", 
        "Offset": {
            "y": "-3.2803362865965109e-07", 
            "x": "-3.2803362865965109e-07"
        }, 
        "Detector": "BM-Ceta", 
        "PixelSize": {
            "width": "1.6017267024397026e-10", 
            "height": "1.6017267024397026e-10"
        }, 
        "PixelUnitY": "m"
    }, 
    "Instrument": {
        "InstrumentId": "9950088", 
        "ComputerName": "TALOS-9950088", 
        "InstrumentClass": "Talos", 
        "InstrumentModel": "Talos Arctica", 
        "ControlSoftwareVersion": "1.14.1", 
        "Manufacturer": "FEI Company"
    }, 
    "Acquisition": {
        "AcquisitionStartDatetime": {
            "DateTime": "1562657936"
        }, 
        "SourceType": "XFEG", 
        "AcquisitionDatetime": {
            "DateTime": "1562657936"
        }, 
        "BeamType": ""
    }, 
    "CustomProperties": {
        "Detectors[BM-Ceta].TimeStamp": {
            "type": "int64", 
            "value": "1562657941799119"
        }, 
        "Aperture[SA].Name": {
            "type": "string", 
            "value": "None"
        }, 
        "Detectors[BM-Ceta].DigitalGain": {
            "type": "double", 
            "value": "1"
        }, 
        "Detectors[BM-Ceta].FrameID": {
            "type": "ulong", 
            "value": "1"
        }, 
        "Aperture[OBJ].Name": {
            "type": "string", 
            "value": "None"
        }, 
        "Detectors[BM-Ceta].SaturationPoint": {
            "type": "long", 
            "value": "4947"
        }, 
        "RecodeFactor": {
            "type": "double", 
            "value": "1"
        }, 
        "Detectors[BM-Ceta].MaxPossiblePixelValue": {
            "type": "long", 
            "value": "9805"
        }, 
        "StemMagnification": {
            "type": "bool", 
            "value": "0"
        }, 
        "Detectors[BM-Ceta].CombinedSubFrames": {
            "type": "long", 
            "value": "1"
        }, 
        "Aperture[C2].Name": {
            "type": "string", 
            "value": "50"
        }, 
        "Detectors[BM-Ceta].TransferOK": {
            "type": "bool", 
            "value": "1"
        }
    }, 
    "Vacuum": {
        "VacuumMode": "Vented"
    }, 
    "Optics": {
        "LastMeasuredScreenCurrent": "0", 
        "Defocus": "-9.519695163107122e-11", 
        "Projector1LensIntensity": "0.87779033184051514", 
        "DiffractionLensIntensity": "0.64980185031890869", 
        "EFTEMOn": "false", 
        "ExtractorVoltage": "3700", 
        "OperatingMode": "1", 
        "SpotIndex": "10", 
        "GunLensSetting": "4", 
        "BeamConvergence": "0.0012142857142857142", 
        "ProbeMode": "2", 
        "Projector2LensIntensity": "0.96061533689498901", 
        "HighMagnificationMode": "SA", 
        "Apertures": {
            "Aperture-1": {
                "MechanismType": "Motorized", 
                "Type": "None", 
                "Name": "OBJ", 
                "PositionOffset": {
                    "y": "-0.00028464000000000001", 
                    "x": "7.3759999999999999e-05"
                }, 
                "Number": "4"
            }, 
            "Aperture-0": {
                "Diameter": "5.0000000000000002e-05", 
                "Name": "C2", 
                "Enabled": "3", 
                "Number": "2", 
                "MechanismType": "Motorized", 
                "Type": "Circular", 
                "PositionOffset": {
                    "y": "0.00080016000000000006", 
                    "x": "0.0095753599999999998"
                }
            }, 
            "Aperture-2": {
                "MechanismType": "Motorized", 
                "Type": "None", 
                "Name": "SA", 
                "PositionOffset": {
                    "y": "-0.00021968000000000001", 
                    "x": "0.00011024"
                }, 
                "Number": "5"
            }
        }, 
        "IntermediateLensIntensity": "0.59939253330230713", 
        "IlluminationMode": "None", 
        "AccelerationVoltage": "200000", 
        "NominalMagnification": "92246.809999999998", 
        "ProjectorMode": "2", 
        "ScreenCurrent": "0", 
        "TemOperatingSubMode": "BrightField", 
        "C2LensIntensity": "0.45822128653526306", 
        "ObjectiveLensMode": "HM", 
        "ObjectiveLensIntensity": "0.85571491718292236", 
        "C1LensIntensity": "0.33570191264152527", 
        "MiniCondenserLensIntensity": "0.91795682907104492"
    }, 
    "EnergyFilter": {
        "EntranceApertureType": ""
    }, 
    "Stage": {
        "AlphaTilt": "0.52171428295138011", 
        "Position": {
            "y": "-1.1214399999945335e-07", 
            "x": "1.7197999999986075e-07", 
            "z": "-1.7700000000018482e-09"
        }, 
        "BetaTilt": "0"
    }
}

For example metadata["Stage"]["AlphaTilt"] encodes alpha tilt angle in radians, metadata[“CustomProperties”][“Detectors[BM-Ceta].TimeStamp"]["value"] encodes time stamp in microseconds.

Image data is stored in /Data/Image/*/Data as 3D integer array. The n-th image can be obtained with h[data_path][:,:,n].

For data processing with DIALS, please use this format file.