canSAS - User contributions [en]

2012 Data Discussion

2012-07-27T17:53:48Z

Andyfaff: /* Suggest format framework */

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments in the appropriate section below:

= Agenda with Discussion =

== 1D Format ==
=== Agree a proposed extension of the current 1D standard ===
* This is an extension proposed by Steve King (ISIS) that is required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files
** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
=== Agree resultant changes to the "official" schema/stylesheet ===
{{Comment|SMK|For clarity, this extension was proposed ''after'' adoption of the existing version of the standard. Ratification of this extension at CanSAS-2012 would give facilities/software developers the necessary confidence to implement it.}}

{{Comment | SMK | This proposed extension would allow more-or-less any non-I(Q) data to be included in a CanSAS-1D data file under a suitable foreign namespace. That is remarkable flexibility, and it does not require any significant revision of the existing standard. However, it could be argued that certain types of non-I(Q) data are so integral to a SAS experiment that they should instead be given explicit SASXML tags within the standard? Transmission data is a good example of this: the existing standard explicitly includes a tag for the transmission value from a ''monochromatic'' measurement (SASsample\transmission) but makes no provision for transmission values from ''white beam'' measurements. The proposed foreign namespace extension will make such provision possible ''but only'' as an extension, not as a formal part of the standard. Is that acceptable? The downside of explicitly adding new SASXML tags to the existing standard is that it would require issuing a new version of the standard.
}}
{{Comment | ARJN | I would recommend that during this meeting that the current format then be 'parked', the new 2D format should be flexible enough to handle 1D and 2D data.
}}
{{Comment | SMK | Agreed.
}}
{{Comment | ARR | Perhaps we should be cautious about suggesting no further development for the existing 1-D format. Unless there is scope for adapting to future needs, it will be seen as not suitable for new uses. This, of course, does not imply that there have to be active plans for further changes at present.
}}
{{Comment | AJJ | As to this particular case, I suggest that since SMK and I have it done, that we accept the additional tags, call it v1.1 and move on. If other extensions arise later and become "widely" used, then we can incorporate them as they arise. This is the key benefit of using an extensible format.
}}
{{Comment | ARJN | Just as a matter of interest, why is it necessary to include transmission measurements for TOF instruments? Does the analysis depend on these values? Not saying it's not necessary, just wondering about why it is useful.
}}

== 2D Format ==
=== Define minimum information necessary for reduced data ===
* definition of what '''reduced''' data is and hence the problems we should address.
* review previous discussions on 2D
* considerations specific to 2D reduced data
* consider forward looking issues such as
** grazing incidence
** event mode analysis

=== Suggest format framework ===
* NeXus extension, canSAS 1D extension, other, with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
** compare with NeXus definitions for:
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml

{{Comment | ARJN | Having looked at both those Nexus definitions it is not clear to me how:
** multiple detectors are handled.
** What happens if the detector pixels are not grid shaped?
** how does one stuff multiple datasets (e.g. from event mode acquisition)?
Remember, all we need is instrument independent information in this file, nothing else.
:[[User:Andyfaff|Andyfaff]] 12:45, 27 July 2012 (CDT)
}}

----
* James Hester (an expert in the CIF format) has compelling ideas on the subject of common data files. [[CIF format notes Hester]]

* Andrew Nelson has commented via email exchange with Adrian Rennie : [[Email ARJN ARR | Notes on Reduced Data formats with examples from reflectometry]]

----

* Ron Ghosh has laid out details of a proposal to use HDF5/Nexus : [[HDF5 Notes Ghosh]]

----

* Peter Boesecke misses definitions of parameters that are needed to describe small-angle and wide-angle scattering experiments. Parameters like "Distance", "Center", "Rotation" have only a meaning when they are part of a geometrical model, otherwise they are useless. He has collected parameters that are required for SAS/WAS experiments with position sensitive detectors (1D and 2D). When saving experimental data it should be checked that these parameters are either implicitly or explicitly described by the supplied metadata: [[User:Boesecke | SX_parametrization]].

----

=== Guide to how to make implementation easy. ===
=== Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.===
=== Provide a plan for presentation at SAS 2012 ===

[[Category: canSAS 2012]]

2012 Data Discussion

2012-07-27T17:52:00Z

Andyfaff: /* Agree resultant changes to the "official" schema/stylesheet */

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments in the appropriate section below:

= Agenda with Discussion =

== 1D Format ==
=== Agree a proposed extension of the current 1D standard ===
* This is an extension proposed by Steve King (ISIS) that is required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files
** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
=== Agree resultant changes to the "official" schema/stylesheet ===
{{Comment|SMK|For clarity, this extension was proposed ''after'' adoption of the existing version of the standard. Ratification of this extension at CanSAS-2012 would give facilities/software developers the necessary confidence to implement it.}}

{{Comment | SMK | This proposed extension would allow more-or-less any non-I(Q) data to be included in a CanSAS-1D data file under a suitable foreign namespace. That is remarkable flexibility, and it does not require any significant revision of the existing standard. However, it could be argued that certain types of non-I(Q) data are so integral to a SAS experiment that they should instead be given explicit SASXML tags within the standard? Transmission data is a good example of this: the existing standard explicitly includes a tag for the transmission value from a ''monochromatic'' measurement (SASsample\transmission) but makes no provision for transmission values from ''white beam'' measurements. The proposed foreign namespace extension will make such provision possible ''but only'' as an extension, not as a formal part of the standard. Is that acceptable? The downside of explicitly adding new SASXML tags to the existing standard is that it would require issuing a new version of the standard.
}}
{{Comment | ARJN | I would recommend that during this meeting that the current format then be 'parked', the new 2D format should be flexible enough to handle 1D and 2D data.
}}
{{Comment | SMK | Agreed.
}}
{{Comment | ARR | Perhaps we should be cautious about suggesting no further development for the existing 1-D format. Unless there is scope for adapting to future needs, it will be seen as not suitable for new uses. This, of course, does not imply that there have to be active plans for further changes at present.
}}
{{Comment | AJJ | As to this particular case, I suggest that since SMK and I have it done, that we accept the additional tags, call it v1.1 and move on. If other extensions arise later and become "widely" used, then we can incorporate them as they arise. This is the key benefit of using an extensible format.
}}
{{Comment | ARJN | Just as a matter of interest, why is it necessary to include transmission measurements for TOF instruments? Does the analysis depend on these values? Not saying it's not necessary, just wondering about why it is useful.
}}

== 2D Format ==
=== Define minimum information necessary for reduced data ===
* definition of what '''reduced''' data is and hence the problems we should address.
* review previous discussions on 2D
* considerations specific to 2D reduced data
* consider forward looking issues such as
** grazing incidence
** event mode analysis

=== Suggest format framework ===
* NeXus extension, canSAS 1D extension, other, with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
** compare with NeXus definitions for:
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml

----
* James Hester (an expert in the CIF format) has compelling ideas on the subject of common data files. [[CIF format notes Hester]]

* Andrew Nelson has commented via email exchange with Adrian Rennie : [[Email ARJN ARR | Notes on Reduced Data formats with examples from reflectometry]]

Having looked at both those Nexus definitions it is not clear to (ARJN) how:
** multiple detectors are handled.
** What happens if the detector pixels are not grid shaped?
** how does one stuff multiple datasets (e.g. from event mode acquisition)?
Remember, all we need is instrument independent information in this file, nothing else.
:[[User:Andyfaff|Andyfaff]] 12:45, 27 July 2012 (CDT)

----

* Ron Ghosh has laid out details of a proposal to use HDF5/Nexus : [[HDF5 Notes Ghosh]]

----

* Peter Boesecke misses definitions of parameters that are needed to describe small-angle and wide-angle scattering experiments. Parameters like "Distance", "Center", "Rotation" have only a meaning when they are part of a geometrical model, otherwise they are useless. He has collected parameters that are required for SAS/WAS experiments with position sensitive detectors (1D and 2D). When saving experimental data it should be checked that these parameters are either implicitly or explicitly described by the supplied metadata: [[User:Boesecke | SX_parametrization]].

----

=== Guide to how to make implementation easy. ===
=== Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.===
=== Provide a plan for presentation at SAS 2012 ===

[[Category: canSAS 2012]]

2012 Data Discussion

2012-07-27T17:47:07Z

Andyfaff: /* Suggest format framework */

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments in the appropriate section below:

= Agenda with Discussion =

== 1D Format ==
=== Agree a proposed extension of the current 1D standard ===
* This is an extension proposed by Steve King (ISIS) that is required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files
** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
=== Agree resultant changes to the "official" schema/stylesheet ===
{{Comment|SMK|For clarity, this extension was proposed ''after'' adoption of the existing version of the standard. Ratification of this extension at CanSAS-2012 would give facilities/software developers the necessary confidence to implement it.}}

{{Comment | SMK | This proposed extension would allow more-or-less any non-I(Q) data to be included in a CanSAS-1D data file under a suitable foreign namespace. That is remarkable flexibility, and it does not require any significant revision of the existing standard. However, it could be argued that certain types of non-I(Q) data are so integral to a SAS experiment that they should instead be given explicit SASXML tags within the standard? Transmission data is a good example of this: the existing standard explicitly includes a tag for the transmission value from a ''monochromatic'' measurement (SASsample\transmission) but makes no provision for transmission values from ''white beam'' measurements. The proposed foreign namespace extension will make such provision possible ''but only'' as an extension, not as a formal part of the standard. Is that acceptable? The downside of explicitly adding new SASXML tags to the existing standard is that it would require issuing a new version of the standard.
}}
{{Comment | ARJN | I would recommend that during this meeting that the current format then be 'parked', the new 2D format should be flexible enough to handle 1D and 2D data.
}}
{{Comment | SMK | Agreed.
}}
{{Comment | ARR | Perhaps we should be cautious about suggesting no further development for the existing 1-D format. Unless there is scope for adapting to future needs, it will be seen as not suitable for new uses. This, of course, does not imply that there have to be active plans for further changes at present.
}}
{{Comment | AJJ | As to this particular case, I suggest that since SMK and I have it done, that we accept the additional tags, call it v1.1 and move on. If other extensions arise later and become "widely" used, then we can incorporate them as they arise. This is the key benefit of using an extensible format.
}}

== 2D Format ==
=== Define minimum information necessary for reduced data ===
* definition of what '''reduced''' data is and hence the problems we should address.
* review previous discussions on 2D
* considerations specific to 2D reduced data
* consider forward looking issues such as
** grazing incidence
** event mode analysis

=== Suggest format framework ===
* NeXus extension, canSAS 1D extension, other, with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
** compare with NeXus definitions for:
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml

----
* James Hester (an expert in the CIF format) has compelling ideas on the subject of common data files. [[CIF format notes Hester]]

* Andrew Nelson has commented via email exchange with Adrian Rennie : [[Email ARJN ARR | Notes on Reduced Data formats with examples from reflectometry]]

Having looked at both those Nexus definitions it is not clear to (ARJN) how:
** multiple detectors are handled.
** What happens if the detector pixels are not grid shaped?
** how does one stuff multiple datasets (e.g. from event mode acquisition)?
Remember, all we need is instrument independent information in this file, nothing else.
:[[User:Andyfaff|Andyfaff]] 12:45, 27 July 2012 (CDT)

----

* Ron Ghosh has laid out details of a proposal to use HDF5/Nexus : [[HDF5 Notes Ghosh]]

----

* Peter Boesecke misses definitions of parameters that are needed to describe small-angle and wide-angle scattering experiments. Parameters like "Distance", "Center", "Rotation" have only a meaning when they are part of a geometrical model, otherwise they are useless. He has collected parameters that are required for SAS/WAS experiments with position sensitive detectors (1D and 2D). When saving experimental data it should be checked that these parameters are either implicitly or explicitly described by the supplied metadata: [[User:Boesecke | SX_parametrization]].

----

=== Guide to how to make implementation easy. ===
=== Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.===
=== Provide a plan for presentation at SAS 2012 ===

[[Category: canSAS 2012]]

2012 Data Discussion

2012-07-27T17:46:36Z

Andyfaff: /* Suggest format framework */

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments in the appropriate section below:

= Agenda with Discussion =

== 1D Format ==
=== Agree a proposed extension of the current 1D standard ===
* This is an extension proposed by Steve King (ISIS) that is required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files
** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
=== Agree resultant changes to the "official" schema/stylesheet ===
{{Comment|SMK|For clarity, this extension was proposed ''after'' adoption of the existing version of the standard. Ratification of this extension at CanSAS-2012 would give facilities/software developers the necessary confidence to implement it.}}

{{Comment | SMK | This proposed extension would allow more-or-less any non-I(Q) data to be included in a CanSAS-1D data file under a suitable foreign namespace. That is remarkable flexibility, and it does not require any significant revision of the existing standard. However, it could be argued that certain types of non-I(Q) data are so integral to a SAS experiment that they should instead be given explicit SASXML tags within the standard? Transmission data is a good example of this: the existing standard explicitly includes a tag for the transmission value from a ''monochromatic'' measurement (SASsample\transmission) but makes no provision for transmission values from ''white beam'' measurements. The proposed foreign namespace extension will make such provision possible ''but only'' as an extension, not as a formal part of the standard. Is that acceptable? The downside of explicitly adding new SASXML tags to the existing standard is that it would require issuing a new version of the standard.
}}
{{Comment | ARJN | I would recommend that during this meeting that the current format then be 'parked', the new 2D format should be flexible enough to handle 1D and 2D data.
}}
{{Comment | SMK | Agreed.
}}
{{Comment | ARR | Perhaps we should be cautious about suggesting no further development for the existing 1-D format. Unless there is scope for adapting to future needs, it will be seen as not suitable for new uses. This, of course, does not imply that there have to be active plans for further changes at present.
}}
{{Comment | AJJ | As to this particular case, I suggest that since SMK and I have it done, that we accept the additional tags, call it v1.1 and move on. If other extensions arise later and become "widely" used, then we can incorporate them as they arise. This is the key benefit of using an extensible format.
}}

== 2D Format ==
=== Define minimum information necessary for reduced data ===
* definition of what '''reduced''' data is and hence the problems we should address.
* review previous discussions on 2D
* considerations specific to 2D reduced data
* consider forward looking issues such as
** grazing incidence
** event mode analysis

=== Suggest format framework ===
* NeXus extension, canSAS 1D extension, other, with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
** compare with NeXus definitions for:
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml

----
* James Hester (an expert in the CIF format) has compelling ideas on the subject of common data files. [[CIF format notes Hester]]

* Andrew Nelson has commented via email exchange with Adrian Rennie : [[Email ARJN ARR | Notes on Reduced Data formats with examples from reflectometry]]

*Having looked at both those Nexus definitions it is not clear to me how:
** multiple detectors are handled.
** What happens if the detector pixels are not grid shaped?
** how does one stuff multiple datasets (e.g. from event mode acquisition)?
Remember, all we need is instrument independent information in this file, nothing else.
:[[User:Andyfaff|Andyfaff]] 12:45, 27 July 2012 (CDT)

----

* Ron Ghosh has laid out details of a proposal to use HDF5/Nexus : [[HDF5 Notes Ghosh]]

----

* Peter Boesecke misses definitions of parameters that are needed to describe small-angle and wide-angle scattering experiments. Parameters like "Distance", "Center", "Rotation" have only a meaning when they are part of a geometrical model, otherwise they are useless. He has collected parameters that are required for SAS/WAS experiments with position sensitive detectors (1D and 2D). When saving experimental data it should be checked that these parameters are either implicitly or explicitly described by the supplied metadata: [[User:Boesecke | SX_parametrization]].

----

=== Guide to how to make implementation easy. ===
=== Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.===
=== Provide a plan for presentation at SAS 2012 ===

[[Category: canSAS 2012]]

2012 Data Discussion

2012-07-27T17:45:30Z

Andyfaff: /* Suggest format framework */

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments in the appropriate section below:

= Agenda with Discussion =

== 1D Format ==
=== Agree a proposed extension of the current 1D standard ===
* This is an extension proposed by Steve King (ISIS) that is required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files
** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
=== Agree resultant changes to the "official" schema/stylesheet ===
{{Comment|SMK|For clarity, this extension was proposed ''after'' adoption of the existing version of the standard. Ratification of this extension at CanSAS-2012 would give facilities/software developers the necessary confidence to implement it.}}

{{Comment | SMK | This proposed extension would allow more-or-less any non-I(Q) data to be included in a CanSAS-1D data file under a suitable foreign namespace. That is remarkable flexibility, and it does not require any significant revision of the existing standard. However, it could be argued that certain types of non-I(Q) data are so integral to a SAS experiment that they should instead be given explicit SASXML tags within the standard? Transmission data is a good example of this: the existing standard explicitly includes a tag for the transmission value from a ''monochromatic'' measurement (SASsample\transmission) but makes no provision for transmission values from ''white beam'' measurements. The proposed foreign namespace extension will make such provision possible ''but only'' as an extension, not as a formal part of the standard. Is that acceptable? The downside of explicitly adding new SASXML tags to the existing standard is that it would require issuing a new version of the standard.
}}
{{Comment | ARJN | I would recommend that during this meeting that the current format then be 'parked', the new 2D format should be flexible enough to handle 1D and 2D data.
}}
{{Comment | SMK | Agreed.
}}
{{Comment | ARR | Perhaps we should be cautious about suggesting no further development for the existing 1-D format. Unless there is scope for adapting to future needs, it will be seen as not suitable for new uses. This, of course, does not imply that there have to be active plans for further changes at present.
}}
{{Comment | AJJ | As to this particular case, I suggest that since SMK and I have it done, that we accept the additional tags, call it v1.1 and move on. If other extensions arise later and become "widely" used, then we can incorporate them as they arise. This is the key benefit of using an extensible format.
}}

== 2D Format ==
=== Define minimum information necessary for reduced data ===
* definition of what '''reduced''' data is and hence the problems we should address.
* review previous discussions on 2D
* considerations specific to 2D reduced data
* consider forward looking issues such as
** grazing incidence
** event mode analysis

=== Suggest format framework ===
* NeXus extension, canSAS 1D extension, other, with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
** compare with NeXus definitions for:
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml

*Having looked at both those Nexus definitions it is not clear to me how:
** multiple detectors are handled.
** What happens if the detector pixels are not grid shaped?
** how does one stuff multiple datasets (e.g. from event mode acquisition)?
Remember, all we need is instrument independent information in this file, nothing else.
:[[User:Andyfaff|Andyfaff]] 12:45, 27 July 2012 (CDT)

----
* James Hester (an expert in the CIF format) has compelling ideas on the subject of common data files. [[CIF format notes Hester]]

* Andrew Nelson has commented via email exchange with Adrian Rennie : [[Email ARJN ARR | Notes on Reduced Data formats with examples from reflectometry]]

----

* Ron Ghosh has laid out details of a proposal to use HDF5/Nexus : [[HDF5 Notes Ghosh]]

----

* Peter Boesecke misses definitions of parameters that are needed to describe small-angle and wide-angle scattering experiments. Parameters like "Distance", "Center", "Rotation" have only a meaning when they are part of a geometrical model, otherwise they are useless. He has collected parameters that are required for SAS/WAS experiments with position sensitive detectors (1D and 2D). When saving experimental data it should be checked that these parameters are either implicitly or explicitly described by the supplied metadata: [[User:Boesecke | SX_parametrization]].

----

=== Guide to how to make implementation easy. ===
=== Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.===
=== Provide a plan for presentation at SAS 2012 ===

[[Category: canSAS 2012]]

CIF format notes Hester

2012-07-27T16:59:52Z

Andyfaff: Created page with "The datamodel: key to developing a SAS data transfer framework Hester, J. R. “Data” in a broad sense includes both the core measurements and the descriptive information s..."

The datamodel: key to developing a SAS data transfer framework
Hester, J. R.

“Data” in a broad sense includes both the core measurements and the descriptive information surrounding those measurements. In order to transfer data effectively, both parties must agree on (I) the electronic representation of the information (commonly referred to as the “data format”) and (II) the precise meanings and attributes assigned to the data items encapsulated in the format. While the former requires little more than choosing from any number of off-the-shelf format specifications, the latter requires broad agreement on unambiguous definitions. At the interface between the data format of (I) and the data definitions of (II) lies the data model: this is simultaneously the abstract data structure produced by the data format parser, and the abstract data description that the authors of definitions must link back to the scientific domain. A complex data model will deter members of the scientific community from participation in definition development. Conversely, an overly simple data model will not provide sufficient expressive power to allow computer-driven data manipulation, and may lack flexibility to handle inevitable future changes. The “sweet spot” lies at the simple end of the complexity spectrum where data models describe little more than collections of key-value pairs or collections of data tables. The currently popular hierarchical data formats encourage complexity for little, if any, gain.
Data transfer protocols benefit from a set of standards for judging dataset quality. The standards provide ready-made data definitions, and journals and referees can require that submissions meet the standards, driving adoption of a data framework that can incorporate information required by the journals at each stage of data processing.
Requirements for a successful SAS data framework therefore become: (i) a set of standards for SAS data; (ii) a datamodel that is just complex enough to efficiently express this data and potential future data; (iii) one or more formats that can be transformed to and from the target datamodel.

2012 Data Discussion

2012-07-27T16:59:09Z

Andyfaff: /* Suggest format framework */

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments in the appropriate section below:

= Agenda with Discussion =

== 1D Format ==
=== Agree a proposed extension of the current 1D standard ===
* This is an extension proposed by Steve King (ISIS) that is required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files
** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
=== Agree resultant changes to the "official" schema/stylesheet ===
{{Comment|SMK|For clarity, this extension was proposed ''after'' adoption of the existing version of the standard. Ratification of this extension at CanSAS-2012 would give facilities/software developers the necessary confidence to implement it.}}

{{Comment | SMK | This proposed extension would allow more-or-less any non-I(Q) data to be included in a CanSAS-1D data file under a suitable foreign namespace. That is remarkable flexibility, and it does not require any significant revision of the existing standard. However, it could be argued that certain types of non-I(Q) data are so integral to a SAS experiment that they should instead be given explicit SASXML tags within the standard? Transmission data is a good example of this: the existing standard explicitly includes a tag for the transmission value from a ''monochromatic'' measurement (SASsample\transmission) but makes no provision for transmission values from ''white beam'' measurements. The proposed foreign namespace extension will make such provision possible ''but only'' as an extension, not as a formal part of the standard. Is that acceptable? The downside of explicitly adding new SASXML tags to the existing standard is that it would require issuing a new version of the standard.
}}
{{Comment | ARJN | I would recommend that during this meeting that the current format then be 'parked', the new 2D format should be flexible enough to handle 1D and 2D data.
}}
{{Comment | SMK | Agreed.
}}
{{Comment | ARR | Perhaps we should be cautious about suggesting no further development for the existing 1-D format. Unless there is scope for adapting to future needs, it will be seen as not suitable for new uses. This, of course, does not imply that there have to be active plans for further changes at present.
}}
{{Comment | AJJ | As to this particular case, I suggest that since SMK and I have it done, that we accept the additional tags, call it v1.1 and move on. If other extensions arise later and become "widely" used, then we can incorporate them as they arise. This is the key benefit of using an extensible format.
}}

== 2D Format ==
=== Define minimum information necessary for reduced data ===
* definition of what '''reduced''' data is and hence the problems we should address.
* review previous discussions on 2D
* considerations specific to 2D reduced data
* consider forward looking issues such as
** grazing incidence
** event mode analysis

=== Suggest format framework ===
* NeXus extension, canSAS 1D extension, other, with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
** compare with NeXus definitions for:
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
*** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml
----
* James Hester (an expert in the CIF format) has compelling ideas on the subject of common data files. [[CIF format notes Hester]]

* Andrew Nelson has commented via email exchange with Adrian Rennie : [[Email ARJN ARR | Notes on Reduced Data formats with examples from reflectometry]]

----

* Ron Ghosh has laid out details of a proposal to use HDF5/Nexus : [[HDF5 Notes Ghosh]]

----

* Peter Boesecke misses definitions of parameters that are needed to describe small-angle and wide-angle scattering experiments. Parameters like "Distance", "Center", "Rotation" have only a meaning when they are part of a geometrical model, otherwise they are useless. He has collected parameters that are required for SAS/WAS experiments with position sensitive detectors (1D and 2D). When saving experimental data it should be checked that these parameters are either implicitly or explicitly described by the supplied metadata: [[User:Boesecke | SX_parametrization]].

----

=== Guide to how to make implementation easy. ===
=== Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.===
=== Provide a plan for presentation at SAS 2012 ===

[[Category: canSAS 2012]]

HDF5 Notes Ghosh

2012-07-27T16:51:22Z

Andyfaff:

canSAS 2D treated data
<pre>
1. The treated data to be stored might include:

a. sequences of 1D data, measured against a scan variable, eg temperature
b. one or a sequence of 2D maps, again with possibility of scan variables
eg shear rate, temperature, tomography scan etc.
c. For neutron work the data might be recorded as multiple wave-length
slices (TOF); for X-rays these might be from an energy dispersive detector
e. the data may be measured with polarised neutrons
f. results from multiple detector configurations.

The common characteristic is the intensity, S(Q,v1,v2..), or S(Qx,Qy,v1,v2...),
the deviation in S, Q, Qx, Qy, v1... etc.

In addition, continuing from the discussions of canSAS 1D XML data, it
is useful to add titles, short titles, and a history of precursor(s)
and treatment (SASprocessnote).

Since the contents have been corrected as far as possible to remove
instrument characteristics there is no need to propagate these items
in this treated data; they remain fully described in the raw data. It
should thus be possible to merge data from different instruments directly
at this stage (so item c. might even be results from distinct instruments).

2. Storage Formats

a. The three platforms where data will be treated are PC-Windows, Macintosh-OSX,
and Linux.

b. The primary aim of the canSAS-1D format was to provide a data file which
could be automatically read by existing well known programs. The XML
design allows the files to be imported easily into Excel and to be
displayed clearly by most web browsers. The stylised ASCII text can
also be read easily without any tools on all three systems. Attribute
fields contained vital information, notably units, for stored values.

c. For potentially large data sets the text readability is much less valuable.

Navigating large tables of numbers is impracticable. This was recognised
early by the initial proponents of NeXus, notably Jon Tischler (APS).
The format proposed for NeXus was based on HDF, the Hierachical Data Format
(1990-). This stored data in system independent binary,
in a single file. For the user this was strucured superficially like
a unix file structure, with possiblities of creating links (these, for
example, allow creating a slab of a selected region of data as a
new entity, but without actually copying the data, simply linking the
remapping information to the initial data). The attraction for very
large image data with multiple parameters is evident. Consequently
a number of general data visualisation tools have been developed,
and are freely available for all three computer platforms. It was
not initially designed to have large volumes of single valued metadata.

d. The NeXus project started in 1995 with the aim of standardising storage
of Neutron and Xray scattering data. The file format chosen was HDF,
based on existing tools and generic visualisation software, but only
using a very limited subset for simplicity. A hierachical dictionary of
class names was created and these were attached to the hierachical data
items as attributes. The datafile components were hence navigable through these
class names (see appendix 1). The files are however standard HDF files.

e. In 2003 the increasing limitations (complexity, lack of text methods, etc )
of the HDF file format (versions 1 to 4) were finally overcome witha complete
change of internal structure, creating HDF5. The existing petabytes of data
in HDF4 compatible format requires continued maintenance of the basic toolset,
but all new efforts were directed to HDF5. The NeXus project too had
lead to considereable amounts of data being stored in the HDF4 format,
and the NeXus programmers have striven to create a library which
is transparent to the type of data file involved. This is done at
a cost of some complexity.

f. For canSAS-2D treated data the need to map the metadata between instruments
has been removed; there are few reasons to follow the guidelines for writing
a NeXus compatible file. There are problems of installing software which
are dependent on other components, which themselves have dependencies
(see appendix 2).

3. Proposal for HDF5 based data format for canSAS-2D

a. The necessary identifiers for treated data were discussed for canSAS-1D.
The majority of items were optional.
It is necessary to extend the information concerning the actual mapped
data for canSAS-2D.

b. The prescription should be extensible. Space requirements are less important
than ease of access, dumping sections and browsing.

c. The scattering data could be stored as a sequence of data items or
It would be useful for the first data component in a group to be a
2-D plottable array of corrected intensity, or simply identify with the NeXus attribute
of Signal=1. Another option option might be
to interpolate this (simple bi-linear, or more sophisticated splines)
onto a regular grid linear in x and y. The plots would then be easier
to compare and merge.
Additional data items could include scan variables, short identifier
titles etc.

d. In addition to the intensities there could be additional similar maps
of intensity deviation, x-deviation, y-deviation, etc. The notion of
links allows the other data items from the first map to be included
without needing copies, hence each component appears similar in "shape".

e. The data could be stored additionally as a sequence of tuples of
(S, S_dev, x, x_dev, y, y_dev, polarisation, etc..) This would allow
different measurements to be sorted and merged, for example in TOF
measurements several wavelength bands could be saved separately.

With the data it is also useful to include the component information
and treatment information. An attribute can contain muli-line text easily.

f. Since a new set of classes would have to be created for NeXus anyway
there seems little point in using/maintaining another package to
read/write data. It is easy to decorate the hdf5 file with the
few attributes expected in a NeXus file.

a simple structure might have the following layout

canSAS-2D
Data
S(Qx,Qy) (say 128x128,10,2)
Attributes X="Qx_scale", Y="Qy_scale", Units="1/cm", Scan1="Temperature"
Scan2="Polarisation", Legend="short_title", Interpolation="None"
Process="multi-line analysis summary", Signal=1
Sdev(Qx,Qy) (128,128,10,2)
Qx_scale (128)
Qx_scale_dev (128)
Qy_scale (128)
Qy_scale_dev(128)
Temperature(10)
Polarisation(2)

Sample
Main_Title

4. Data browsing for HDF files

Browsers can open files and restructure internal components; most importantly
they can not only list the contents of data fields, but usually have several
plotting options for 1D cuts, 2D maps etc.

They include:
HDFView http://www.hdfgroup.org/hdf-java-html/hdfview/
HDFExplorer (semi-commercial product $39)
ISAW (?)
PyMca http://pymca.sourceforge.net

HDF5 Data manipulation
h5py http://code.google.com/p/h5py
The HDF5 library includes working interfaces and examples for C, C++,
F77, F90 and java
The HDF5 library is incorporated into Mathlab, IDL etc.

The NeXus library is built on top of HDF5 and has support for python
and C. The fortran90 works only on Windows.

Examples

(One weakness of the NeXus project was the paucity of examples, hence
the divergence in local implementations.)

Typical example files might include
- simple SAS from radially symmetric data to check interpolation procedures
etc.
- GSAS having a pattern offset from nominal detector centre

etc.

Appendix 1 Annotated summary of NeXus raw data file

Structure of Nexus HDF5 file from HDFView

top level 054289.nxs
group size 1
4 attributes
HDF5_Version = 1.8.3
NeXus_Version = 4.2.0
file_name = /users/data/054289.nxs
file_time = 2012-04-19T11:12:34+01:00

entry0
Group size = 14
Number of attributes = 1
NX_class = NXentry ! standard starting point for
! NeXus files

D22 ! instrument name
Group size = 14
Number of attributes = 1
NX_class = NXinstrument !instrument components & values

BS
Group size = 12
Number of attributes = 1
NX_class = NXbeamstop
bx_actual
32-bit floating-point
Number of attributes = 0
Value = 1.81 ! (no units!)

bx_offset etc
Detector
Group size = .....

etc. for each attenuator, collimation, selector ...
data
Group size = 1
Number of attributes = 1
NX_class = NXdata
data
32-bit integer, 128 x 128 x 1
Number of attributes = 1
signal = 1 ! shows plottable data entity
value table[,,]

sample
Group size = 20
Number of attributes = 1
NX_class = NXsample
temperature
32-bit floating-point
Number or attributes = 0
san_actual ! sample rotation angle
32-bit floating-point
Number or attributes = 0

Appendix 2 installing and running programs on different systems using gcc

The dependencies for hdf based software are illustrated below.

Macintosh OSX, Leopard 10.5.8 (2008) with Xcode, gfortran, gcc 4.2.3
Linux: Fedora Core-11 (2009) gfortran, gcc 4.4

HDFView 2.7 for pre-2010 systems, v2.8 current
HDF5 version 1.8.5 for pre-2010 systems..currently 1.8.9
h5py 1.3.1.tar.gz (requires python 2.5-2.6, HDF5 1.6.5 to 1.8.5)
requires HDF5 built without Fortran support (needs shared libraries)
NeXus-4.2.1
Notes: requires mxml package, mxml-2.7.tar.gz

Several binary packages for NeXus (.dmg, .rpm failed dependencies ) all
finally rebuilt from source.

To build each component requires inspecting the INSTALL information
to create a suitable set of libraries. The HDF5 package built
easily though one of the checks in the x86_64 linux package stopped
the system (large number test).

Windows - installation binaries Windows-XP for MinGW
zlib- built by MSYS
hdf5-1.8.5 built by MSYS -- manually: -lws2_32 added to link library list
(a binary distribution exists for Intel compilers)
python-2.7.3.msi
numpy-1.6.2.win32-py2.7.exe
h5py-2.0.1.win32-py2.7.msi
NeXus-4.2.0.zip requires mxml
mxml mxml-2.7.tar.gz hand-built libmxml.a (without MSYS)

There are severe problems in using the most recent versions perhaps
linked to the changes for the x86_64 architecture. The pre-built
NeXus packages all depend on the hdf4 and hdf5 and mxml libraries.
The last would require installing the MSYS package, or hand-building.

Appendix 3 Summary of test file cd2_050506_001.h5

Data are from mondisperse spheres (A. Rennie) D22, run 50506+

h5dump -n cd2_050506_001.h5

HDF5 "cd2_050506_001.h5" {
FILE_CONTENTS {
group /
group /canSAS2D
group /canSAS2D/ASample
dataset /canSAS2D/ASample/Title
group /canSAS2D/Data
dataset /canSAS2D/Data/Qx
dataset /canSAS2D/Data/Qy
dataset /canSAS2D/Data/S
dataset /canSAS2D/Data/Sdev
}
}
showing the structure in more detail with NeXus decorations:

h5dump -A cd2_050506_001.h5

HDF5 "cd2_050506_001.h5" {
GROUP "/" {
GROUP "canSAS2D" {
GROUP "ASample" {
ATTRIBUTE "NXclass" {
DATATYPE H5T_STRING {
STRSIZE 8;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "NXsample"
}
}
DATASET "Title" {
DATATYPE H5T_STRING {
STRSIZE 50;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
GROUP "Data" {
DATASET "Qx" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 128 ) / ( 128 ) }
ATTRIBUTE "Units" {
DATATYPE H5T_STRING {
STRSIZE 3;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "1/A"
}
}
}
DATASET "Qy" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 128 ) / ( 128 ) }
ATTRIBUTE "Units" {
DATATYPE H5T_STRING {
STRSIZE 3;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "1/A"
}
}
}
DATASET "S" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 128, 128 ) / ( 128, 128 ) }
ATTRIBUTE "Interpolation" {
DATATYPE H5T_STRING {
STRSIZE 4;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "None"
}
}
ATTRIBUTE "NXclass" {
DATATYPE H5T_STRING {
STRSIZE 6;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "NXdata"
}
}
ATTRIBUTE "Process" {
DATATYPE H5T_STRING {
STRSIZE 80;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 5 ) / ( 5 ) }
DATA {
(0): "Created by apl8 27-Jun-2012 22:01:09 MASK: m12a.msk ",
(1): " AvA1 0.0000E+00 AsA2 8.2300E-01 XvA3 0.0000E+00 XsA4 8.2300E-02 XfA5 0.0000E+00",
(2): "S... 50506 0 6.80E+02 sple A 0.4% Sbak 50505 0 6.79E+02 MT cell ",
(3): "Cd/E 50510 0 3.40E+02 blocked beam ",
(4): " "
}
}
ATTRIBUTE "Signal" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 1
}
}
ATTRIBUTE "Units" {
DATATYPE H5T_STRING {
STRSIZE 4;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "1/cm"
}
}
ATTRIBUTE "x_scale" {
DATATYPE H5T_STRING {
STRSIZE 2;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "Qx"
}
}
ATTRIBUTE "y_scale" {
DATATYPE H5T_STRING {
STRSIZE 2;
STRPAD H5T_STR_SPACEPAD;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "Qy"
}
}
}
DATASET "Sdev" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 128, 128 ) / ( 128, 128 ) }
}
}
}
}
}

python h5toText.py cd2_050506_001.h5
cd2_050506_001.h5
canSAS2D
ASample
@NXclass = ['NXsample']
Title:char[50][] = __array
__array =
Data
Qx:float32[128] = __array
@Units = ['1/A']
__array = [-0.0093729430809617043, -0.0091350506991147995, -0.0088971592485904694, '...', 0.020839333534240723]
Qy:float32[128] = __array
@Units = ['1/A']
__array = [-0.015177506022155285, -0.01493961364030838, -0.01470172218978405, '...', 0.015034771524369717]
S:float32[128,128] = __array
@NXclass = ['NXdata']
@x_scale = ['Qx']
@y_scale = ['Qy']
@Interpolation = ['None']
@Units = ['1/cm']
@Signal = [1]
@Process = ['Created by apl8 27-Jun-2012 22:01:09 MASK: m12a.msk'
' AvA1 0.0000E+00 AsA2 8.2300E-01 XvA3 0.0000E+00 XsA4 8.2300E-02 XfA5 0.0000E+00'
'S... 50506 0 6.80E+02 sple A 0.4% Sbak 50505 0 6.79E+02 MT cell'
'Cd/E 50510 0 3.40E+02 blocked beam' '']
__array = [
[0.0, 0.0, 0.0, '...', 0.0]
[0.0, 0.0, 0.0, '...', 0.0]
[0.0, 0.0, 0.0651400014758, '...', 0.0]
...
[0.0, 0.0, 0.0, '...', 0.0]
]
Sdev:float32[128,128] = __array
__array = [
[0.0, 0.0, 0.0, '...', 0.0]
[0.0, 0.0, 0.0, '...', 0.0]
[0.0, 0.0, 0.0420599989593, '...', 0.0]
...
[0.0, 0.0, 0.0, '...', 0.0]
]

The file is easily read and dumped by h5dump,
and may be plotted with HDFView, and PyMCA.

The file size is 140768 bytes for 128x128 data and errors
The ASCII orginal data are 370694 bytes
</pre>

The example data file may be obtained from

ftp://ftp.ill.fr/pub/cs/reg/canSAS2D/cd2_050506_001.h5

:[[User:Ron Ghosh|Ron Ghosh]] 10:51, 2 July 2012 (CDT)

We have had quite a serious issue with HDF5 - it has often issues when you are trying to read and write from NFS mounted network drives. For example, your instrument acquisition is trying to write a (read only) datafile to a network drive. However, you are trying to access it at the same time. Sometimes this leads to data corruption in the HDF5 file. This may not immediately be an issue for reduced datasets, but it is still a deficiency of the hdf5 approach.
I do like the h5py approach to accessing for hdf5 files. However, at the end of the day hdf5 is binary. And it is hierarchical - you simply cannot change much about the file format once it is written. Plus, binary is simply a step too far for me. It takes me away from the data and I don't want that. I remember we had a big discussion at the last canSAS meeting about the iron rule that it had to be readable in Excel. I'm pretty sure you can't read HDF5 from Excel. Any format that is decided on simply has to be able to cope with 1D, 2D and all the other bells and whistles. I don't think that HDF5 gives the user the easy access they need to the data, nor does it give any flexibility for future changes.
:[[User:Andyfaff|Andyfaff]] 11:51, 27 July 2012 (CDT)

2012 Data Discussion

2012-06-08T00:38:34Z

Andyfaff:

*The following is the agenda of work posted under business for [[canSAS-2012]]. Please add comments here:
** '''1D Format'''
** agree a proposed foreign namespace extension of the current 1D standard (required to enable, for example, t-o-f instruments to store auxillary wavelength-dependent non-I(Q) data in the same output files)
*** see Section 1.6.5 in [http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.0/doc/cansas-1d-1_0-manual.pdf?format=raw here]
*** and look at this [http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml example]
*** or [http://www.smallangles.net/wgwiki/images/f/f5/ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML this] one
** agree resultant changes to the schema/stylesheet
ARJN - I would recommend that during this meeting that the current format then be 'parked', the new 2D format
should be flexible enough to handle 1D and 2D data
----

** '''2D Format'''
** define minimum necessary for reduced data
*** review previous discussions on 2D
*** considerations specific to 2D reduced data
*** consider forward looking issues such as
**** grazing incidence
**** event mode analysis
** suggest format framework (NeXus extension, canSAS 1D extension, other) with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
*** compare with NeXus definitions for:
**** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
**** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']
** Guide to how to make implementation easy.
** Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.
** Provide a plan for presentation at SAS 2012

----
* A proposal for a 2D version of the canSAS 1D data format has been posted for discussion: [[2D Data Format Proposal]]
* NeXus has a definition for multi-dimensional data:
** http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas
** http://svn.nexusformat.org/definitions/trunk/applications/NXsas.nxdl.xml
----

----

== Discussion ==

[[Category: canSAS 2012]]

canSAS-2012

2012-03-15T04:01:08Z

Andyfaff:

__TOC__

Meeting of canSAS working groups

== Purpose ==

Work on specific matters important to the canSAS community as decided in previous conference calls.
It is expected that those attending this working session will already have strong foundation and
understanding and will come ready with knowledge, tools, and ideas to make significant progress
during the event. The specific projects and tasks, as well as the list of participants, are
described on this wiki and will be revised as the event draws near.

== Outcomes ==

* 2-d data format
* A plan for standards
* Global SAS Web Portal
* presentation at SAS2012

== Dates ==

* 2012 July 28 - 31 (Saturday - Tuesday, inclusive)

== Meeting Location ==

Uppsala University
 Building: Angstrom Laboratory [http://www.polacksbacken.uu.se/Valkommen_till_Polacksbacken/Polacksbackens+historia/Angstromlaboratoriet/] Room(s): TBA

== Schedule ==

* arrive the previous day (Friday, 2012-07-27)
* conference check-in first morning (Saturday, 2012-07-28), location TBA
* meet all day (Sunday, 2012-07-29), location TBA
* meet all day (Monday, 2012-07-30), location TBA
* meet all day (Tuesday, 2012-07-31), location TBA
* checkout next day (Wednesday, 2012-08-01)

=== Daily Agenda ===

Meeting room is TBA

Here is the daily schedule (tentative):

{| border="1"
|-
! time || description || comments
|-
| 8:45 - 10:00 AM
| align="center" | plenary
| align="center" | previous day's report and question
|-
| 10:15 - 12:30 PM
| align="center" | morning session
| align="center" | work in groups
|-
| 12:30 - 2:00 PM
| align="center" | lunch
| align="center" | TBA
|-
| 2:00 - 4:30 PM
| align="center" | afternoon session
| align="center" | work in groups
|-
| 4:30 - 4:45 PM
| align="center" | plenary
| align="center" | tea, coffee
|-
| 4:45 - 6:30 PM
| align="center" | plenary
| align="center" | reports and discussions
|-
| 6:30 - 8:00 PM
| align="center" | dinner
| align="center" | TBA
|-
| 8:00 PM - ?
| align="center" | evening discussions
| align="center" | ad hoc
|}

== Planning ==

=== Business Matters ===

* resolve remaining ISIS 1D issue (probably should be done PRIOR to meeting)
ARJN - I would recommend that during this meeting that the current format be 'parked', the new 2D format should be flexible enough to handle 1 and 2D data.
* Enunciate purpose and benefits of a standard ''reduced data'' format
** define minimum necessary for reduced data
*** review previous discussions on 2D
*** considerations specific to 2D reduced data
*** consider forward looking issues such as
**** grazing incidence
**** event mode analysis
** suggest format framework (NeXus extension, canSAS 1D extension, other) with brief discussion of the reason for the choice (including options considered, pros and cons of each, and final weighing)
*** compare with NeXus definitions for:
**** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXsas '''NXsas''']
**** [http://download.nexusformat.org/doc/html/ClassDefinitions-Application.html#NXiqproc '''NXiqproc''']
** Guide to how to make implementation easy.
** Create straw format suitable for test/demonstration use and convert some test data for such a demonstration.
** Provide a plan for presentation at SAS 2012
* Standards
** Define purpose/goals
*** (e.g. QA/QC,
*** improvement in uncertainties of measurements in general,
*** helping each facility continuously improve
*** etc.)
** what types of tests are interesting/important
*** (e.g. beam intensity standards - there are several different ways to quantify-
*** ? resolution?
*** abs intensity calibration,
*** Q calibration,
*** etc. etc,
** other contribution issues -
*** inelastic,
*** multiple scattering,
*** wavelength contamination,
*** detector efficiencies at diff wavelengths etc..;
*** limits in signal to noise - how weak a signal can be reliable extracted;
*** etc etc etc)
** written plan for long term sustaining of effort
*** (assuming “ad-hoc” projects how to seed them,
*** how frequently, and
*** how to disseminate/share results including “advertizing:” project
** Define list of 2 or 3 projects to work on in the near term including plan of action and participants for each.
** plan for presentation at SAS 2012 (could be just to announce plan and see who wants to participate?)
* Global SAS Web Portal
** define scope, purpose, and goal of portal
** list content type to which such a portal should give access.
** Suggest method for hosting
*** ?more distributed or more centralized,
*** ?under auspices of a particular facility or SAS commission
*** ? .. or both etc)
** Build a working straw landing page prototype
** Build at least 2 or 3 subpages and/or designs on paper
** plan for presentations
*** SAS 2012
*** SAS commision

=== Projects and Tasks ===

These items are sorted in order of priority that they be finished.

# 2-D format
# Standardization
# SAS web portal
# ...

=== Presentations ===

# the canSAS 1D, lessons learned
# ...

== Practical Matters ==

=== Registration ===

TBA

=== Hotel ===

Clarion Hotel Gillet, Uppsala
 Dragarbrunnsgatan 23,
 753 20 Uppsala, Sweden

*Expect hotel rate to be 570 SEK for 1 Single Bed (approximate, subject to change)
** Please reserve your own rooms directly with the hotel (website, phone, ...)

Please reserve your own rooms directly with the hotel:
;Web site: http://www.clarionhotelgillet.com/
;telephone: +46 (0) 18 68 18 00
;fax: +46 (0) 18 68 18 18
;e-mail: cl.uppsala@choice.se
;[http://maps.google.com/maps/place?q=Clarion+Hotel+Gillet,+Uppsala++Dragarbrunnsgatan+23,+753+20+Uppsala,+Sweden&hl=en&cid=6476341721226749910 map reference]:

=== Directions and Maps ===

* map to hotel: [http://maps.google.com/maps?saddr=Stockholm+Arlanda+(ARN):+international+airport&daddr=Uppsala,+Sweden&hl=en&sll=59.745326,17.796478&sspn=0.290609,0.701065&geocode=FRE_jgMdbLsRASG2Z_q9nBX8JQ%3BFYRekQMdDyYNASmNqzKF-8tfRjEHCIKFCulPqg&gl=us&dirflg=r&ttype=now&noexp=0&noal=0&sort=def&mra=prev&t=h&z=11&start=0 google directions from airport]
* airport: Stockholm Arlanda (ARN): international
* taxi: Uppsala Taxi +4618 100000
* train: [http://maps.google.com/maps?q=type:transit_station:%22Uppsala+Centralstation%22&hl=en&ll=59.858518,17.645931&spn=0.004525,0.010954&sll=59.858534,17.646086&sspn=0.000000,0.000000&view=map&ftid=0x465fcbf9a0d697b1:0x1901cc46b512aff6&ftt=321&geocode=FWZekQMdBkINAQ&hnear=Uppsala+Centralstation&t=h&z=17 Trains] from airport to Uppsala take about 20 minutes. Frequency every 30 minutes. [http://maps.google.com/maps/place?q=type:transit_station:%22Uppsala+Centralstation%22&hl=en&ftid=0x465fcbf9a0d697b1:0x1901cc46b512aff6 map reference]

=== Host ===

* Adrian Rennie
** institute: Uppsala University
** email: Adrian.Rennie@fysik.uu.se
** office phone: +46184713596

=== Costs ===

* Conference fee: no fee
* regular meals (breakfast, lunch, dinner) will be traveler's expense

=== Equipment ===

{| border="1"
|-
! item || quantity || who provides?
|-
| projector & screen
| align="center" |  
| align="center" | host
|-
| LCD displays, keyboards, & mice
| align="center" |  
| align="center" | host
|-
| electrical power extension cords
| align="center" |  
| align="center" | host
|-
| foreign electrical power adapter
| align="center" | as needed
| align="center" | traveler
|-
| Wi-Fi connections
| align="center" | sufficient
| align="center" | host venue
|}

== Participants ==

{| border="1"
|-
! participant || affiliation || arriving || departing || registration? || other
|-
| Adrian Rennie
| align="center" | [http://www.uu.se Uppsala University]
| align="center" | host
| align="center" | host
| align="center" | host
| align="center" |  
|-
| Paul Butler
| align="center" | [http://www.nist.gov NIST]
| align="center" | 2012-07-27
| align="center" | 2012-08-01
| align="center" | tba
| align="center" |  
|-
| Andrew Jackson
| align="center" |
| align="center" | 2012-07-27
| align="center" | 2012-08-01
| align="center" | tba
| align="center" |  
|-
| Pete Jemian
| align="center" | [http://www.aps.anl.gov APS]
| align="center" | 2012-07-27
| align="center" | 2012-08-01
| align="center" | tba
| align="center" |  
|-
| Steve King
| align="center" | ISIS
| align="center" | 2012-07-27
| align="center" | 2012-08-01
| align="center" | tba
| align="center" |  
|-
|}

== Final Report ==

-tba-

cansas1d.xml

2007-12-24T01:40:59Z

Andyfaff: /* XML template: cansas1d.xml */

=XML template: cansas1d.xml=
--[[User:Jemian|Jemian]] 11:49, 21 December 2007 (EST)
 
(cansas1d.xml has been validated against [[cansas1d_xsd |cansas1d.xsd]])

<pre>
<?xml version="1.0"?>
<SASroot xmlns="http://www.smallangles.net/cansas1d"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.smallangles.net/cansas1d cansas1d.xsd"
version="0.1a">
<SASentry>
<Title></Title>
<Run></Run>
<SASdata>
<Idata>
<Q unit="1/A">0.02</Q>
1000
<Qdev unit="1/A">0.01</Qdev>
<Idev unit="1/cm">3</Idev>
<Qfwhm unit="1/A"></Qfwhm>
<Qmean unit="1/A"></Qmean>
<Shadowfactor></Shadowfactor>
</Idata>
</SASdata>
<SASsample>
<ID>SI600-new-long</ID>
<thickness unit="mm">1.03</thickness>
<transmission>0.327</transmission>
<temperature unit="C">0.0000</temperature>
<position>
<x unit="mm">10.00</x>
<y unit="mm">0.00</y>
</position>
<orientation>
<roll unit="degree">22.5</roll>
<pitch unit="degree">0.020</pitch>
</orientation>
<details>

http://chemtools.chem.soton.ac.uk/projects/blog/blogs.php/bit_id/2720
</details>
</SASsample>
<SASinstrument>
<name>canSAS instrument</name>
<SASsource>
<radiation>neutron</radiation>
<beam_size>
<x unit="mm">12.00</x>
<y unit="mm">12.00</y>
</beam_size>
<beam_shape>disc</beam_shape>
<wavelength unit="A">6.00</wavelength>
<wavelength_min unit="nm">0.22</wavelength_min>
<wavelength_max unit="nm">1.00</wavelength_max>
<wavelength_spread unit="percent">
14.3
</wavelength_spread>
</SASsource>
<SAScollimation>
<distance unit="m">11.000</distance>
<aperture name="source" type="radius">
<size>
<x unit="mm">50</x>
</size>
</aperture>
<aperture name="sample" type="radius">
<size>
<x unit="mm">0</x>
</size>
</aperture>
</SAScollimation>
<SASdetector>
<name>fictional hybrid</name>
<SDD unit="m">

4.150
</SDD>
<orientation>
<roll unit="degree">0.00</roll>
<pitch unit="degree">0.00</pitch>
<yaw unit="degree">0.00</yaw>
</orientation>
<beam_center>
<x unit="mm">322.64</x>
<y unit="mm">327.68</y>
</beam_center>
<pixel_size>
<x unit="mm">5.00</x>
<y unit="mm">5.00</y>
</pixel_size>
</SASdetector>
</SASinstrument>
<SASprocess>
<name>spol</name>
<date>04-Sep-2007 18:35:02</date>
<term name="radialstep" unit="mm">10.000</term>
<term name="sector_width" unit="degree">180.0</term>
<term name="sector_orient" unit="degree">0.0</term>
<term name="MASK_file">USER:MASK.COM</term>
<SASprocessnote>
AvA1 0.0000E+00 AsA2 1.0000E+00 XvA3 1.0526E+03 XsA4
5.2200E-02 XfA5 0.0000E+00
</SASprocessnote>
<SASprocessnote>
S... 13597 0 2.26E+02 2A 5mM 0%D2O Sbak 13594 0 1.13E+02
H2O Buffer
</SASprocessnote>
<SASprocessnote>V... 13552 3 1.00E+00 H2O5m</SASprocessnote>
</SASprocess>
<SASprocess>
<name>NCNR-IGOR</name>
<date>03-SEP-2006 11:42:47</date>
<description />
<term name="average_type">Circular</term>
<term name="SAM_file">SEP06064.SA3_AJJ_L205</term>
<term name="BKD_file">SEP06064.SA3_AJJ_L205</term>
<term name="EMP_file">SEP06064.SA3_AJJ_L205</term>
<term name="DIV_file">SEP06064.SA3_AJJ_L205</term>
<term name="MASK_file">SEP06064.SA3_AJJ_L205</term>
<term name="ABS:TSTAND">1</term>
<term name="ABS:DSTAND" unit="mm">1</term>
<term name="ABS:IZERO">230.09</term>
<term name="ABS:XSECT" unit="mm">1</term>
</SASprocess>
<SASnote />
</SASentry>
</SASroot>
</pre>

sasXML and IGOR

2007-12-14T03:20:23Z

Andyfaff: New page: The [http://www.igorexchange.com/project/XMLutils XMLutils.xop] can be used to read and write XML data in IGOR. This means that IGOR can be used as a program to create and use sasXML file...

The [http://www.igorexchange.com/project/XMLutils XMLutils.xop] can be used to read and write XML data in IGOR. This means that IGOR can be used as a program to create and use sasXML files.

Based on the current suggested sasXML format it has been used to create a demo sasXML file, using the IGOR procedure file given below. The procedure is fairly basic, but is fairly easy to understand. It could be adapted easily to write several scans to the file.

Since the sasXML file will be fairly rigid in terms of what node names are used, a single Igor structure could be used to carry the data before it is written to XML. The consequence of this is that this could be the '''only''' file that is needed by '''any''' SAS reduction program to produce a sasXML file, i.e. IRENA, NCNRIGOR, ORNLIGOR.

sasXML file written:
[[Image:ARJNtranscribeAJJ.xml]]

IGOR procedure file used:
[[Image:NISTsasxml.ipf]]

Data Formats Working Group

2007-12-14T02:59:21Z

Andyfaff: /* Considerations */

[http://smallangles.net/pipermail/cansas-1dwg_smallangles.net/ Mailing List Archive]

===Timeline===
* 2007-12-31 agree on v1.0 format
* 2008-01-01 start implementing v1 at facilities
* 2008-06 representative sampling of data available for inter-facility comparison
* 2008-10 presentation of results at NOBUGS2008 meeting (date TBA)

===Considerations===
* a key point of what we discussed at NIST:
namely that our goal is to agree a format which that whilst using as much best XML practice as is reasonable, leaves the file instantly human-readable, editable in the simplest of editors, and importable by simple text import filters in programs that don't recognise the XML.
* document what we decide
**1DWG will take care of documenting the format it defines.
*** make that definition with a schema (for absolute validation of any proposed XML file against the standard)
*** instructions on how to use that schema
*** XSL style sheets to present the XML contents in various forms (also serves as examples)
*** a couple of examples
*** maybe also some words.
** move some of this discussion to
*** discussion page
*** other wiki pages
*** /dev/null after its usefulness has been exhausted
* coordinate with other communities
** [http://www.nexusformat.org NeXus (http://www.nexusformat.org)]
** reflectivity
** powder diffraction
*** [http://www.xrdml.com XRDML (http://www.xrdml.com)]
* should we consider a file naming convention?
* should we consider a SAS scan naming convention?
** sequential run number from facility
** convention set by the detector software provider
* XML representation of the I vs. Q data
** tabular format
** vector format
* general XML coding style
** readability by humans
*** with lots of computer skills
*** with rudimentary computer skills
** readability by computers
*** standard XML libraries
*** generic visualization tools
*** common software such as MS Excel or Open Office or [ftp://ftp.ill.fr/pub/cs/rxml XMLPLO source code for windows/linux/OSX86]
*** A plugin for IGOR has been written, [http://www.igorexchange.com/project/XMLutils XMLutils.xop], that can handle XML data. The [[sasXML and IGOR]] page gives details on the IGOR code required to write the prototype sasXML file.
** availability of style sheets
* scalability of XML format to 2D data?
* What is required?
* What is optional?
* Use the same tags again in similar contexts
** X,Y pairs for example, whether detector position, beam center, sample position
{| width="100%" border="1" style="border:1px solid navy; border-collapse: collapse"
|-
! width="50%" | inconsistent
! width="50%" | consistent
|-
| <pre>
<beam_size axis="x" units="mm">12.00</beam_size>
<beam_size axis="y" units="mm">12.00</beam_size>
<x0 units="mm">322.64</x0>
<y0 units="mm">327.68</y0>
<pixel_x units="mm">5.00</pixel_x>
<pixel_y units="mm">5.00</pixel_y>
</pre>
| <pre>
<beam_size axis="x" units="mm">12.00</beam_size>
<beam_size axis="y" units="mm">12.00</beam_size>
<beam_center axis="x" units="mm">322.64</beam_center>
<beam_center axis="y" units="mm">327.68</beam_center>
<pixel_size axis="x" units="mm">5.00</pixel_size>
<pixel_size axis="y" units="mm">5.00</pixel_size>
</pre>
|}

===Points for Discussion===

*Do we want to advocate/recommend particular names for particular tags; eg, SASdata, SASsample, Idata, etc.?
** which ones?
* provide for (optional) inclusion of sample prep details
* provide for (optional) inclusion of other (non-SAS) data in the XML
* Need to allow for more than a single SAS data set in one .xml file

===Other Points===
* It's not clear how to specify that multiple runs were reduced together
**(AJJ) Assuming that those multiple runs were first stored as XML then referencing the individual files would give all that back information (a la Ghosh suggestion). At NIST we take absolute I vs Q files and combine them to produce an absolute I vs Q file thus that is reasonable here. What about elsewhere?
* How does one include the instrument information of the many runs that we used to make up the composite file
* If we have reduction information, then everything needs to be in there, i.e. the run numbers for the can, the standard, the uniform field, etc.
* Information on the averaging, is it radial, sector, rectangular, etc.

===Members===
* Andrew Jackson (NIST)
* Pete Jemian (APS)
* Steve King (ISIS)
* Ken Littrell (ORNL)
* Andy Nelson (ANSTO)
* Ron Ghosh (ILL)
* Jan Ilavsky (APS)
===News/Status===

[[Category:Working Groups]]