Data Formats Working Group: Difference between revisions

From canSAS
No edit summary
 
(87 intermediate revisions by 8 users not shown)
Line 1: Line 1:
[http://smallangles.net/pipermail/cansas-1dwg_smallangles.net/ Mailing List Archive]


===Timeline===
= Overview =
* 2007-12-31 agree on v1.0 format
 
* 2008-01-01 start implementing v1 at facilities
As of early 2017, for new work, it is recommended to use the NXcanSAS (multi-dimensional)
* 2008-06 representative sampling of data available for inter-facility comparison
standard for both 1-D and n-D data to maximize the chance that other
* 2008-10 presentation of results at NOBUGS2008 meeting (date TBA)
software will be able to read your data files.
 
We strongly encourage all authors of small-angle scattering related software to add support for NXcanSAS.
 
= Latest News =
 
[[User:Smk|Steve King]]: 2024-09 : Proposals for [[NXcanSAS v1.1|NXcanSAS 1.1]] in development
 
[[User:Butler|Paul Butler]]: 2019-07 : Lots of discussion at [[canSAS-XI]] about metadata. Discussion and actions from the meeting are available [[canSAS-XI/DataFormats]]
 
= Links =
 
=== nD: NXcanSAS ===
 
standard for reduced small-angle scattering data of any dimension, using NeXus (HDF5)
 
* '''v1.1''': Proposals in development
* '''v1.0''': 2017-01-18
* '''Announced''': 2017-06-06 (at [[canSAS-IX]])
* '''Documentation''': [http://download.nexusformat.org/doc/html/classes/applications/NXcanSAS.html http://download.nexusformat.org/doc/html/classes/applications/NXcanSAS.html]
* '''Examples''': [https://github.com/canSAS-org/NXcanSAS_examples https://github.com/canSAS-org/NXcanSAS_examples]
* '''GitHub (NeXus)''': [https://github.com/nexusformat/definitions https://github.com/nexusformat/definitions]
* '''Poster''': [[Media:2017-05-09_NXcanSAS_2D_format.pdf|PDF]]
* '''CanSAS 2012 Discussion''': 2012-07-31 [[2012_Data_Discussion|Data formats discussion page]]


===Considerations===
=== 1D: cansas1d ===
* a key point of what we discussed at NIST: 
namely that our goal is to agree a format which that whilst using as much best XML practice as is reasonable, leaves the file instantly human-readable, editable in the simplest of editors, and importable by simple text import filters in programs that don't recognise the XML.
* document what we decide
**1DWG will take care of documenting the format it defines.
*** make that definition with a schema (for absolute validation of any proposed XML file against the standard)
*** instructions on how to use that schema
*** XSL style sheets to present the XML contents in various forms (also serves as examples)
*** a couple of examples
*** maybe also some words.
** move some of this discussion to
*** discussion page
*** other wiki pages
*** /dev/null after its usefulness has been exhausted
* coordinate with other communities
** [http://www.nexusformat.org NeXus (http://www.nexusformat.org)]
** reflectivity
** powder diffraction
*** [http://www.xrdml.com XRDML (http://www.xrdml.com)]
* should we consider a file naming convention?
* should we consider a SAS scan naming convention?
** sequential run number from facility
** convention set by the detector software provider
* XML representation of the I vs. Q data
** tabular format
** vector format
* general XML coding style
** readability by humans
*** with lots of computer skills
*** with rudimentary computer skills
** readability by computers
*** standard XML libraries
*** generic visualization tools
*** common software such as MS Excel  or Open Office  or [ftp://ftp.ill.fr/pub/cs/rxml XMLPLO source code for windows/linux/OSX86]
** availability of style sheets
* scalability of XML format to 2D data?
* What is required?
* What is optional?
* Use the same tags again in similar contexts
** X,Y pairs for example, whether detector position, beam center, sample position
{|  width="100%" border="1" style="border:1px solid navy; border-collapse: collapse"
|-
! width="50%" | inconsistent
! width="50%" | consistent
|-
| <pre>
<beam_size axis="x" units="mm">12.00</beam_size>
<beam_size axis="y" units="mm">12.00</beam_size>
<x0 units="mm">322.64</x0>
<y0 units="mm">327.68</y0>
<pixel_x units="mm">5.00</pixel_x>
<pixel_y units="mm">5.00</pixel_y>
</pre>
| <pre>
<beam_size axis="x" units="mm">12.00</beam_size>
<beam_size axis="y" units="mm">12.00</beam_size>
<beam_center axis="x" units="mm">322.64</beam_center>
<beam_center axis="y" units="mm">327.68</beam_center>
<pixel_size  axis="x" units="mm">5.00</pixel_size>
<pixel_size  axis="y" units="mm">5.00</pixel_size>
</pre>
|}


===Points for Discussion===
standard for 1-D reduced small-angle scattering data, using XML (text)


*Do we want to advocate/recommend particular names for particular tags; eg, SASdata, SASsample, Idata, etc.?
* v1.1: 2013-03-29
** which ones?
* v1.0: 2009-05-12
* provide for (optional) inclusion of sample prep details
* provide for (optional) inclusion of other (non-SAS) data in the XML
* Need to allow for more than a single SAS data set in one .xml file


===Other Points===
* '''Documentation''': [http://www.cansas.org/formats/canSAS1d/1.1/doc/ http://www.cansas.org/formats/canSAS1d/1.1/doc/]
* It's not clear how to specify that multiple runs were reduced together
* '''Examples''':      [https://github.com/canSAS-org/1dwg/tree/master/examples https://github.com/canSAS-org/1dwg/tree/master/examples]
(AJJ) Assuming that those multiple runs were first stored as XML then referencing the individual files would give all that back information (a la Ghosh suggestion). At NIST we take absolute I vs Q files and combine them to produce an absolute I vs Q file thus that is reasonable here. What about elsewhere?
* '''GitHub''':      [https://github.com/canSAS-org/1dwg  https://github.com/canSAS-org/1dwg]
* How does one include the instrument information of the many runs that we used to make up the composite file
* If we have reduction information, then everything needs to be in there, i.e. the run numbers for the can, the standard, the uniform field, etc.
* Information on the averaging, is it radial, sector, rectangular, etc.


===Members===
= Members =
* Andrew Jackson (NIST)
* Andrew Jackson (ESS)- Chair
* Jeff Krzywon (NIST)
* Pete Jemian (APS)
* Pete Jemian (APS)
* Steve King (ISIS)
* Steve King (ISIS)
* Ken Littrell (ORNL)
* Andrew Nelson (ANSTO)
* Andy Nelson (ANSTO)
* Ron Ghosh (ILL)
* Jan Ilavsky (APS)
* Jan Ilavsky (APS)
===News/Status===
* Lisa Debeer-Schmitt (ORNL)


= Timeline =
* 2007-12-31 agree on v1.0 format
* 2008-01-01 start implementing v1 at facilities
* 2008-06    representative sampling of data available for inter-facility comparison
* 2008-11-3  presentation of results at NOBUGS2008 meeting ([http://www.nbi.ansto.gov.au/nobugs2008 http://www.nbi.ansto.gov.au/nobugs2008])
* 2009-05-12 tagged SVN changeset 67 as v1.0 since it has been unmodified since January
* 2009-09-17 agreed that v1.1 revision should be forthcoming with corrections/additions to be identified in TRAC tickets
* 2017-01    NXcanSAS 1.0 released
* 2015-12    multidim moved to github a while ago
= Status =
[[User:Jemian|Jemian]] 11:10, 06 September 2008 (CDT) IgorPro GUI tool included in Irena [http://usaxs.xor.aps.anl.gov/staff/ilavsky/irena.html Irena] analysis suite<br />
[[User:Jemian|Jemian]] 13:08, 14 March 2008 (EDT) IgorPro tool to read cansas1d/1.0 XML files available<br />
[[User:Jemian|Jemian]] 13:38, 25 February 2008 (EST) starting to add case studies <br />
[[User:Jemian|Jemian]] 00:39, 21 December 2007 (EST) complete first draft (v0.1a) ready for comments from 1DWG <br />
[[User:Jemian|Jemian]] 12:00, 14 December 2007 (EST) draft status updated <br />
[[User:Jemian|Jemian]] 17:07, 12 May 2009 (EDT) changeset 67 tagged as v1.0
= Examples and Usage =
The following are example v1.1 compliant 1D data files containing [besides essential metadata] - right-click & save as to view, stylesheet is on the TRAC site:<br /><br />
SANS data only [[media:ISIS_SASXML_v1_1_SANS_Example.XML | ISIS_SASXML_v1_1_SANS_Example.XML]]<br />
Wavelength-dependent transmission [[media:ISIS_SASXML_v1_1_Transmission_Example.XML | ISIS_SASXML_v1_1_Transmission_Example.XML]]<br />
Wavelength-dependent monitor spectrum [[media:ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML | ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML]]<br />
Wavelength-dependent detector efficiency ratio [[media:ISIS_SASXML_v1_1_Det_Efficiency_Example.XML | ISIS_SASXML_v1_1_Det_Efficency_Example.XML]]<br />
The files can be usefully displayed in many web browsers: [[Rendering_canSAS1D_in_Web_Browsers]]


[[Category:Working Groups]]
[[Category:Working Groups]]
= Mailing List=
The mailing list is now maintained by subscribing to this Google group:
* [https://groups.google.com/forum/#!forum/cansas-dfwg https://groups.google.com/forum/#!forum/cansas-dfwg]
Please address all questions and answers to this Google group.
= Session Notes SAS2024 - Sun 2024-11-03 =
=== Topic 1: Magnetic/ Polarized SANS ===
* A new proposal for the .nxs format has been published on the wiki (see section "Latest News"), e.g. including the combination of all four polarization spin channels, as well as the necessary storage of the polarization efficiency information, the metadata, sample history, etc.
=== Topic 2: Complementary data ===
* It has to be addressed how multimodal experiments can be taken into account, i.e., we will need a data format for the storage, reading, and processing of multi-dimensional data.
=== Topic 3: Summary & To-Do ===
* Further extending the magnetic/ polarized SANS data format description?
* Complementary data description
* Advanced resolution descriptions of 1D,2D,3D resolution functions - here the question was raised if this topic should form a separate working group!
* Which techniques all have to be taken into account for multimodal experiments?
* Should a hackathon be organized? If yes, at the European Spallation Source?
=== Notes from the discussion round: ===
==== Related to magnetic/ polarized SANS ====
* Polarized beam issues - who can be contacted for this topic at which source? (This hasnt been discussed thoroughly)
* This formulism should not only be inserted for polarized neutrons, but also for polarized X-ray measurements! Note1: The proposed format is specialised on the polarization of neutrons, the proposed data format should be checked. Note2: Anyhow also a proposition for the data format of polarized X-ray measurements should be done! Should for this purpose the community of polarized neutrons and polarized X-rays work together?
* Also depolarization effects from the sample have to additionally be taken into accound! Note: Shouldnt this be part of the data analysis, at it is not per se clearly distinguishable from the polarization efficiency?
==== Related to resolution functions ====
* Important raised question: When speaking about a 3D Q-resolution function: what do we define as Qx,Qy,Qz? Do the Q-components in the standard definition have to be corrected for a projection issues due to flat detectors? Note1: Typically the measurement of Qx,Qy,Qz on a flat plane (flat detector) has to be corrected to correspond to the definition of Qx,Qy,Qz on Ewald sphere. Only these corrected Q-values should be exported for users. Note2: Actually this is not a question of correcting Q, but correcting the angles. Note3: This should be done for both, neutron and X-ray measurements! Note4: For a reliable correction one should measure and correct for a reference sample.
* In reference measurements the beam profile should be logged, as it goes hand in hand with defining the resolution function
* The question has been raised how to store/record the information on the resolution function? How can we keep the description of data formats in general easily extendable?
* The action has been raised to have a working meeting for this topic next year (2025)
* In general, the formation of a seperate working group on this topic was favoured by the audience! The relevant people should come together, and importantly, from different facilities/fields

Latest revision as of 08:52, 5 November 2024

Overview

As of early 2017, for new work, it is recommended to use the NXcanSAS (multi-dimensional) standard for both 1-D and n-D data to maximize the chance that other software will be able to read your data files.

We strongly encourage all authors of small-angle scattering related software to add support for NXcanSAS.

Latest News

Steve King: 2024-09 : Proposals for NXcanSAS 1.1 in development

Paul Butler: 2019-07 : Lots of discussion at canSAS-XI about metadata. Discussion and actions from the meeting are available canSAS-XI/DataFormats

Links

nD: NXcanSAS

standard for reduced small-angle scattering data of any dimension, using NeXus (HDF5)

1D: cansas1d

standard for 1-D reduced small-angle scattering data, using XML (text)

  • v1.1: 2013-03-29
  • v1.0: 2009-05-12

Members

  • Andrew Jackson (ESS)- Chair
  • Jeff Krzywon (NIST)
  • Pete Jemian (APS)
  • Steve King (ISIS)
  • Andrew Nelson (ANSTO)
  • Jan Ilavsky (APS)
  • Lisa Debeer-Schmitt (ORNL)

Timeline

  • 2007-12-31 agree on v1.0 format
  • 2008-01-01 start implementing v1 at facilities
  • 2008-06 representative sampling of data available for inter-facility comparison
  • 2008-11-3 presentation of results at NOBUGS2008 meeting (http://www.nbi.ansto.gov.au/nobugs2008)
  • 2009-05-12 tagged SVN changeset 67 as v1.0 since it has been unmodified since January
  • 2009-09-17 agreed that v1.1 revision should be forthcoming with corrections/additions to be identified in TRAC tickets
  • 2017-01 NXcanSAS 1.0 released
  • 2015-12 multidim moved to github a while ago

Status

Jemian 11:10, 06 September 2008 (CDT) IgorPro GUI tool included in Irena Irena analysis suite
Jemian 13:08, 14 March 2008 (EDT) IgorPro tool to read cansas1d/1.0 XML files available
Jemian 13:38, 25 February 2008 (EST) starting to add case studies
Jemian 00:39, 21 December 2007 (EST) complete first draft (v0.1a) ready for comments from 1DWG
Jemian 12:00, 14 December 2007 (EST) draft status updated
Jemian 17:07, 12 May 2009 (EDT) changeset 67 tagged as v1.0

Examples and Usage

The following are example v1.1 compliant 1D data files containing [besides essential metadata] - right-click & save as to view, stylesheet is on the TRAC site:

SANS data only ISIS_SASXML_v1_1_SANS_Example.XML
Wavelength-dependent transmission ISIS_SASXML_v1_1_Transmission_Example.XML
Wavelength-dependent monitor spectrum ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML
Wavelength-dependent detector efficiency ratio ISIS_SASXML_v1_1_Det_Efficency_Example.XML

The files can be usefully displayed in many web browsers: Rendering_canSAS1D_in_Web_Browsers

Mailing List

The mailing list is now maintained by subscribing to this Google group:

Please address all questions and answers to this Google group.

Session Notes SAS2024 - Sun 2024-11-03

Topic 1: Magnetic/ Polarized SANS

  • A new proposal for the .nxs format has been published on the wiki (see section "Latest News"), e.g. including the combination of all four polarization spin channels, as well as the necessary storage of the polarization efficiency information, the metadata, sample history, etc.

Topic 2: Complementary data

  • It has to be addressed how multimodal experiments can be taken into account, i.e., we will need a data format for the storage, reading, and processing of multi-dimensional data.

Topic 3: Summary & To-Do

  • Further extending the magnetic/ polarized SANS data format description?
  • Complementary data description
  • Advanced resolution descriptions of 1D,2D,3D resolution functions - here the question was raised if this topic should form a separate working group!
  • Which techniques all have to be taken into account for multimodal experiments?
  • Should a hackathon be organized? If yes, at the European Spallation Source?

Notes from the discussion round:

Related to magnetic/ polarized SANS

  • Polarized beam issues - who can be contacted for this topic at which source? (This hasnt been discussed thoroughly)
  • This formulism should not only be inserted for polarized neutrons, but also for polarized X-ray measurements! Note1: The proposed format is specialised on the polarization of neutrons, the proposed data format should be checked. Note2: Anyhow also a proposition for the data format of polarized X-ray measurements should be done! Should for this purpose the community of polarized neutrons and polarized X-rays work together?
  • Also depolarization effects from the sample have to additionally be taken into accound! Note: Shouldnt this be part of the data analysis, at it is not per se clearly distinguishable from the polarization efficiency?

Related to resolution functions

  • Important raised question: When speaking about a 3D Q-resolution function: what do we define as Qx,Qy,Qz? Do the Q-components in the standard definition have to be corrected for a projection issues due to flat detectors? Note1: Typically the measurement of Qx,Qy,Qz on a flat plane (flat detector) has to be corrected to correspond to the definition of Qx,Qy,Qz on Ewald sphere. Only these corrected Q-values should be exported for users. Note2: Actually this is not a question of correcting Q, but correcting the angles. Note3: This should be done for both, neutron and X-ray measurements! Note4: For a reliable correction one should measure and correct for a reference sample.
  • In reference measurements the beam profile should be logged, as it goes hand in hand with defining the resolution function
  • The question has been raised how to store/record the information on the resolution function? How can we keep the description of data formats in general easily extendable?
  • The action has been raised to have a working meeting for this topic next year (2025)
  • In general, the formation of a seperate working group on this topic was favoured by the audience! The relevant people should come together, and importantly, from different facilities/fields
Retrieved from "https://wiki.cansas.org/index.php?title=Data_Formats_Working_Group&oldid=2718"