Jemian: /* data structures */

2008-05-08T17:34:00Z

data structures

← Older revision		Revision as of 17:34, 8 May 2008
Line 53:		Line 53:
	</pre>		</pre>

	Here the '''Dell-D830''' attribute will associate ~~the~~ '''SASdata''' with the similarly named <pre><SASinstrument name="Dell-D830" /></pre> element.		Here the '''instrument="Dell-D830"''' attribute will associate this '''SASdata''' with the similarly named <pre><SASinstrument name="Dell-D830" /></pre> element.
	Similar for the '''source''' attribute: it identifies <pre><SASsource name="simulated" /></pre>		Similar for the '''source="simulated"''' attribute: it identifies <pre><SASsource name="simulated" /></pre>
	'''Run''' is used (in practice) as an attribute of a specific SASdata. (''Note that order of attributes is not important to the syntax of a well-formed XML file.'')		'''Run''' is used (in practice) as an attribute of a specific SASdata. (''Note that order of attributes is not important to the syntax of a well-formed XML file.'')

Jemian at 17:32, 8 May 2008

2008-05-08T17:32:08Z

← Older revision		Revision as of 17:32, 8 May 2008
Line 1:		Line 1:
	''Note: please sign and date your contributions. Reverse chronological order works for now. Keep this line on the top of the page.''		''Note: please sign and date your contributions. Reverse chronological order works for now. Keep this line on the top of the page.''

			= Discussion on the standard for reduced small-angle scattering data with multi-dimensional data =

	== data structures ==		== data structures ==

Jemian: sign/date your work

2008-05-08T17:09:26Z

sign/date your work

← Older revision		Revision as of 17:09, 8 May 2008
Line 1:		Line 1:
			''Note: please sign and date your contributions. Reverse chronological order works for now. Keep this line on the top of the page.''

	== data structures ==		== data structures ==

Jemian at 17:07, 8 May 2008

2008-05-08T17:07:47Z

← Older revision		Revision as of 17:07, 8 May 2008
Line 1:		Line 1:
	== data structures ==		== data structures ==

			* '''2008-05-08, Pete Jemian'''

	The main difference between the cansas1d/1.0 and the multi-dimensional		The main difference between the cansas1d/1.0 and the multi-dimensional

Jemian: start the discussion page

2008-05-08T17:07:01Z

start the discussion page

New page

== data structures ==

The main difference between the cansas1d/1.0 and the multi-dimensional
formats is the need to handle multi-dimensional data. The ''table'' structure
used in the 1D format is quite 'tag-heavy' and just will not extend easily.
While the table format makes it easy to use XSLT v1 to display the data
in a browser, it takes a little bit more work to extract the I(Q) data into vectors
for use in processing or analysis software.

We need to define some new data structures. Rather than create anew, I chose to model the IgorPro data structures. Here's a quick sampler:
<pre>

<text name="shape">round</text>
<scalar name="wavelength" unit="pm">100.00</scalar>
<vector name="Q" unit="1/A" count="5">0.04 0.05 0.06 0.07 0.08</vector>
<array name="myArr" unit="none" rows="2" columns="3">
11 12 13
21 22 23
</array>
<matrix name="myMatrix" unit="none" dimension="3" count="2 4 3">
111 112 113
121 122 123
131 132 133
141 142 143

211 212 213
221 222 223
231 232 233
241 242 243
</matrix>
</pre>

There's more possible here but this just gives an idea.

Also, the ''Title'', ''Run'', ''Date'', etc. can be rolled up into attributes such as
<pre>
<SASentry date="2008-05-07" name="name-for-entry">
<SASdata
name="cansas2-example"
date="2008-05-04"
time="18:00:00"
instrument="Dell-D830"
run="1"
source="simulated"
>

</pre>

Here the '''Dell-D830''' attribute will associate the '''SASdata''' with the similarly named <pre><SASinstrument name="Dell-D830" /></pre> element.
Similar for the '''source''' attribute: it identifies <pre><SASsource name="simulated" /></pre>
'''Run''' is used (in practice) as an attribute of a specific SASdata. (''Note that order of attributes is not important to the syntax of a well-formed XML file.'')

== brief example==

* '''2008-05-06, Pete Jemian'''

Here is an attenuated example (lots of data points removed) of some
[http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/bimodal-test1.xml 1-D data]
in the multi-dimensional XML format:
<pre>
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="cansas2.xsl"?>

<SASroot version="2.0a"
xmlns="http://www.smallangles.net/cansas/2.0a"
xsi:schemaLocation="http://www.smallangles.net/cansas/2.0a cansas2.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:cansas="http://www.smallangles.net/cansas/2.0a"
>

<SASentry name="SAS bimodal test1" date="1992-01-31">
<SASdata name="simulated small-angle scattering data">
<vector name="Q" unit="1/A" count="4">0.004015714 0.004540865 0.005009597  0.3850296</vector>
<vector name="I" unit="1/cm" count="4">3497.473 3340.003 3322.474  0.110684</vector>
<vector name="Idev" unit="1/cm" count="4">90.72816 84.95314 79.63133  0.010393647</vector>
</SASdata>
<SASsample name="bimodal-test1">
<comment> # comments from FORTRAN source code </comment>
<comment> # ------------------------------------------------------------------------ </comment>
<comment> # Calculated (bimodal) test distribution. </comment>
<comment> # created 31 January 1992 by Pete R. Jemian </comment>
<comment> # ------------------------------------------------------------------------ </comment>
<comment> # Model consists of a bimodal, log-normal volume fraction </comment>
<comment> # size distribution. Parameters are as follows: </comment>
<comment> # PARAMETER ( contrast = 100.0 ) ! * 10^20, 1/cm^4 </comment>
<comment> # PARAMETER ( Background = 0.1 ) ! 1/cm </comment>
<comment> # PARAMETER ( sMult = 1000. ) ! counts per 1/cm (for shot noise) </comment>
<comment> # PARAMETER ( sNoise = 0.025 ) ! minimum level </comment>
<comment> # ------------------------------------------------------------------------ </comment>
<comment> # ! Vf rBar(A) sDev(A) </comment>
<comment> # PARAMETER ( a1 = 0.012, c1 = 75., s1 = 15. ) </comment>
<comment> # PARAMETER ( a2 = 0.008, c2 = 180., s2 = 60. ) </comment>
<comment> # ! Vf : volume fraction </comment>
<comment> # ! rBar : peak center (A) </comment>
<comment> # ! sDev : peak half-width (A) </comment>
<comment> # ------------------------------------------------------------------------ </comment>
</SASsample>
<SASinstrument name="simulated SAS calculation">
<SASsource name="simulated monochromatic source">
<radiation>artificial</radiation>
<scalar name="wavelength" unit="A">1.00</scalar>
</SASsource>
<SAScollimation />
<SASdetector name="calculation" />
</SASinstrument>
<SASprocess name="create the SAS data" date="1992-01-31">
<text name="shape">spheres</text>
<scalar name="Vf1" unit="dimensionless">0.012</scalar>
<scalar name="rBar1" unit="A">75</scalar>
<scalar name="sDev1" unit="A">15</scalar>
<scalar name="Vf2" unit="dimensionless">0.008</scalar>
<scalar name="rBar2" unit="A">180</scalar>
<scalar name="sDev2" unit="A">60</scalar>
<scalar name="contrast" unit="cm^4">100E20</scalar>
<scalar name="Background" unit="1/cm">0.1</scalar>
<scalar name="sMult" unit="cts/cm">1000.0</scalar>
<scalar name="sNoise" unit="fraction">0.25</scalar>
<comment>construct distribution from two log-normal volume-fraction size distributions</comment>
<comment>Vf : volume fraction</comment>
<comment>rBar : peak radius</comment>
<comment>sDev : peak half-width</comment>
<dataset name="initial size distribution">
<vector name="D" unit="A" count="4">25 61.5 73.345  880.6937</vector>
<vector name="f" unit="1/A" count="4">9.07952E-11 7.73813E-09 2.65831E-07  7.25662E-07</vector>
</dataset>
</SASprocess>
</SASentry>
</SASroot>
</pre>

== comments below culled were from email discussion ==

* '''2008-05-06, Ron Ghosh'''

I think this is where I came in, or when the ILL
appreciated that other sites were competitive with ILL-SANS.

NeXus should have started with these voluminous problems
of TREATED data, (RG at IPNS, Argonne 1995 or was it '96....)
and the commensurate simplification of minimal instrument
requirements. We have benefited a little from their
instrument dictionary. The time is now ripe to show the
use of HDF for storing multi-dimensional data, only using pure
HDF5 and able to ignore the needs for compatibility with
the multiferous raw data variants wrapped around with NeXus.

The arguments for tools and visualisers has been well ventilated,
and there really are advantages for the compression
and fast indexing offered by hdf5, and long-term support by
NCSA.

* '''2008-05-06, Stephen King'''

[Quick reply for now]

Agree we need to make a start on more complicated data. On this I would like to draw attention to Pete's term 'multi-dimensional' data rather than Andrew's '2D' data. We must address data beyond simple 2D detector patterns; so kinetic/time-resolved stuff (frames, periods, whatever you call them), and other experiments where data is being collected as a function of some variable (temperature, pH, or whatever) - what NeXus calls 'scanned' variables.

And before anyone asks, for 'multi-dimensional' data I am quite happy to abandon human-readability! I'm even prepared to consider binary storage!

* '''2008-05-04, Pete Jemian'''

Thanks for kicking this off. I have a rough outline of a multi-dimensional canSAS format.
Multi-dimensional data must provide for any reasonable number of dimensions. Definitely,
it is a vector format if XML is used.

The multi-dimensional format should reduce some inconsistencies discovered while working
with the 1D format. Little things such as ''SASsample'' has
an element called ''ID'' but other elements such as ''SASentry'' use an element called ''Title'' or even an attribute called ''name.'. For consistency, that should be changed to an attribute called ''name.''
But it is these little differences that make coding an interface program more difficult.

But, we should also consider how different we are from NeXus and then why.
NeXus has two underlying formats, HDF (v4 and v5) and also XML. An XSLT
translation for cansas1d/1.0 to NeXus XML can be built to translate if desired.
Also Freddie Akeroyd has added some capabilities to NeXus XML to accept our table format.
The rough outline of a multi-dimensional canSAS format I've developed is
so close to NeXus that we need to consider strongly
why to develop on our own and not use what many other facilities have adopted.

Our cansas1d/1.0 standard is defined by an XML Schema (XSD) and this is a major strength. NeXus uses meta-DTDs. How
easy is it to try define NeXus using XSDs? Doesn't look that easy.
I've found a great reason this last week to have a standard that is defined by an
XSD. That is, autogeneration of a data structure in a particular programming language
by software that reads the XSD directly. For Java, it is JAXB. By a Google search,
there are such things for Python as well. I've been testing the Java one and it makes Java
coding very easy. Non-compliant files will be identified (with rather cryptic diagnostics -
maybe that could be improved).

By the way, the JAXB technology makes it clear why a vector format would make
input/output much simpler. In our table format, each Q[i], I[i], etc. is a separate
leaf in the data structure tree. The "leaves" of the SASdata "branch" need to be collated
into structures of double[] Q, double[] I before they can actually be used in reasonable
work. Similar problem with the IgorPro reader. It takes 5 minutes to parse the many
thousand elements in the 1998spheres.xml example. A vector format of that data loads
in IgorPro within quite reasonable time (forgot the numbers) using Andrew Nelson's
XMLutils XOP.

* '''2008-05-04, Andrew Jackson'''

The SNS software group is working with
the DANSE project on getting their reduced data into the DANSE
software. Paul Butler has suggested that making use of the canSAS
format would be a good idea. However 2D data needs to be represented.

Thus I suggest two things. Firstly with the 1D format agreed, we
should look at 2D data. AFAICS we can do this by introducing the
vector format proposed by Andrew N and Pete with little other pain -
am I right Pete? Secondly, how do you all feel about adding Michael
Reuter to this group to get his input on the 2D since he will be
implementing it in short order?

To head off the argument, I realize that this 2D implementation won't
be easily readable without using an XML reader, nor will it be "human
readable". However, we only aimed to make a 1D format that was those
two things and I think they are unreasonable for a 2D format.

Talk:cansas multid - Revision history

Jemian: /* data structures */

Jemian at 17:32, 8 May 2008

Jemian: sign/date your work

Jemian at 17:07, 8 May 2008

Jemian: start the discussion page