[MBDyn-users] reference frames
Patrick.Rix at repower.de
Wed Jul 7 16:08:04 CEST 2010
>> I feel, it would be more advantegeous to have multiple but smaller
>> files and of binary (NetCDF) format (using float values the data size
>> is over 3 times smaller compared to the text format).
>I'd stick with double, for two reasons:
> - we save casting effort, and directly pass pointers to actual
>storage arrays to NetCDF
> - the NetCDF format could eventually become a restart database at any
>(saved) simulation time with no loss of accuracy (long term project far
>in the future)
I did not mean to change MBDyn's NetCDF output. Using DOUBLE and writing to multi-variable *.nc output files would be perfectly o.k. for run time (especially if you say that casting to FLOAT would significantly slow down the simulation). Also Ascii output is fine - though of course a binary format would be preferable as we would use less disk space (by a factor of somewhat ~1.5 when using DOUBLE) for the result files.
What I meant was doing the splitting of a multi-var files into several single-var files & double-to-float conversion as post-processing.
Then everybody can safely decide what should be his/her preferred data type and structure for storing the results.
I personally feel that having many but small files is more flexible and faster during subsequent post-proc steps (e.g. viewing data from large multi-var files takes definitely longer - compared to small single-var files - as for each data set / time series always the whole file has to be read until the data are complete).
I agree that for restarting a model surely a very precise data type is required - but for all-day data handling for most of the users FLOATs might be precise enough (with a factor ~3 of less disk space compared to text). For our results its even sufficient to go down to (2-byte) SHORT INT compression with a saving of factor ~5 w.r.t. text format - then we get 65000 discrete niveaus to represent the data range of a time series what means due to the loss of precison we have to accept an uncertainty/error of 1/65000 = 1.5e-5 = 0.0015% . Compared to the uncertainties of many of the model parameters in the order of a few % and considering the gain of using 5x less disk space this seems to be a good trade off for our needs.
The only thing that is required is a bunch of tools providing these data manipulation features - and fortunately for NetCDF format there is quite a big palette of tools availble.
>> experimenting with that. Having many but small units is also a good
>> environment to easily bring the post-proc on a network cluster like
>> CONDOR where you can have many machines working in parallel on the
>> same task with each node machine working on small units.
>If you have multiple cores, a single DB should work fine as well, as
>NetCDF is multiple-read, single write thread safe.
How many cores do nowadays machine have ? 4 and if one can effort maybe 8.
In a CONDOR cluster you can easily have up to several hundeds of machines - all the PCs of a company could join the cluster and you would use the power of those which are free (e.g. where the user is in meeting). So at night when people have left the office the full power is available. And everything with buying extra machines or extra software as CONDOR is free. We made good experiences with that.
Now as the data files are transferred to the node machines smaller file sizes are better suited for network transfer and take less time to be processed on a node machine.
Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte umgehend den Absender und l?schen Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist nicht gestattet. Wie Sie wissen, kann die Sicherheit von ?bermittlungen per E-Mail nicht gew?hrleistet werden, E-Mails k?nnen missbr?uchlich unter fremdem Namen erstellt oder ver?ndert werden. Aus diesem Grund bitten wir um Verst?ndnis daf?r, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erkl?rungen ausschlie?en m?ssen. Diese Regelung gilt nur dann nicht, wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung ?ber die Einhaltung von Sicherheits- und Verschl?sselungsstandards getroffen haben.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. As you know, the security of e-mail transmissions can not be guaranteed. E-mails can be misused to be written or modified under false names. For that reason, we ask you to understand the necessity for us to rule out the legal obligation of the above statement, for your protection and ours. This regulation is only invalid if we have concluded a special written agreement with you about the compliance with security and encryption standards.
REpower Systems AG Sitz: Hamburg Vorstand: Andreas Nauen (Vorsitz), Per Hornung Pedersen, Lars Rytter Kristensen, Derrick Noe, Matthias Schubert
Aufsichtsratsvorsitzender: Tulsi Tanti Registergericht: AG Hamburg (Mitte) HRB Nr.: 75543
More information about the MBDyn-users