[MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)

Rix, Patrick Patrick.Rix at repower.de
Thu May 5 16:01:06 CEST 2011


Hi Jens,

Indeed a very interesting and astonishing fact !
..seems to make it worth trying to compile MBDyn with Visual studio - maybe we get the same speed up there.
But I guess this would be a bigger work package than the small utilities :-(

Thanks for you investigations !

Patrick.



-----Ursprüngliche Nachricht-----
Von: Jens.vanSchelve at Emerson.com [mailto:Jens.vanSchelve at Emerson.com]
Gesendet: Donnerstag, 5. Mai 2011 15:54
An: mbdyn-users at mbdyn.org
Cc: Rix, Patrick
Betreff: AW: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)


Hi Patrick,

just to let you know -> I compiled my code under Visual studio and this speed it up by factor > 2!
My test case takes now 30s instead of 64s with the gcc. -still much slower than linux, but whatever.
Also the program is now just 100kb instead of 900kb -however this is not a big advantage to me :)

Jens


Jens van Schelve | Research Engineer for System Engineering | SSB Wind Systems GmbH & Co.KG
Emerson Industrial Automation | Neuenkirchener Straße 13 | 48499 Salzbergen | Germany
T +49 5976 946 2867 | F +49 5976 946 222 286
Jens.vanSchelve at Emerson.com | www.ssbwindsystems.de

Handelsregister | Commercial Register: Amtsgericht Osnabrueck HRA 100080
Geschaeftsfuehrer | Managing Directors: Dr. Gisbert Schulze, Dipl.-Betriebswirt (FH) Dirk Hamenstaedt
phG | General Partner: SSB-Antriebstechnik Verwaltungs- und Beteiligungsgesellschaft mbH | Neuenkirchener Str. 13 | 48499 Salzbergen
Handelsregister | Commercial Register: Amtsgericht Osnabrueck HRB 100015

-----Ursprüngliche Nachricht-----
Von: mbdyn-users-bounces at mbdyn.org [mailto:mbdyn-users-bounces at mbdyn.org] Im Auftrag von Rix, Patrick
Gesendet: Mittwoch, 4. Mai 2011 14:28
An: mbdyn-users at mbdyn.org
Betreff: Re: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)

Jens,

thank you very much for your detailed report - this are very interesting results.
I always suspected that programs under Windows+Cygwin run by some factor slower than on Linux but I'm impressed of the big difference in file IO performance between Linux and Windows+Cygwin.
Would be interesting doing this Windows/Linux benchmark on identical machines.
I'll try to compile  abs2rel  with Visual C++  ..I haven't tried this up to now.

..and your observation is correct: both programs (abs2rel & mbdyn_bulk_xtract) work in a buffered fashion, reading a block of some thousand text lines into buffer, splitting+converting+transforming+writing the data of complete time steps until all lines in the buffer are processed and then reading the next block until the end of file. I have chosen this strategy in order to conquer super large text files up to 3GB (as e.g. could be the case for MBDyn's .aer output files).

Patrick.


-----Ursprüngliche Nachricht-----
Von: Jens.vanSchelve at Emerson.com [mailto:Jens.vanSchelve at Emerson.com]
Gesendet: Mittwoch, 4. Mai 2011 13:49
An: Rix, Patrick; mbdyn-users at mbdyn.org
Cc: Jens.Spielmann at Emerson.com
Betreff: AW: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)

Hi Patrick,

is it possible that the performace issue is related to either windows or the gnu compiler under windows?
I did a performance check of all the tools on a mbdyn wind turbine simulation with a *.mov file output of about 75mb.

I could not run your compiled cygwin version: "Der Prozedureinsprungspunkt "_deinitilise" wurde in der DLL "cygwin1.dll" nicht gefunden", so I compiled it myself under MinGW - which is nice because its smaller and does NOT NEED the cygwin dll to execute - I attached it.

Windows (Notebook):
abs2rel.awk             -> 48 seconds
abs2rel-0.0.2   -> 47 seconds (compiled under MinGW)
abs2rel-ssb             -> 64 seconds (36s reading input, 1s calculation, 27s writing output)

My tool (abs2rel-ssb) is here suprisingly slow -however it also reads some other files like *.log, *.out, *.jnt ectr. and splits the output like your bulk_utilis into seperate files for each node. And reading and writing seems to be the trouble maker here.
Under Linux however the story looks completely different:

Linux (old Desktop):
abs2rel.awk             -> 54 seconds (a bit slower than windows because of slower cpu)
Abs2rel-0.0.2   -> 28 seconds (2 times faster than awk)
Abs2rel-ssb             -> 13 seconds (4 times fater than awk. Also just <1s calc and same features like the win version (other files and splitting..))

To me this result is very surprising and I don't have an explanation for it, except that file reading under Linux in c (gcc) seems MUCH faster than under Windows (or str2d?).
The main difference between abs2rel-0.0.2 (Patrick) and abs2rel-ssb (me) seems to be that (if I understand correctly) I first read the complete *.mov file into memory than do the calculations and then write the output, you (Patrick) seem to it simultaneously.

I also tried to compile abs2rel-0.0.2 under Visual C++ without success. Maybe it is gcc's fault? -> NetCDF would help the windows version probably alot.

Jens

Jens van Schelve | Research Engineer for System Engineering | SSB Wind Systems GmbH & Co.KG Emerson Industrial Automation | Neuenkirchener Straße 13 | 48499 Salzbergen | Germany T +49 5976 946 2867 | F +49 5976 946 222 286 Jens.vanSchelve at Emerson.com | www.ssbwindsystems.de

Handelsregister | Commercial Register: Amtsgericht Osnabrueck HRA 100080 Geschaeftsfuehrer | Managing Directors: Dr. Gisbert Schulze, Dipl.-Betriebswirt (FH) Dirk Hamenstaedt phG | General Partner: SSB-Antriebstechnik Verwaltungs- und Beteiligungsgesellschaft mbH | Neuenkirchener Str. 13 | 48499 Salzbergen Handelsregister | Commercial Register: Amtsgericht Osnabrueck HRB 100015

-----Ursprüngliche Nachricht-----
Von: mbdyn-users-bounces at mbdyn.org [mailto:mbdyn-users-bounces at mbdyn.org] Im Auftrag von Rix, Patrick
Gesendet: Mittwoch, 4. Mai 2011 09:30
An: mbdyn-users at mbdyn.org
Betreff: Re: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)

Hi Jens,

thank you for your annotation - it gives me the hope that there is still some potential for optimization.
My impression is that reading and writing of pure string streams are not a problem if the strings exist already complete in the memory. Also splitting of a text line into tokens seems not to be too time consuming. I'm suspecting the string-to-double conversion to be CPU intensive.
I discovered something similar with the Blender interface. It does (nearly) not make a difference if I import twice the same positions & orientation data of a .mov file, one time all columns (1..18) and another time only columns 1..6.
If I correctly understand the Python scripts the interface always reads a complete line and splits it into tokens but only converts the first 6 data columns after the label to double. As for both tests the CPU-time was nearly equal
(1 or 2 sec of total ~30sec) though the file sizes differ by a factor of 3 my conclusion was that reading and splitting of strings seems to be much less time consuming than string-to-double conversion (at least for Python but I assume for C things might be similar).

Concerning NetCDF implementation, you should definitely do that - the interface is very easy to understand and use.
You could use the codes of the programs  "ncdump" / "ncgen"   as a starting point. These are included in the netcdf package.
I guess apart from speed and saving disk space the biggest benefit in switching to NetCDF is that a lot of third party tools for data manipulaton & plotting are already available, so the MBDyn user community could choose from a wide variety of tools without re-inventing the wheel.

Schöne Grüße,
Patrick.


-----Ursprüngliche Nachricht-----
Von: Jens.vanSchelve at Emerson.com [mailto:Jens.vanSchelve at Emerson.com]
Gesendet: Dienstag, 3. Mai 2011 20:02
An: Rix, Patrick; mbdyn-users at mbdyn.org
Betreff: AW: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)


Hi Patrick,

Regarding the speed I can not confirm your point. I did not test your program yet, but I have written my own version of abs2rel in c some time ago which is about 4 times faster than the awk version.. and I have not implemented fancy things like only Nth time step or position and orientation only, like you did.

Unfortunatly I am not allowed to publish the tool here, but I will have a look into your code and see whats going on.

However, I can confirm that also for me about 85% of the time is spend in reading and writing the ascii files.
Binary netcdf files would speed up the stuff significantly, I guess.
-> and ist also on my todo list :)

Jens


Jens van Schelve | Research Engineer for System Engineering | SSB Wind Systems GmbH & Co.KG Emerson Industrial Automation | Neuenkirchener Straße 13 | 48499 Salzbergen | Germany T +49 5976 946 2867 | F +49 5976 946 222 286 Jens.vanSchelve at Emerson.com | www.ssbwindsystems.de

Handelsregister | Commercial Register: Amtsgericht Osnabrueck HRA 100080 Geschaeftsfuehrer | Managing Directors: Dr. Gisbert Schulze, Dipl.-Betriebswirt (FH) Dirk Hamenstaedt phG | General Partner: SSB-Antriebstechnik Verwaltungs- und Beteiligungsgesellschaft mbH | Neuenkirchener Str. 13 | 48499 Salzbergen Handelsregister | Commercial Register: Amtsgericht Osnabrueck HRB 100015

-----Ursprüngliche Nachricht-----
Von: mbdyn-users-bounces at mbdyn.org [mailto:mbdyn-users-bounces at mbdyn.org] Im Auftrag von Rix, Patrick
Gesendet: Dienstag, 3. Mai 2011 19:18
An: mbdyn-users at mbdyn.org
Betreff: Re: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)

Hello Pierangelo,

Of course the awk script is definitely useful: it's easy to use, it's small and it comes without any compilation hassle - and last but not least it's not slow (as I thought before). I was disappointed when I discovered that the C-version is of equal speed for an identical task as the awk script because I started the work in the hope that in C the tool would be significantly faster - but it wasn't. To get at least something useful out of it I tried to make it a little bit more comfortable by adding options and gain some speed by skipping labels and time steps - but this could be easily implemented too into the script.
I found out that most of the CPU time seems to be spend on the Ascii-to-double conversion.
For this reason and for using less disk space [..supposed the user can choose FLOAT as NetCDF output type in MBDyn ;-) ] I absolutely agree with you that the next step to a more efficient and faster data post-processing would be to directly work on binary NetCDF files.
I'm planing to re-implement/enhance abs2rel with the capability to work on NetCDF files.
I guess binary data access would also boost the import to Blender - I wanted to look on that too/next(?).

Do you have plans to complete NetCDF output for the rest of MBDyn's result files that currently only exist in text format ?

What do you think, which task is most important ?
I'd suggest adding a Blender NetCDF interface first, then adding a FLOAT switch to MBDyn's NetCDF format, afterwards abs2rel and finally completing MBDyn's NetCDF output.
Which would be your preferred order ?

Ciao,
Patrick.

________________________________________
From: mbdyn-users-bounces at mbdyn.org [mbdyn-users-bounces at mbdyn.org] On Behalf Of masarati at aero.polimi.it [masarati at aero.polimi.it]
Sent: Friday, April 29, 2011 9:10 AM
To: patrick.rix at online.de
Cc: MBDyn user-mailing-list
Subject: Re: [MBDyn-users] Abs2Rel-0.0.2(BugFix) & mbdyn_bulk_xtract-0.0.1 (NEW)

>
> Hello everybody,
>
> I wanted to inform about a BUGFIX in the C-version of abs2rel (Sorry!)
> --> the "--Steps=N1,N2" option did not output the correct time step
> indices; should be fixed now.
> the corrected version Abs2Rel-0.0.2  can be downloaded at:
>
> <http://sourceforge.net/projects/mbdynbulkutils/files/mbdyn_bulk_utils
> /1.1.5/>
>
> I've added yet another C-utility,  "mbdyn_bulk_xtract"-0.0.1 , which
> performs data extraction of MBDyn bulk text files ( .out time file and
> data files as e.g. .mov, .frc, .jnt, etc.) into a separate text file
> (and faster than the old shell scripts). One can extract time series
> by selecting one or more items (nodes/elems) by label, select the data
> columns or the time step range to be kept by giving the start/stop
> index inclusive an optional step output frequency to output every N-th
> time step.
> For the .out time file the progam accounts for / operates on the 9th
> column containing the OutFlag=0|1 indicating if a time step was
> written or not when using an output meter.
> One purpose of "mbdyn_bulk_xtract" was the idea to use it as a filter
> for the .mov file to reduce the motion data before importing them e.g.
> into Blender for animation.

Patrick,

thanks for the contribution.  I found the abs2rel feature very useful in many cases, but never needed to go beyond a quick'n'dirty script because I seldom suffered from performance issues with our typical analysis (usually the bottleneck is somewhere else :)).

In general, the need to perform some (mathematically consistent) bulk operations on data surfaces relatively often (we have other custom scripts/tools that we do not distribute because they are specifically designed for some purpose of non-general usefulness).

I think tools like abs2rel would strongly benefit from the capability to directly access the NetCDF database; the spirit would be:

- write a complete but minimal dataset during analysis, to minimize the overhead

- perform data manipulation off-line as a post-processing

Cheers, p.


_______________________________________________
MBDyn-users mailing list
MBDyn-users at mbdyn.org
https://mail.mbdyn.org/cgi-bin/mailman/listinfo/mbdyn-users

Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte umgehend den Absender und l?schen Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist nicht gestattet. Wie Sie wissen, kann die Sicherheit von ?bermittlungen per E-Mail nicht gew?hrleistet werden, E-Mails k?nnen missbr?uchlich unter fremdem Namen erstellt oder ver?ndert werden. Aus diesem Grund bitten wir um Verst?ndnis daf?r, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erkl?rungen ausschlie?en m?ssen. Diese Regelung gilt nur dann nicht, wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung ?ber die Einhaltung von Sicherheits- und Verschl?sselungsstandards getroffen haben.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. As you know, the security of e-mail transmissions can not be guaranteed. E-mails can be misused to be written or modified under false names. For that reason, we ask you to understand the necessity for us to rule out the legal obligation of the above statement, for your protection and ours. This regulation is only invalid if we have concluded a special written agreement with you about the compliance with security and encryption standards.
REpower Systems AG Sitz: Hamburg Vorstand: Andreas Nauen (Vorsitz), Gregor Gn?dig, Derrick Noe, Matthias Schubert
Aufsichtsratsvorsitzender: Tulsi Tanti Registergericht: AG Hamburg (Mitte) HRB Nr.: 75543

_______________________________________________
MBDyn-users mailing list
MBDyn-users at mbdyn.org
https://mail.mbdyn.org/cgi-bin/mailman/listinfo/mbdyn-users

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte umgehend den Absender und löschen Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist nicht gestattet. Wie Sie wissen, kann die Sicherheit von Übermittlungen per E-Mail nicht gewährleistet werden, E-Mails können missbräuchlich unter fremdem Namen erstellt oder verändert werden. Aus diesem Grund bitten wir um Verständnis dafür, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erklärungen ausschließen müssen. Diese Regelung gilt nur dann nicht, wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung über die Einhaltung von Sicherheits- und Verschlüsselungsstandards getroffen haben.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. As you know, the security of e-mail transmissions can not be guaranteed. E-mails can be misused to be written or modified under false names. For that reason, we ask you to understand the necessity for us to rule out the legal obligation of the above statement, for your protection and ours. This regulation is only invalid if we have concluded a special written agreement with you about the compliance with security and encryption standards.
REpower Systems AG Sitz: Hamburg Vorstand: Andreas Nauen (Vorsitz), Gregor Gnädig, Derrick Noe, Matthias Schubert
Aufsichtsratsvorsitzender: Tulsi Tanti Registergericht: AG Hamburg (Mitte) HRB Nr.: 75543

_______________________________________________
MBDyn-users mailing list
MBDyn-users at mbdyn.org
https://mail.mbdyn.org/cgi-bin/mailman/listinfo/mbdyn-users

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte umgehend den Absender und löschen Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist nicht gestattet. Wie Sie wissen, kann die Sicherheit von Übermittlungen per E-Mail nicht gewährleistet werden, E-Mails können missbräuchlich unter fremdem Namen erstellt oder verändert werden. Aus diesem Grund bitten wir um Verständnis dafür, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erklärungen ausschließen müssen. Diese Regelung gilt nur dann nicht, wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung über die Einhaltung von Sicherheits- und Verschlüsselungsstandards getroffen haben.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. As you know, the security of e-mail transmissions can not be guaranteed. E-mails can be misused to be written or modified under false names. For that reason, we ask you to understand the necessity for us to rule out the legal obligation of the above statement, for your protection and ours. This regulation is only invalid if we have concluded a special written agreement with you about the compliance with security and encryption standards.
REpower Systems AG Sitz: Hamburg Vorstand: Andreas Nauen (Vorsitz), Gregor Gnädig, Derrick Noe, Matthias Schubert
Aufsichtsratsvorsitzender: Tulsi Tanti Registergericht: AG Hamburg (Mitte) HRB Nr.: 75543

_______________________________________________
MBDyn-users mailing list
MBDyn-users at mbdyn.org
https://mail.mbdyn.org/cgi-bin/mailman/listinfo/mbdyn-users

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte umgehend den Absender und löschen Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist nicht gestattet. Wie Sie wissen, kann die Sicherheit von Übermittlungen per E-Mail nicht gewährleistet werden, E-Mails können missbräuchlich unter fremdem Namen erstellt oder verändert werden. Aus diesem Grund bitten wir um Verständnis dafür, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erklärungen ausschließen müssen. Diese Regelung gilt nur dann nicht, wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung über die Einhaltung von Sicherheits- und Verschlüsselungsstandards getroffen haben.
This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. As you know, the security of e-mail transmissions can not be guaranteed. E-mails can be misused to be written or modified under false names. For that reason, we ask you to understand the necessity for us to rule out the legal obligation of the above statement, for your protection and ours. This regulation is only invalid if we have concluded a special written agreement with you about the compliance with security and encryption standards.
REpower Systems AG Sitz: Hamburg Vorstand: Andreas Nauen (Vorsitz), Gregor Gnädig, Derrick Noe, Matthias Schubert
Aufsichtsratsvorsitzender: Tulsi Tanti Registergericht: AG Hamburg (Mitte) HRB Nr.: 75543



More information about the MBDyn-users mailing list