Generating Bahn Layouts

This area is dedicated to our foreign guests of this forum. Contributions can be written in any language.
[Diese Kategorie ist den ausländischen Gästen dieses Forums gewidmet. Beiträge können in beliebiger Sprache verfasst werden.]
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

Hello everyone

Due to some progress at Openstreetmap (they create another way off accessing Openstreetmap data (in a compressed way) and some bug resolving I've created last 2 weeks version 0.19.
This version is available as from now.

So take a look at:

http://www.xs4all.nl/~wanl0049/ops2jbss/

again.


It solves also the issue where Chris is struggling with and it is much faster.

One small issue still appears (the 'cache' are is not cleaned after everything is well processed).

Please use the CCA possibility in the menu, to clean this.

Regards, Bert Tijhuis
Chris
Beiträge: 175
Registriert: Donnerstag 24. Mai 2007, 03:26
Wohnort: Cardiff

Re: Generating Bahn Layouts

Beitrag von Chris »

I am getting an error: "XMLfile GB.OSM is invalid and will be deleted (Please restart the extractor again)" gb.osm in this case was the file from http://download.geofabrik.de/osm/europe/ which I am assured is valid. It turns out that the problem was converting the supplied filename to upper case. There's no need to do this: on a case-sensitive filesystem you will not find the file, and on a case-insensitive filesystem it doesn't matter what case you supply.
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

Chris hat geschrieben:I am getting an error: "XMLfile GB.OSM is invalid and will be deleted (Please restart the extractor again)" gb.osm in this case was the file from http://download.geofabrik.de/osm/europe/ which I am assured is valid. It turns out that the problem was converting the supplied filename to upper case. There's no need to do this: on a case-sensitive filesystem you will not find the file, and on a case-insensitive filesystem it doesn't matter what case you supply.
I wasn't aware of this nice GEOFABRIK.
In the meantime I will download a file a try to find why it is failing
Regards,
Chris
Beiträge: 175
Registriert: Donnerstag 24. Mai 2007, 03:26
Wohnort: Cardiff

Re: Generating Bahn Layouts

Beitrag von Chris »

My Perl is about as good as my Swahili, but there are two problems here. First the casing:

Code: Alles auswählen

# ops2jbss.pl:647
                $veld2 =~ tr/a-z/A-Z/;
I don't know why this line is here, since the few places I've see that handle user input are already rigged for both cases, e.g.

Code: Alles auswählen

                if ($choise =~ /^[gG]$/) { # both cases specified
                if ($choise =~ /^cca/i) { # case-insensitive match
The error message comes from here:

Code: Alles auswählen

# ops2jbss_e.pl:170
  print "Process file: $file\n" if ($debug);
  $dbh->do("begin transaction immediate");
  my $ref = eval { XMLin($file); };
  if ($@) {
  	 print "XMLfile $file is invalid and will be deleted (Please restart the extractor again)\n" if ($debug);
  	 unlink($file);
  	 return 0;
	}
That looks like it's producing that same message regardless of what the error actually was, which IMO makes it poor to assume it'll be an invalid file, and rather nasty to insist on unlinking the file afterwards.

Just my .02
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

Hello Chris,

Thanks for digging through the code:

The real problem is:\
Choise ?: > o great_britain.osm
Process file: GREAT_BRITAIN.OSM
Out of memory!
Out of memory!
Out of memory!

Because these OSM file are realy large (I've tested it with a file of 3 GBytes)

Due to the fact that XMLin is part of the XML::Simple module (not maintained by myself) I can't solve this issue. Try to use smaller files


But please in you're case download the newest sources,
These won't delete the file afterwards, and specified the root course of the failure.

I've noticed that you are still working with an older release please use release 0.20 or upwards

Regards,
Bert Tijhuis
Chris
Beiträge: 175
Registriert: Donnerstag 24. Mai 2007, 03:26
Wohnort: Cardiff

Re: Generating Bahn Layouts

Beitrag von Chris »

BertTijhuis hat geschrieben:Because these OSM file are realy large (I've tested it with a file of 3 GBytes)
That was not the cause of the issues I raised. The issue I raised was that you are incorrectly converting the case of the filename, and then throwing back an error message which is inaccurate. This is independent of the file being impossible to process this way on a 32-bit machine.

Code: Alles auswählen

chris@testbox:~/ops2jbss$ ls -l test.osm
-rw-r--r-- 1 chris chris 0 2010-05-28 19:22 test.osm
[...]
Choise ?:  > o test.osm
Process file: TEST.OSM
XMLfile TEST.OSM is invalid (Please restart the extractor when nescessarely)
Reason:
File does not exist: TEST.OSM at ops2jbss_e.pl line 187
I've noticed that you are still working with an older release please use release 0.20 or upwards
I was using 0.19 yesterday, and you released 0.20 today. I notice that 0.20 still has the upper-case tr - why is it there?

Also, what release should I have been using yesterday? ;)
Benutzeravatar
micha88
Beiträge: 1987
Registriert: Freitag 18. Februar 2005, 12:50
Wohnort: Marbach am Neckar
Kontaktdaten:

Re: Generating Bahn Layouts

Beitrag von micha88 »

You can reduce file size with the tool osmosis: you can create a much smaller xml-file containing only railway lines.
Bild
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

Chris hat geschrieben:
BertTijhuis hat geschrieben:Because these OSM file are realy large (I've tested it with a file of 3 GBytes)
That was not the cause of the issues I raised. The issue I raised was that you are incorrectly converting the case of the filename, and then throwing back an error message which is inaccurate. This is independent of the file being impossible to process this way on a 32-bit machine.

Code: Alles auswählen

chris@testbox:~/ops2jbss$ ls -l test.osm
-rw-r--r-- 1 chris chris 0 2010-05-28 19:22 test.osm
[...]
Choise ?:  > o test.osm
Process file: TEST.OSM
XMLfile TEST.OSM is invalid (Please restart the extractor when nescessarely)
Reason:
File does not exist: TEST.OSM at ops2jbss_e.pl line 187
I've noticed that you are still working with an older release please use release 0.20 or upwards
I was using 0.19 yesterday, and you released 0.20 today. I notice that 0.20 still has the upper-case tr - why is it there?

Also, what release should I have been using yesterday? ;)

-----
I've now 2 answers to you're questions

1) I will fix the bug that only occurs in a none Windows environment. (within 2 months)

2) I will start to work on a version that could use precompiled OSM files from the internet (e.g. http://download.geofabrik.de/osm/europe/ )
and use the splitter that is also used for the MKgmap utility. (within 2 months).



Of course in the mean time you can use OSMOSIS and rename you're filenames to lower case.


Regards,
Bert Tijhuis
Benutzeravatar
micha88
Beiträge: 1987
Registriert: Freitag 18. Februar 2005, 12:50
Wohnort: Marbach am Neckar
Kontaktdaten:

Re: Generating Bahn Layouts

Beitrag von micha88 »

I've cut of the German postings and moved them to "Probleme und Fehlermeldungen".
Bild
Chris
Beiträge: 175
Registriert: Donnerstag 24. Mai 2007, 03:26
Wohnort: Cardiff

Re: Generating Bahn Layouts

Beitrag von Chris »

Something seems to be out of order somewhere. Did my first live extraction with 0.20 and got the following:
DBD::SQLite::db do failed: SQL logic error or missing database
cannot start a transaction within a transaction at ops2jbss_e.pl line 199.
It comes at the point where "$cache-file will be processed". Centred at 51.74N 4.065W, 50x50km, scale 64.
Chris
Beiträge: 175
Registriert: Donnerstag 24. Mai 2007, 03:26
Wohnort: Cardiff

Re: Generating Bahn Layouts

Beitrag von Chris »

More bugs:

1. Start with no database. Attempt to import a file. You get a zero-length file where the database should be, and the SQL process bails with "table 'way' does not exist" (etc.). The database has not been initialised correctly. CDB does not rectify this (since all it does is unlink the file).
Workaround: start an online extraction, and abort shortly after it starts - the database is then initialised and the import can be performed.

2. Program deletes files it has attempted to import, even when the import has failed due to #1.
ramops
Beiträge: 4
Registriert: Sonntag 20. November 2011, 15:37

Re: Generating Bahn Layouts

Beitrag von ramops »

Anything yet on this?
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

Hello,

Just after one year, there is some news about this.

Due to a remark of Hamzaa I've found out that the current procedure isn't working anymore correctly.

So I've updated the version (021) and this is available as of 20 november 2011.

@Chris: With this current version it is possible to process large files from www.geofabrik.de but of course you've to unzip these files first (7zip)
The program doesn't consume that much memory anymore and you don't run out of memory.

Due to the fact that we don't run out of memory anymore I've introduced a disadvantage that its running slower than the previous versions.

Please try this new version, and give me some feedback
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

It seems that there is something wrong in Generating Bahn layouts.

Partly this has to do with the richness of the Openstreetmap data. So you cant find the needle in the haystack anymore.
And sometimes some large rectancular lines are generated.

So I still working on this issue and hope to solve this asap

Regards :o
BertTijhuis
Beiträge: 27
Registriert: Donnerstag 8. Mai 2008, 23:56
Wohnort: Nederland

Re: Generating Bahn Layouts

Beitrag von BertTijhuis »

Hello,

Just before going to Kuala Lumpur I've finalized this version 0.22. (This is a bugfix of 0.21)
Please DONt use version 0.21

Tested with several area's (from Geofabrik.de) with Luxembourg (Pretty small country and I like this), Ile de France (on request) and Kuala Lumpur and ofcourse the Netherlands

Everything seems to be working correctly

:oops: :oops: :mrgreen:
Antworten