The exporting of resulting OSM files can potentially be sped up

[Vectorial1024]

This requires confirmation later, but I noticed on this StackOverflow discussion:

https://stackoverflow.com/questions/44560655/python-writelines-and-write-huge-time-difference

Why is there such a large difference in the file writing time for write() and writelines() even though it is the same data?

---

I have used ogr2osm for a while, and I notice that it can be quite slow on larger files. Like, unusually slow.

It seems the exporting can be sped up. Will investigate later.

[Vectorial1024]

Benchmarking the existing method

First, I must admit my current PC is at mid-high tier, and so things might be faster than average. But the point should still stand even for slower computers. Also, extra care must be taken because the files to be processed can be very large.

---

Some details:

• Command run: python -m ogr2osm -t test_translate -o target.osm source.geojson
• Size of data source: about 430 MB
• Measuring the duration: adding some basic measurement at DataWriterContextManager.output using time.time()
• I/O are all on SSD

I run the command for 5 times.

Measured time (average): 19.455 seconds

[Vectorial1024]

One thing that sticks out when doing some detailed profiling:

Beginning to time the export
Writing file header
Writing nodes
Writing took (to_xml, write): 7.521965265274048, 0.6387271881103516
Writing ways
Writing took (to_xml, write): 10.0064537525177, 0.5809998512268066
Writing relations
Writing took (to_xml, write): 0, 0
Writing file footer
Time elapsed was 18.999 seconds

It is actually the to_xml part which is slow, not the IO.

It seems we may continue with some sort of multi-threading.

[Vectorial1024]

Hmmm. We are already using lxml for fast export.

Spawning new threads does not work due to Python's GIL, which effectively encourages single-threaded code.

Playing around with multiprocessing did not bring much immediate results because we will need to do extra work to pass values into the subprocesses. This might be viable in the long term, but not something that can be done in a single day.

If we are able to somehow utilize multi-processing effectively, then perhaps there will be a significant speedup.

[Vectorial1024]

This just dropped a few days ago:

https://www.bitecode.dev/p/whats-up-python-the-gil-removed-a

THe removal of GIL in Python can be very useful to this speed up: instead of spawning difficult-to-control subprocesses to parallelize XML-to-string, we may finally have a easy-to-control multi-threaded XML-to-string process to speed up exporting.

[kabilsharma]

Is this still active scope of work where the tool can be sped up ?

[roelderickx]

For me it is not active scope, but I did not close this issue either. Any performance improvement is welcome.

[Vectorial1024]

To be fair, so much has happened to me in this past year or two, and I have forgotten I even made this GitHub issue.

The idea was/is, if the GIL news was dropped too recently, then we need to give time for the bloggers, influencers, and content farms to cook up the new articles and tutorials. When they've finished cooking, we go in and learn the new situation, and then relate it to the "multiprocessing to-xml" idea proposed by past-me.

Only after this waiting period will we clearly know whether we can improve performance in this direction. (I hope we can.)

[kabilsharma]

I request anyone can help me here to make it faster as it takes lot of ram sometimes 300 gb RAM for a 15 GB Geojson some guidance would definitely help.

[roelderickx]

Making it faster and making it use less memory are two opposite goals. Which one are you trying to achieve? Do you have a suggestion where we can either improve the performance or use less memory?

[kabilsharma]

sorry for late reply and thanks a lot for making this. I would say make it faster is like currently it is single core and if using all cpu cores the osm file building process could be way faster. currently it is taking upto 3 hours for 15 gb file from geojson to PBF or OSM.PBF. I tried my hands with vibe coding but not achieved the goal. So missed you a lot. Big ram consumption should not be an issue but yes it takes at peak 300 Gb ram so used bigger server but it is still slow.

[Vectorial1024]

I keep forgetting this thread exists.

Python 3.14 was released some time ago, and it introduces PEP 734 "multiple interpreters in the stdlib" (see https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-pep734).

My takeaway from the update notes is that, by having multiple interpreters in the same process, we can achieve some sort of efficient "multithreading". But, this feature (just like the threading module, as noted in the text) does not support WASI (aka WebAssembly).

The 3.14 "multiple interpreters" is not fully optimized yet, but they promise further improvements later.

It seems we may need a decision: do we embrace "multiple interpreters" and reject WASI environments from using this library?

---

Disclaimer: haven't used this new module yet, so no real info on whether it will be helpful. But in theory it feels very helpful.

[kabilsharma]

Man I am no coder and only do vibe coding. You can forget this but i tell you this is such a beautiful library. here what i got from calude it sounds so simple but vibe coding is not

Your insight about Python 3.14's multiple interpreters is spot-on, but there's an important clarification:
Python 3.14 is currently in beta (final release October 2025) Python Hits the Big Screen and Other Python News for June 2025 – Real Python, so it's not fully released yet. However, PEP 734 has been approved by the Python Steering Council PEP 734: Multiple Interpreters in the Stdlib - #36 by barry - PEPs - Discussions on Python.org and the concurrent.interpreters module is already available in the 3.14 beta concurrent.interpreters — Multiple interpreters in the same process — Python 3.14.0b3 documentation.
Regarding the WASI decision:
I'd recommend a hybrid approach rather than an either/or decision:

Keep current implementation as default - maintains WASI compatibility
Add optional multiple interpreters support - detects Python 3.14+ and automatically uses it when available
Feature flag - let users choose: --use-subinterpreters for speed vs default for compatibility

Implementation strategy:
pythontry:
if sys.version_info >= (3, 14) and not is_wasi_environment():
from concurrent.interpreters import Interpreter
use_subinterpreters = True
else:
use_subinterpreters = False
except ImportError:
use_subinterpreters = False
Why this approach works:

Multiple interpreters provide "an opportunity for changes in CPython that will allow simultaneous use of multiple CPU cores" PEP 554 – Multiple Interpreters in the Stdlib | peps.python.org
No breaking changes for existing users
Future-proofs the library as Python 3.14 adoption grows
WASI users aren't excluded

[Vectorial1024]

Ultimately I am not the decision maker, but depending on the situation, we might make it backwards compatible.

And then we don't need to act until October 2025 (see PEP 745 Python 3.14 release schedule https://peps.python.org/pep-0745/); good find on the beta status of Python 3.14!

[kabilsharma]

roelderickx this is out of scope work for you but could help if you can help us make it faster thanks. this is such a beautiful work you have done it needs support.