Choosing and Converting Between Formats

3.6. Choosing and Converting Between Formats#

3.6.1. Learning Objectives#

By the end of this section, you should be able to:

Select an appropriate geospatial format for a given task using a comparison table.
Use Fiona for low-level, memory-efficient inspection and streaming of large files, and for writing with an explicit schema.
Apply a safe, verifiable workflow when converting data between formats.
Predict what each format changes or loses in a round trip (field names, geometry types, field order and names, write failures), and check for it.

Section Goal

The previous sections covered six formats and one live data source in isolation. This section steps back and compares them side by side, then gives you a repeatable, verifiable procedure for converting between formats without silently losing fields, geometries, or coordinate reference system information.

3.6.2. Format Selection Guide#

The table below summarizes when to choose each format covered so far.

Table 3.34 Format Selection Guide#
Format	Best For	Avoid When
GeoJSON	Web APIs, small datasets, rapid sharing	Large datasets (> a few hundred MB)
Shapefile	Receiving legacy/government data	Creating new outputs (use GeoPackage instead)
GeoPackage	Multi-layer file storage, replacing Shapefile	Need web-native JSON delivery
File Geodatabase	Receiving data from ArcGIS-based systems, enterprise workflows needing topology/relationships	Open-source-only pipelines with no ArcGIS access downstream
KML	Google Earth visualization, field data	Analytical pipelines (too verbose, limited schema)
GML	Government/standards-compliant data exchange	Rapid prototyping (schema complexity overhead)
OSM	Global road networks, POIs, building footprints	Data with quality/completeness requirements

Behind those recommendations are a few technical properties that decide what a format can and cannot hold [Butler et al., 2016, ESRI, 1998, Portele, 2007, Wilson, 2008, Open Geospatial Consortium, 2023]:

Table 3.35 Format Capabilities at a Glance#
Format	Container	Layers per file	CRS	Field names	Styling
GeoJSON	One text file	One	Fixed: WGS 84, longitude-latitude	Any	No
Shapefile	Several files	One	Any, in a `.prj` file	10 characters	No
GeoPackage	One SQLite file	Many	Any	Any	No
File Geodatabase	One directory	Many	Any	Any	No
KML	One XML file (or KMZ)	Folders	Fixed: WGS 84, plus altitude	Fixed vocabulary, plus `ExtendedData`	Yes
GML	XML, often with an `.xsd`	Feature types	Any, named by `srsName`	Defined by the application schema	No
OSM	`.osm.pbf` or XML	Thematic layers made from tags	Fixed: WGS 84	Free-form tags	No

3.6.2.1. Format, Storage, and Delivery Are Different Choices#

As the chapter began, the format used to receive data does not have to be the format used for analysis or delivery. Most workflows pass through three stages, and each stage has different needs:

Table 3.36 Choosing a Format by Workflow Stage#
Stage	Goal	Good choices
Receive	Keep exactly what the provider sent	The original format, left untouched, with the download date
Analyze	Reliable, typed, multi-layer storage	GeoPackage, or a spatial database for shared or very large data
Deliver	Match the recipient’s tools	GeoJSON for web maps and APIs; GeoPackage for GIS users; KML for Google Earth; GML where a standard requires it; Shapefile only when asked

3.6.3. Working with Fiona for Fine-Grained Control#

While GeoPandas handles most reading/writing needs, Fiona [Gillies and others, 2011–2026] provides lower-level OGR access [Rouault et al., 2026] useful for:

Inspecting file metadata (CRS, schema, layer count) without reading all data.
Streaming large files feature-by-feature to avoid loading everything into memory.
Writing features with full control over schema and driver options.

The cell below shows all three. It downloads the Natural Earth country boundaries [Natural Earth contributors, 2026] used earlier in this chapter, stores them as a GeoPackage, inspects the file’s metadata, streams its features to count geometry types without building a table, and finally writes a new GeoPackage with an explicit schema: the field names, types, and text width are stated, not guessed from data.

Show code cell source

Hide code cell source

import shutil
import urllib.request
import warnings
from collections import Counter
from pathlib import Path

import fiona
import geopandas as gpd
import matplotlib.pyplot as plt
import pandas as pd
from shapely.geometry import MultiPolygon, mapping

DATA_DIR = Path("data")
DATA_DIR.mkdir(exist_ok=True)


def download(url, target):
    """Download a file once; reuse the local copy afterwards."""
    if not target.exists():
        request = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
        with urllib.request.urlopen(request, timeout=60) as response:
            target.write_bytes(response.read())
    return target


countries_path = download(
    "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_110m_admin_0_countries.geojson",
    DATA_DIR / "ne_110m_admin_0_countries.geojson",
)
countries = gpd.read_file(countries_path)
gpkg_path = DATA_DIR / "countries.gpkg"
countries.to_file(gpkg_path, layer="countries", driver="GPKG")

# 1. Inspect metadata without reading any geometry into a table
with fiona.open(gpkg_path) as src:
    print("Driver        :", src.driver)
    print("CRS           :", src.crs)
    print("Feature count :", len(src))
    print("Geometry type :", src.schema["geometry"])
    print("First fields  :", list(src.schema["properties"].items())[:4])

# 2. Stream features one at a time (memory-efficient for large files)
with fiona.open(countries_path) as src:
    type_counts = Counter(feature["geometry"]["type"] for feature in src)
print("Geometry types streamed from the GeoJSON:", dict(type_counts))

# 3. Write with full control of the schema
schema = {"geometry": "MultiPolygon", "properties": {"name": "str:60", "population_millions": "float"}}
custom_path = DATA_DIR / "custom_schema.gpkg"
custom_path.unlink(missing_ok=True)
with fiona.open(custom_path, "w", driver="GPKG", crs="EPSG:4326", schema=schema, layer="countries") as dst:
    for _, row in countries.head(5).iterrows():
        geom = MultiPolygon([row.geometry]) if row.geometry.geom_type == "Polygon" else row.geometry
        dst.write({"geometry": mapping(geom),
                   "properties": {"name": row["NAME"], "population_millions": row["POP_EST"] / 1e6}})
print()
print(gpd.read_file(custom_path).drop(columns="geometry"))

Driver        : GPKG
CRS           : EPSG:4326
Feature count : 177
Geometry type : MultiPolygon
First fields  : [('featurecla', 'str'), ('scalerank', 'int32'), ('LABELRANK', 'int32'), ('SOVEREIGNT', 'str')]
Geometry types streamed from the GeoJSON: {'MultiPolygon': 29, 'Polygon': 148}

                       name  population_millions
0                      Fiji             0.889953
1                  Tanzania            58.005463
2                 W. Sahara             0.603253
3                    Canada            37.589262
4  United States of America           328.239523

Fiona reports the file’s schema without building a table: 177 features, EPSG:4326, and a declared geometry type of MultiPolygon, because the GeoPackage promoted every polygon. Streaming the original GeoJSON shows the mix that was promoted, 148 Polygon and 29 MultiPolygon features. In the last step, the field names, types, and text width of the new file are the ones we stated, not ones guessed from the data.

3.6.4. A Safe Conversion Workflow#

For every conversion:

Preserve the source file and note its download date.
Inspect layer names, schema, feature count, CRS, and geometry types.
Convert into a target format that supports the required fields and geometry.
Reopen the output and compare feature counts, CRS, null values, and bounds.
Test a small spatial query or plot before deleting intermediate files.

In short form, the core of steps 3 and 4 is:

source = gpd.read_file("incoming/data.shp")
target = source.to_crs(epsg=32615)
target.to_file("staged/data.gpkg", layer="features", driver="GPKG")

round_trip = gpd.read_file("staged/data.gpkg", layer="features")
assert len(round_trip) == len(target)
assert round_trip.crs == target.crs

The round-trip check catches silent failures such as dropped records, an unexpected driver, or missing CRS metadata.

3.6.4.1. A Round-Trip Experiment#

Different formats lose different things, so it helps to see the checklist at work. The experiment below takes one layer, writes it to six formats, reopens each result, and checks five things: feature count, CRS, field names, geometry types, and whether the write succeeded at all. The test layer is the country boundaries with three deliberate additions: a field with a long name (population_estimate_millions), a date-time field (last_checked), and the natural mix of Polygon and MultiPolygon geometries that the source already has.

Show code cell source

Hide code cell source

original = countries[["NAME", "CONTINENT", "POP_EST", "geometry"]].copy()
original["population_estimate_millions"] = original["POP_EST"] / 1e6       # a long field name
original["last_checked"] = pd.Timestamp("2026-10-18")                       # a date-time value
original_fields = [c for c in original.columns if c != "geometry"]
original_types = original.geom_type.value_counts().to_dict()

WORK = DATA_DIR / "round_trip"
shutil.rmtree(WORK, ignore_errors=True)
WORK.mkdir()

FORMATS = {                                   # label: (file name, GDAL driver)
    "GeoJSON": ("countries.geojson", "GeoJSON"),
    "GeoPackage": ("countries.gpkg", "GPKG"),
    "Shapefile": ("countries.shp", "ESRI Shapefile"),
    "GML": ("countries.gml", "GML"),
    "KML": ("countries.kml", "KML"),
    "File Geodatabase": ("countries.gdb", "OpenFileGDB"),
}
MULTI_LAYER_DRIVERS = {"GPKG", "GML", "KML", "OpenFileGDB"}               # these take a layer name


def convert_and_verify(gdf, label, filename, driver):
    """Write gdf in the given format, reopen it, and report what changed."""
    folder = WORK / label.replace(" ", "_")
    folder.mkdir()
    path = folder / filename
    result = {"format": label}
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        try:
            options = {"layer": "countries"} if driver in MULTI_LAYER_DRIVERS else {}
            gdf.to_file(path, driver=driver, **options)
        except Exception as error:
            result["error"] = f"{type(error).__name__}: {error}"
            return result
    result["size_kb"] = round(sum(p.stat().st_size for p in folder.rglob("*") if p.is_file()) / 1024)
    result["warnings"] = len(caught)

    back = gpd.read_file(path)
    fields_back = [c for c in back.columns if c != back.geometry.name]
    result["features"] = len(back)
    result["crs"] = back.crs.to_epsg() if back.crs else None
    result["geometry"] = back.geom_type.value_counts().to_dict()
    result["fields_lost"] = [c for c in original_fields if c not in fields_back]
    result["fields_added"] = [c for c in fields_back if c not in original_fields]
    return result


results = [convert_and_verify(original, label, *spec) for label, spec in FORMATS.items()]
report = pd.DataFrame(results).set_index("format")
display(report[["features", "crs", "geometry", "fields_lost", "fields_added", "size_kb", "error"]])

	features	crs	geometry	fields_lost	fields_added	size_kb	error
format
GeoJSON	177.0	4326.0	{'Polygon': 148, 'MultiPolygon': 29}	[]	[]	267.0	NaN
GeoPackage	177.0	4326.0	{'MultiPolygon': 177}	[]	[]	324.0	NaN
Shapefile	177.0	4326.0	{'Polygon': 148, 'MultiPolygon': 29}	[population_estimate_millions, last_checked]	[population, last_check]	220.0	NaN
GML	177.0	4326.0	{'Polygon': 148, 'MultiPolygon': 29}	[]	[gml_id]	375.0	NaN
KML	177.0	4326.0	{'Polygon': 148, 'MultiPolygon': 29}	[NAME]	[id, Name, description, timestamp, begin, end,...	324.0	NaN
File Geodatabase	NaN	NaN	NaN	NaN	NaN	NaN	DataLayerError: Unsupported geometry type

The table is dense, so the next cell turns it into a pass/fail grid with the five checks, and charts the size of each result. A format passes the field-name check only if every original name survived exactly as written.

../_images/format_roundtrip_checks.png — Fig. 3.23 Round-trip checks for the same country layer written to six formats (left) and the size on disk of each result (right).#

Fig. 3.23 shows that only GeoJSON and GML pass all five checks. GeoPackage fails one, but only because single polygons were promoted to multipolygons, so a red cell is a change to inspect rather than always a loss. Shapefile and KML fail on field names, and the File Geodatabase row is red throughout because nothing was written, so its other four checks could not run. The size chart shows that size does not track fidelity: Shapefile is the smallest result and also lost field names, while the two formats that kept everything, GeoJSON and GML, are 267 KB and 375 KB.

Read the grid format by format. Each result below is something this run showed, and each corresponds to a limitation described earlier in the chapter:

GeoJSON kept everything, including the mix of Polygon and MultiPolygon.
GeoPackage kept every field and feature, but every geometry came back as a MultiPolygon: a layer declares one geometry type [Open Geospatial Consortium, 2023], so the writer promoted the single polygons. Nothing was lost, but the types changed.
Shapefile truncated the two long field names to ten characters [GDAL/OGR contributors, 2026] (population_estimate_millions became population, and last_checked became last_check), and GDAL raised warnings that are easy to miss.
GML kept the data and added a gml_id field [GDAL/OGR contributors, 2026]. Its verbose XML made it the largest file, although the differences in size are modest for a layer this small.
KML renamed NAME to Name and added its own bookkeeping fields (description, timestamp, and others) [GDAL/OGR contributors, 2026]: its fixed vocabulary reshapes the schema.
File Geodatabase refused to write at all [GDAL/OGR contributors, 2026], because the layer mixes geometry types. The error is the failure mode that the chapter warned about when it discussed mixed Polygon and MultiPolygon layers.

The last item has a standard fix: convert toward the more general type before writing. The cell below promotes every Polygon to a one-part MultiPolygon and retries.

Retry after normalizing to MultiPolygon:
  features    : 177
  crs         : 4326
  geometry    : {'MultiPolygon': 177}
  fields_lost : []
  size_kb     : 164
  error       : None

Once every geometry is a MultiPolygon, the geodatabase accepts all 177 features with no field lost, in 164 KB. That is about half the size of the GeoPackage in this run, but, as before, it is one observation rather than a rule.

The experiment illustrates the rule that closes this section: a successful write is not proof that no fields or geometries were lost. Use the same inspect-convert-reopen checklist every time, and compare the reopened output with the source, rather than trusting the absence of an error message.

For guided practice applying this checklist, see the labs.

Practice

Pick a layer of your own with at least one field name longer than 10 characters. Write it to a Shapefile and to a GeoPackage, reopen both, and list the fields that differ from the original. Then write the same layer to GML or KML and list the fields that were added. Finally, record in two sentences which format you would choose for analysis, and which for delivery to a web map, and why.

3.6.5. Key Terms#

Table 3.37 Key Terms for Format Selection and Conversion#
Term	Meaning
Round trip	Writing data to a format, reading it back, and comparing the result with the original.
Explicit schema	A statement of field names, types, and widths, instead of letting software guess them from data.
Streaming	Reading a file one feature at a time, so the whole file never needs to fit in memory.
Normalization (of geometry types)	Converting a mixed layer to its most general geometry type, for example `Polygon` to `MultiPolygon`.
Silent failure	A conversion that raises no error but drops, renames, or alters data.
Receive, analyze, deliver	The three workflow stages, each of which may call for a different format.

3.6.6. Review Questions#

Which format would you choose to receive, to analyze, and to deliver to a web map? Explain each choice.
Name two things Fiona can do that are awkward in GeoPandas.
A Shapefile export raises no error, but two columns are missing from the output. What happened?
Why did GeoPackage change the geometry types in the experiment, and was any data lost?
A File Geodatabase write fails with an “unsupported geometry type” error. What is the likely cause and the fix?
List the five checks of a safe conversion, and give one failure each catches.

3.6.7. Summary#

The geospatial file format ecosystem has evolved from proprietary, single-purpose formats (Shapefile, KML) toward open, flexible containers (GeoJSON, GeoPackage) that better serve modern data engineering workflows, while OpenStreetMap and File Geodatabases each add their own trade-offs around governance and tooling. Selecting the right format for each pipeline stage, and verifying every conversion with a round-trip check, prevents silent data loss: as the experiment showed, formats truncate names, change geometry types, rename fields, or refuse to write, and only reopening the output reveals it.

The next section moves from files to databases, examining how PostGIS and SpatiaLite extend the relational model with spatial capabilities to support scalable, multi-user geospatial workflows.