Tokyo's Duplicate Image Problem: The Key Decisions Ahead for City Archives and Digital Records

Tokyo's Bureau of General Affairs is sitting on a problem that keeps getting larger. Across the metropolitan government's shared servers — spanning offices from the Shinjuku headquarters tower to ward-level facilities in Kōtō and Nerima — duplicate image files have accumulated into the tens of thousands, bloating storage infrastructure and slowing retrieval for everything from public records requests to disaster-response mapping. The question now is who decides how to clean it up, and who pays.

The timing matters. Tokyo's inbound tourism surge, which pushed visitor numbers to record levels through 2025, forced rapid expansion of the metropolitan government's public communications photography operation. Event coverage, infrastructure showcases, multilingual tourist guides — each campaign generated parallel image libraries, often uploaded by different departments with no central deduplication protocol in place. The result: redundancy at scale, and an administrative headache that budget planners can no longer quietly defer.

What the Problem Actually Looks Like on the Ground

The Tokyo Metropolitan Archives in Kōtō Ward holds physical and digital records stretching back decades. Digital archivists there have flagged the duplicate-image issue in successive internal reviews, noting that different scanning projects — including the digitisation of Edo-period maps and post-war urban planning photographs — produced overlapping file sets when multiple contractors submitted deliverables without a unified naming convention. The National Diet Library's Tokyo branch in Chiyoda, which coordinates with city institutions on shared cataloguing standards, has pushed for adoption of perceptual hashing tools that can identify near-duplicate images even when file names or metadata differ. Neither institution has yet committed to a joint procurement timeline.

The Tokyo Metropolitan Government's digital transformation roadmap, released in fiscal year 2024, flagged storage rationalisation as a tier-two priority. That classification matters bureaucratically: it means the bureau must pass a two-stage review before committing contracts above ¥50 million. Deduplication software packages evaluated by the metropolitan IT division in late 2025 ranged from approximately ¥8 million for a single-department licence to more than ¥120 million for a government-wide deployment with ongoing maintenance, according to the procurement assessment framework Tokyo uses for software services.

Ward governments are watching the metropolitan decision closely. Shibuya Ward's digital office has already piloted a smaller-scale image audit covering its tourism and event photography from 2022 to 2024, identifying roughly 14,000 redundant files — around 30 percent of that catalogue — in a process that took three months and involved two contracted engineers. The pilot has been cited internally as a proof of concept, though Shibuya has not yet moved to a permanent deduplication workflow.

The Decisions That Will Define What Comes Next

Three choices now sit in front of metropolitan planners. First, whether to pursue a centralised, city-wide deduplication system or allow each bureau and ward to procure independently — a fragmented approach that risks recreating the same problem in five years. Second, whether the technical specification should prioritise speed of deduplication or archival accuracy, since aggressive automated deletion carries the risk of removing images that appear identical but have different legal provenance — a concern the Tokyo Metropolitan Archives has raised in writing. Third, whether the project gets folded into the broader Digital Agency alignment process that the national government has been pressing metropolitan administrations to adopt since 2022.

The yen's sustained weakness has pushed up the cost of cloud storage contracts denominated in dollars, adding urgency to reducing unnecessary data volume. Organisations paying for overseas server capacity — several Tokyo cultural institutions use AWS infrastructure based outside Japan — are absorbing storage cost increases that compound each renewal cycle.

The next formal review is scheduled for the third quarter of fiscal 2026, meaning a procurement decision could realistically land before the end of calendar year 2026. If metropolitan planners opt for city-wide centralisation, ward governments will likely be required to migrate their image libraries to a unified platform — a process that Shibuya's pilot suggests could take months per institution. The alternative, leaving each entity to manage its own redundancy, kicks the problem further down the road and deeper into the budget.