There are also various libraries to show OSM as background map on a website and allow you to draw features on top. Those support various formats (GPX, GeoJSON, ...) Leaflet and MapLibre are the biggest here, with leaflet being the easiest and MapLibre being very flexible and performant
If by layers you mean tiles, you can make a server providing ZXY tile images. Some OSM clients (e.g. BRouter) allow adding custom tile servers for baselayer and overlays.
Is there any standard protocol/format for "layers" that could be added to such Map apps?
Let's say I want to write a plugin to add my own kind of data, and then I want to export this new kind of data so that someone can import them.
Or I want to create a website with my own data on top of existing one.