HTML-in-Canvas

(github.com)

216 points | by dannyobrien 1 day ago

39 comments

  • samwillis 1 day ago
    There is a lot valid concern on accessibility and abuse this could result in, but it think it's important to see the other side of the argument.

    There was a really good thread on Twitter a couple of days ago:

    > In light of recent Figma news, lemme reiterate that of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS

    https://x.com/_chenglou/status/1951481453046538493

    And a few choice replies:

    > t’s kind of crazy that a platform specifically designed for presenting text doesn’t provide functionality to manipulate text at a detail level

    > Brute forcing text measurement in tldraw breaks my heart

    Love it or hate it, the web is a platform for application development, making this easer is only good for everyone.

    My argument on web APIs is what we should continue to go lower level, and so font and text metrics APIs for canvas would be awesome and an alternative to this. But I'm also a proponent of "using the platform" and for text layout, web engines are incredible, and very performant. Extending that capability to layout inside a canvas enables many awesome features.

    One that I've repeatedly gone back to over the years is paginated rich text editing. It's simply impossible to do with contenteditable in a product level way - one of the reasons Google docs has a custom layout engine. This proposal would enable full use of contenteditable for rich text, but with full page/print layout control.

    I hope it lands in the browsers.

    • chrismorgan 1 day ago
      > of all the goods that can happen to the web, 90% of them can't happen due to not having access to font rendering & metrics in JS

      I’d be interested to see a representative excerpt of this person’s “goods that can happen to the web”, because it sounds pretty ridiculous to me. Not much needs that stuff, and a lot of that stuff is exposed in JS these days, and a lot of the rest you can work around it without it being ruinous to performance.

      It’s also pretty irrelevant here (that is, about HTML-in-Canvas): allowing drawing HTML to canvas doesn’t shift the needle in these areas at all.

      • heresie-dabord 1 day ago
        > excerpt of this person’s “goods that can happen to the web"

        100% of my concern about the Web is about privacy and security... and why they don't happen.

    • MartinMond 1 day ago
      > One that I've repeatedly gone back to over the years is paginated rich text editing. It's simply impossible to do with contenteditable in a product level way - one of the reasons Google docs has a custom layout engine.

      As do we at Nutrient, we use Harfbuzz in WASM plus our own layouting - see the demo here: https://document-authoring-demo.nutrient.io/

      Getting APIs for that into the Platform would make life significantly easier, but thanks to WASM it’s not a total showstopper.

      Btw, I saw you’re working on sync at ElectricSQL - say hi to Oleksii :)

      • Waraqa 21 hours ago
        I just tested this demo and it noticed it doesn't support Arabic text rendering (the letters should be connected) which is a main feature of Harfbuzz
      • skrebbel 22 hours ago
        Woa that's some heavy SVG lifting going on there!

        If I get it right, every glyph used from the given font is rendered once as a SVG path (upside down! huh!), and then the whole page is a single huge SVG element in which every typed character is a <use> with a reference to that rendered glyph, translated with a CSS transform to the right place (i assume these coordinates come out of HarfBuzz?). Kinda mad that you had to redo 90% of the browser that way but the result is pretty impressive!

        I'm curious why you render the glyphs to paths and not have the browser render those directly using eg svg <text> elements?

        Was it hard to get this to work cross browser?

        ps. srsly I love this about the web. You're doing this amazing engineering feat and I can just pop the trunk and learn all about it. Obviously feel free to not answer anything that's deemed a trade secret, I'm just geeking out hard on this thing :-) :-)

        • rjsw 19 hours ago
          You can't size other SVG elements around text ones since you don't know how much space the text element will occupy.
          • skrebbel 18 hours ago
            I don't mean HTML text nodes, I mean still the single big SVG like they do now, but with SVG <text> elements instead of <path> elements. They do know (I suppose) how much space that element would take since they're asking HarfBuzz to tell them.
            • rjsw 17 hours ago
              You can't know the size of an SVG <text> element.
      • qingcharles 21 hours ago
        And, I think we've come full circle. I'm pretty sure that's how I was rendering text for the online office suite[] I wrote in ~1998 -- a Java Applet embedded in the browser.

        [] VCs: "We're not investing in this crap! No company in their right mind would store their precious, confidential documents on the Internet!"

    • troupo 1 day ago
      > I hope it lands in the browsers.

      Why would you want world's least performant layout/UI engine infect canvas? This literally just cements the situation you quote about having no access to good APIs.

      A reminder that Figma had to "create a browser inside a browser" to work around DOM limitations: https://www.figma.com/blog/building-a-professional-design-to...

      > It's simply impossible to do with contenteditable in a product level way - one of the reasons Google docs has a custom layout engine. This proposal would enable full use of contenteditable for rich text, but with full page/print layout control.

      Why would it enable contenteditable for rich text if you yourself are saying that it doesn't work, and Google had to implement its own engine?

  • ha1zum 1 day ago
    This could have a great utility, but for me HTML in canvas in HTML feels so cursed.

    To make it make sense in my opinion canvas should already be a first class format for web browsers, so it doesn't have to be inside a HTML.

    Then we would have a choice of HTML-first page with canvas elements in it, or a canvas-first page with HTML elements in it.

    But what do I know.

    • bapak 1 day ago
      I don't see a problem with what you're saying at all. For reference, you can already have HTML in SVG in HTML in SVG.

      If you have a canvas-first page, where do you store the title? Right, in <title>, so

          <!doctype html>
          <title>My canvas site</title>
          <canvas style="fill all">
      
      
      In reality they should really just allow content in the canvas element and call it a day:

          <canvas type="html">
              <h1>Canvassing</h1>
          </canvas>
      • ha1zum 1 day ago
        > you can already have HTML in SVG in HTML in SVG

        It's kind of different because SVG and HTML are both XML-like text-based format, doesn't feel that wrong to mix them together. Unlike with canvas..

      • koolala 1 day ago
        Oh god that canvas h1 code looks so sexy. Imagine if a canvas could dynamically resize too.
        • bornfreddy 18 hours ago
          Yes - and maybe you could call it something else, for example... div? ;-)
      • chii 1 day ago
        but then what's the point of the canvas here? Unless if it was possible to mix and match canvas painting operations seamlessly with the declared elements...
        • bapak 1 day ago
          This post is titled HTML-in-Canvas, so you can find the point in the link. A lot of people just want the freedom of canvas rendering/shading and the flexibility of HTML/CSS. Current options may force you to create a layout engine from scratch for example.
        • koolala 1 day ago
          Opening it in WebXR!
    • socalgal2 1 day ago
      canvas first sites suck. They can't use any system services as it would all be a privacy issue. They can't use the system dictionary for correction since to do so they'd need the contents of the dictionary or at least a way to query user customized corrections. Similarly they can't offer the system level accessibility but end up having to roll their own in which case, every app that uses canvas has a completely different UI.
      • nemomarx 1 day ago
        That's basically how flash sites worked, and those were pretty common?
        • desiderantes 1 day ago
          They said that those kind of sites sucked. Common or not.
        • yjftsjthsd-h 1 day ago
          Yes, and Flash sites had severe usability issues.
    • jonplackett 1 day ago
      I think you invented Flash, though saying that embedding flash in harm as also a pain!
      • 38 1 day ago
        [dead]
    • Liron 1 day ago
      What if you want an HTML-first page with a canvas in it, but then you realize you want some layout/styling for the text within the canvas? Seems unnecessary to propagate that situation up to the type of top-level page.
      • thrance 1 day ago
        And then what if you realize you need a canvas-in-the-html-in-the-canvas? It's endless. Canvas-first makes sense, it's basically how it works everywhere outside of the web. Start with the smallest abstractions and build on them (html on canvas) rather than leave escape hatches to your big abstractions because they fail to cover every use cases (canvas in html).
        • sitkack 1 day ago
          If you support the DOM and hitscan, then it doesn't matter. You can red pill Ouroboros yourself all day and not care. Every element a canvas, every raindrop an ocean.
    • lenkite 1 day ago
      Well, folks said NO to WASM DOM Access. So it was inevitable that this would happen.
      • 8n4vidtmkvmk 1 day ago
        Wait, is that never going to happen? I was so excited when WASM was first announced, but then lack of DOM access killed it for me. It was supposed to allow us to use any language instead of just JS.
        • flohofwoe 1 day ago
          You can access the DOM from WASM just fine, you just have to go through a JS shim because the DOM is a Javascript API (just like WebGL, WebGPU, WebAudio and any other API available in browsers).

          In most DOM access libraries (like https://github.com/web-dom/web-dom) this Javascript shim exists but is completely invisible to the library user (e.g. it looks and feels as if WASM would have direct DOM access).

          Why this topic is always brought up I really have no idea, at this point it feels like trolling attempts because from a technical point of view 'direct DOM access from WASM" simply doesn't make a lot of sense. Accessing web APIs from WASM is an FFI scenario, no matter how you look at it.

          • lenkite 23 hours ago
            Actually DOM implementations are all in C++ and DOM interfaces are described in WebIDL. So direct DOM access from WASM is indeed possible if browser vendors chose to open the same. Access via a JS shim is just utterly destroying performance - orders of magnitude worse than mere FFI - and all the trolling attempts are the one pretending otherwise.
            • dfabulich 21 hours ago
              Browser vendors can't simply "choose" to open direct DOM access to WASM.

              When defining standardized DOM APIs in WebIDL, WebIDL assumes that you can use JavaScript strings, JavaScript objects + properties, JavaScript Exceptions, JavaScript Promises, JavaScript garbage collection, and on and on and on. Almost all of the specification of WebIDL itself is about the dozens of types that it assumes the platform already provides. https://webidl.spec.whatwg.org/

              WebAssembly doesn’t have any of those things. As a low-level VM, it supports only modules, functions, bytes, numbers (32-bit and 64-bit integers and floats), arrays (called “tables”), and opaque pointers (“reference types”).

              No one has ever standardized a DOM API for low-level languages. You’d presumably need to start by defining a new low-level WebIDL design language just to define a low-level DOM API.

              Defining WebIDL itself has taken decades.

              Today, the browser vendors aren’t convinced that a new low-level DOM API is worth their time. It’s better to make existing JS web apps faster than it is to begin a multi-year (multi-decade?) project to make a new thing possible that could be better in the long run.

              • lenkite 17 hours ago
                There is the Web Assembly Component Model. Nothing is really preventing browser vendors from exposing a WASM host interface to DOM, exposing it as a Component Model interface. This would allow DOM functions to be invoked from WASM without hand-written/generated JS glue code.

                Nobody is really calling for exposing the full-suite of WebAPI's. But basic DOM access allowing manipulation of page elements would be immediately leveraged by all the WASM-UI frameworks available today. Framework authors would throw out all the generated JS glue code which adds painful overhead pronto with great joy.

                • flohofwoe 1 hour ago
                  Tbh, the WASM Component Model is first and foremost an overengineered mess which probably will add more overhead than a handwritten JS shim just because it is so complex.

                  In the end you'll need to marshall datatypes from one language into another, and that is already a mess between 'native' languages (e.g. a C++ std::string is something entirely different than a Rust or Kotlin String).

                  So in that hypothetical native WASM DOM API, how do you pass something as simple as a string? Let's say the obvious solution would be a ptr/length pair, but then, what encoding UTF-8? UTF-16? UTF-32? No matter what the solution is, you won't find a data representation that directly matches the string representation in all the languages that compile to WASM, so you'll need to do marshalling anyway before calling that hypothetical WASM DOM API.

                  And suddenly the current 'low-tech' solution of letting a JS shim extract the string data from the WASM heap and build a JS string before calling into a web API suddenly doesn't look so terrible anymore.

                  A much more impactful change would be to add more WASM-friendly entry points to web APIs.

                  For instance there's no reason that WebGPU is so 'Javascript object heavy' or uses strings as enum values except that this is common in other Javascript APIs. If WebGPU had additional "WASM-friendly" functions which use plain numbers (as object handles or enum values) a lot of the marshalling overhead when being called from WASM would simply go away.

  • BobbyTables2 1 day ago
    I suppose all we need to do now is to compile a browser into WASM and run that browser inside a canvas element of the main browser…
  • tombh 1 day ago
    Where does SVG's `foreignObject` fit into this? It seems that SVG supports all of thelproposal already? As is evidenced by projects like https://github.com/zumerlab/snapdom that can take "screenshots" of the webpage by copying the DOM with inlined styles into a `foreignObject` tag in an SVG. Then of course that SVG can be rendered to a canvas.
    • progers7 1 day ago
      This proposal is a lot like an easier way to draw foreign object into canvas. This proposal supports new features too, such as updating the canvas when the content changes, and interactivity.
  • mmastrac 1 day ago
    • rapnie 1 day ago
      I searched for it in the text, but it is the missing one to add (and might cross-ref to "TODO: Don't be evil").
  • mircerlancerous 1 day ago
    Please correct me if I'm wrong, but I feel rendering html overtop of canvas solves this with vanilla just fine. Canvas is for rendering things you can't with html, and not replacement for the dom.
    • esperent 1 day ago
      Here's a simple example that's currently very hard to do and requires all kinds of hacky and unsatisfying workarounds:

      1. A 3d model, say of a statue in a museum

      2. Add annotations to the model drawing attention to specific features (especially if the annotations are not just a single word or number)

      If you want the annotations to be properly occluded by the model as you move the camera around, it's hard - you can't use HTML. If you do use HTML, you'll have to do complex calculations to make it match the correct place in the 3d scene, and it will always be a frame delayed, and occlusion is bad - usually just show or hide the entire HTML annotation based on the bounding box of the 3d model (I have seen better solutions but they took a ton of work).

      So you could use 3d text, maybe SDF, but now you've created a entire text rendering system without accessibility or anything like that. Also, if you want anything more than very simple annotations (for example, videos, lists, select menus, whatever) you either have to reinvent them or fall back HTML.

    • flohofwoe 1 day ago
      That only works if the html stuff is on top of everything that's rendered in the canvas, otherwise you need to add another canvas on top of the html (etc pp for each separate z-layer).

      IMHO this step finally starts to fix the "inverted api layer stack" in browsers. All browser rendering should build on top of a universal canvas api.

  • ttoinou 1 day ago
    Soon we’re gonna need Canvas rendering inside of HTML-in-Canvas
    • bastawhiz 1 day ago
      It should already work if the nested canvas uses the same approach. It's not cyclic, though. To make cyclic canvases work, you need to manually draw the parent canvas to a nested canvas.
      • koolala 1 day ago
        Now your thinking with canvas.
  • codelikeawolf 1 day ago
    I immediately got "Pimp My Ride" vibes. Yo dawg, I heard you like HTML so I put HTML inside the canvas inside the HTML.
  • reactordev 1 day ago
    I support this, as odd as it is. There’s times when you’re needing something drawn but can easily reuse an html element from elsewhere. Previously you’d have to render that to a bitmap offscreen and then copy that to a full screen quad or draw it on the canvas. Up until recently, even if you tried to z-index elements with position absolute it would be visually overwritten by the canvas (I think this is mostly fixed though).

    I don’t know if this is the best solution but it’s better than previous hacks. IF you need to go that route. Basically html2canvas.

  • talkingtab 1 day ago
    There is a real problem using canvas to replace HTML.

    Not all but most HTML. I have not found a good solution for the issue of doing something like MDX in canvas. I have tried SDF, looked at 2D canvas Text, Troika, MSDF. You can get text, it is just that laying it out is very difficult. React three drei has the ability to put HTML into the threejs ecosystem, but there are issues about CSS and text that make that impractical.

    For me the use case is very simple. I would like to take an MDX file and show it in a mesh. Laid out. Maybe I am missing something because I am new to the whole threejs thing, but I really tried.

    A good article about text https://css-tricks.com/techniques-for-rendering-text-with-we...

    And an example from the above article: https://codesandbox.io/p/sandbox/css-tricks-msdf-text-fks8w

    This shows it can be done, I gave up trying to reproduce it in React-three-fiber.

    Why? Personally, I think the use of 3D graphics produces an interface for users that is an order or magnitude better for users. The real question (and an interesting one to consider) is why are we still building HTML first websites?

    • squidbeak 1 day ago
      Why do we build HTML first websites? Because most websites serve documents, however much developers might wish they were applications.
  • SeanAnderson 1 day ago
    I read the title and said "shut the fuck up, don't do that." but then I read the rationale and it's fair. It's true there is no layout engine inside canvas, and that is a pain, but I'm not sure it's such a pain as to invite this recursive hell.
    • nine_k 1 day ago
      One of the more senior engineers I worked with told me: "Every real-life data structure I encountered was tree-like".

      It would be easiest to just ask the browser to render a fragment of HTML onto a canvas, or onto some invisible bitmap, like you can with most other UI toolkits.

      • monster_truck 1 day ago
        They would never do this because of fingerprinting, which is already the cause of most of the reasons we cannot 'just' do a lot of things, unfortunately.

        E: And the infamous other half: malware. A bit over a decade ago malware devs started using canvas to do things like hide fragments inside of bitmap data in seemingly harmless ads and then a second script would extract and assemble it to evade detection.

        • rezonant 1 day ago
          The web platform can already do this, see SVG foreignObject elsewhere in the thread. The key is to have the proper bounds in place (cross origin resources, etc), and the infrastructure for that is already in place.

          This just removes the extra step of relying on SVG to accomplish rendering the HTML, adds a path for getting this content into the accessibility tree, and supporting input on the rendered elements.

      • mook 1 day ago
        Yeah, that's already available in Firefox for chrome/extensions, but not allowed for the web due to fingerprinting and other security risks. For example, rendering an iframe of your bank account…

        https://searchfox.org/mozilla-central/rev/f691af5143ebd97034...

      • bongodongobob 1 day ago
        > Every real-life data structure I encountered was tree-like

        I don't understand what the takeaway is here. Is that surprising? Is it not? What does "real-life" mean?

        • nine_k 1 day ago
          As found in production code, not (only) in textbooks and academic papers.

          This is an exaggeration, of course.

      • do_not_redeem 1 day ago
        > Every real-life data structure I encountered was tree-like

        What does this even mean? Is a hash map "tree-like" somehow? Or is a hash map just a toy data structure with no real-life use cases?

        • UltraSane 1 day ago
          Nested hash maps are trees
        • koolala 1 day ago
          branch map. dictionary tree.
    • teaearlgraycold 1 day ago
      You could disallow recursion.
  • hyperhello 1 day ago
    I see reliable rich text editing as more of a priority.
  • c-smile 1 day ago
    From the very beginning in Sciter an Image can be constructed in two ways at runtime ( other than just getting loaded image reference):

    1. By painting on it using Canvas/Graphics API:

        new Graphics.Image(width, height, painter(graphics) [,initColor]);  
    
    Where _painter_ is a function used for paining on the image surface using Canvas/Graphics reference.

    2. By making snapshot of the existing DOM element:

        new Graphics.Image(width, height, element [,initColor])
    
    Such images can be used in DOM, rendered by other Canvas/Graphics as also in WebGL as textures.

    See: https://docs.sciter.com/docs/Graphics/Image#constructor

  • wg0 1 day ago
    What we also need is a simple API that given a dom element, takes a screenshot of the subtree.

    Nothing such is available.

  • fullstackwife 1 day ago
    Text rendering on canvas is currently a huge pain, custom layouting code, lack of accessibility, so I'm happy to see this!
  • pacifika 1 day ago
    Yeah let’s break user control over text.
    • flohofwoe 1 day ago
      Improving accessibility for canvas rendered content is one of the first bullet points in the proposal, so I guess they thought of that little detail.
      • pacifika 1 day ago
        This is just building a browser engine inside a browser engine, can’t see it happen.

        Also unless it has the same feature as the level of accessibility it has no it would be a step back.

        It would be a gargantuan job.

        • flohofwoe 1 day ago
          It's not a browser engine in a browser engine, just making the already existing browser engine available in another context. I bet that at least 90% of the dom implementation code will be shared (since internally the dom is almost certainly rendered through the same renderer process that also runs webgl and webgpu.
  • pbohun 21 hours ago
    The whole point of canvas is to get away from the awful kludge that is HTML and CSS. I'd much rather see a new simple UI library that's developed for canvas.
  • tantalor 1 day ago
    • Waterluvian 1 day ago
      45kb gzipped is pretty beefy but incredibly small when you consider just what it takes to make this work today. If I understand correctly, it’s basically a DOM and CSS renderer.
    • kizer 1 day ago
      There's a bunch of CSS, etc. not supported by that. It would be great to have access to a native API to get bitmaps of the DOM.
      • masswerk 1 day ago
        Well, a fingerprinting dream…

        Meaning, no way, just for the security aspect.

  • j45 3 hours ago
    This isn't bad, or good.

    Having this type of control, for certain use cases can be perfectly valid.

    It also feels Flash like.

    The javascriptists began on a journey 15 years ago to replace Flash. Things have gotten more complicated before becoming simpler, but maybe things will head in a direction soon.

    Flash itself was actionscript (ECMAScript) which is the same syntax as Javascript.

  • cheeaun 1 day ago
    Kinda reminds me of Flipboard https://engineering.flipboard.com/2015/02/mobile-web (2015)
  • nikeee 1 day ago
    Is that being made to make Flutter not being like Flash on the web?
    • kbcool 1 day ago
      I was searching for a comment like this.

      It sounds like a crazy workaround for Flutter's strange architectural choices

  • Ono-Sendai 23 hours ago
    I would love this. I have to do disgusting hacks to get an embedded browser window in my metaverse (https://substrata.info/) that uses webgl. The disgusting hack is to render the browser window behind the main webgl canvas, and then punch a hole through the webgl canvas with zero alpha. Event handling (mouse, keyboard) is also a total pain.
  • jonplackett 1 day ago
    Can it just be CSS on canvas - so you can do styling and layout nicely but not weirdness of an html sandwich?
  • kookamamie 19 hours ago
    HTML with a canvas with HTML with a canvas…
  • rikroots 1 day ago
    Please excuse me for spamming this thread with examples of how my canvas library approaches these issues:

    > Use case: Styled, Laid Out Content in Canvas. There’s a strong need for better styled text support in Canvas. Examples include chart components (legend, axes, etc.), rich content boxes in creative tools, and in-game menus.

    Single line, unstyled text is relatively easy using the Canvas API. Multiline text is a world-of-pain. Styled text is a completely separate world-of-pain. Underlined text? Same! So that's gives us a problem space of world-of-pain-cubed. Don't talk to me about RTL text, vertical text, CJK punctuation, Thai text ignoring spaces as a word separator, heavily kerned fonts (staring at you, Arabic and Devangari), etc.

    Demo: https://scrawl-v8.rikweb.org.uk/demo/canvas-207.html

    This demo takes the following html markup and displays it in a truncated circle shape. The styling itself happens in CSS - see here: https://github.com/KaliedaRik/Scrawl-canvas/blob/v8/demo/can...

      const westernText = '<span class="underline">Lorem</span> ipsum <b>dolor sit</b> amet, con&shy;sectetur  adi&shy;piscing &eacute;lit, sed <s>do eius-mod</s> <u>tempoj yn&shy;figizqunt</u> ut <span class="stamp-outlined">labore et dolore</span> <span class="green-highlight">magna aliqua.</span> Ut enim ad <span class="bold">minim veniam,</span> quis <span class="letter-spaced">nostrud</span> exercit-ation <span class="strike">ullamco laboris</span> nisi ut aliquip ex ea <span class="make-monospace">"commodo"</span> consequat. Duis <em>(aute irure d&ouml;lor)</em> in reprehenderit &icirc;n <i>voluptate</i> velit &copy;2024 <i>esse &lt;cillum&gt; <b>dolore</b> eu fugiat nulla</i> pariatur. <span class="red">Excepteur sint</span> occaecat &iexcl;cupidatat! <strong>non proident,</strong> <span class="word-spaced">sunt in culpa qui</span> offici&thorn;a deserunt <span class="make-bigger"><span class="green-highlight">mollit</span> anim</span> id est laborum.';
    
    As for the other things I don't want to talk about, see this other demo which attempts to overcome those issues: https://scrawl-v8.rikweb.org.uk/demo/canvas-206.html

    > Use case: Accessibility Improvements. There is currently no guarantee that the canvas fallback content currently used for <canvas> accessibility always matches the rendered content, and such fallback content can be hard to generate. With this API, elements drawn into the canvas bitmap will match their corresponding canvas fallback.

    I welcome and applaud this focus on making canvas text accessible. However it's not enough (in my highly opinionated opinion) to just reflect the text back into the DOM. People using screen readers probably don't need every number on the Y axis read out to them every time they navigate onto a chart, they probably just need to hear values as they keyboard navigate the chart.

    Demo: https://scrawl-v8.rikweb.org.uk/demo/modules-001.html

    The canvas element is highly inaccessible - I've tried to detail all the issues that have to be addressed here (again, a highly opinionated take): https://scrawl-v8.rikweb.org.uk/docs/reference/sc-accessibil...

    > Use case: Composing HTML Elements with Shaders. A limited set of CSS shaders, such as filter effects, are already available, but there is a desire to use general WebGL shaders with HTML.

    We already have a comprehensive set of filter effects available through SVG <filter> elements. They are, however, difficult to compose and have a tendancy to be computationally heavy. WebGL shaders can be fantastic, but face the (current) limit of how many WebGL canvas elements you can include on a page; they're also difficult to compose.

    For my library's filter engine, I took inspiration from the SVG approach. Details can be found here: https://scrawl-v8.rikweb.org.uk/docs/reference/sc-filter-eng...

    > Use case: HTML Rendering in a 3D Context. 3D aspects of sites and games need to render rich 2D content into surfaces within a 3D scene.

    HTML canvas elements are just DOM elements, and can be 3d-rotated like other elements. Interacting with 3d-rotated canvas elements is an interesting problem space.

    Classic rotating cube demo: https://scrawl-v8.rikweb.org.uk/demo/dom-008.html

    Tracking the mouse over a 3d-rotated canvas element demo: https://scrawl-v8.rikweb.org.uk/demo/dom-013.html

  • donatj 1 day ago
    I feel like this is bound to get kneecapped for the same reason as other canvas features, fingerprinting.
    • cluckindan 1 day ago
      Fingerprinting is not an issue of canvas drawing, it is an issue of canvas reading (ImageData).
  • TheRealPomax 22 hours ago
    I pointed out text deficiencies in Canvas decades ago, only for Hixie to go "so put your text in a div and position the div over the canvas".

    I wonder if the working groups are still run by that attitude.

  • tinalovely101 1 day ago
    It seems to be impossible with html version.
  • sampton 1 day ago
    Servo + wasm.
  • russellbeattie 1 day ago
    How about a flag on the body element like:

        <body canvas="true"></body> 
    
    This would make the entire visible page into a canvas-like drawing surface which also renders DOM elements as per usual. At some level there's a process which rasterizes the DOM - opening drawing APIs into that might be a better solution.

    It's sort of the same thing as HTML in canvas conceptually, but architecturally it makes DOM rendering and canvas rendering overlapping equals with awareness going both ways. E.g., a line drawn on the page will cause the DOM elements to reflow unless told to ignore it.

    • pyrolistical 1 day ago
      Screen readers all cried in unison
      • koolala 1 day ago
        Why wouldn't they work normally. The HTML is still modeled as a ground truth.
    • koolala 1 day ago
      Would this mean a canvas could also automatically resize? This would be a new era for html composition that could surpass iframes.
      • flohofwoe 1 day ago
        Isn't this already trivial? You just need to be aware that the framebuffer size of a canvas element is different from its dom element size, but you can easily glue the canvas framebuffer size to the element size by listening for resize events (this is for webgl and webgpu canvases, don't know about 2d canvas)
  • turnsout 1 day ago
    Sad to say, but the first thing I thought was "oh good, a new fingerprinting target."
    • koolala 1 day ago
      Can you give an example? If I used this to load HTML into a 3D environment every user could have the same virtual screen size.
      • turnsout 1 day ago
        Every browser renders HTML slightly differently, so you could render some arbitrary HTML into the canvas and then read out the pixels.
        • koolala 1 day ago
          I see what you mean. Isn't that already what the user agent tells you?
          • turnsout 20 hours ago
            Oh my sweet summer child. The game of cat and mouse is so much more vicious than that.
  • ivanjermakov 1 day ago
    RIP accessibility
    • progers7 1 day ago
      Accessibility is a key reason for this proposal. Today, canvas accessibility is quite limited. This proposal enables the browser to know how accessible dom elements map to canvas pixels.
    • koolala 1 day ago
      html is the most accessible language. video games have to hardcode everything.
  • erichocean 1 day ago
    My Blossom library has a layout engine for Canvas [0]

    [0] https://github.com/erichocean/blossom

  • charcircuit 1 day ago
    This sounds really useful for being able to use standard web technology with webxr.
    • socalgal2 1 day ago
      I've never understood why they couldn't have just used the existing 3D CSS for WebXR. All the data is there, all they need to do is render the DOM from 2 POVs, one for each eye. They could even have had some standard to let it auto composite with WebGL.
  • deadbabe 1 day ago
    true horror is finding every element of the HTML page your on has been rendered in a Canvas
  • troupo 1 day ago
    > As a result, <canvas>-based content suffers in accessibilty, internationalization, performance and quality.

    Ah yes. Because HTML is renowned for its performance and quality.

    Instead of pushing this idiocy they should add the things that canvas lacks instead

  • wetpaws 1 day ago
    [dead]
  • tones411 1 day ago
    Next someone will want to render a canvas in the html in the canvas.