Max Limper: 0:05

Welcome to Virtual4Real, the Rapid Pipeline Podcast. I'm Max, your host, and with me today is the amazing Patrick Cozzi. Welcome, Patrick.

Patrick Cozzi: 0:13

Hey Max, hey everybody.

Max Limper: 0:15

Yeah, great to have you here on the podcast. And I think many people already know you, but for the few ones who don't, quick introduction. Patrick Cozzi is a key figure in the 3D industry. He has been contributing to the GLTF standard that is now called the JPEG 43D, like a really important format, but also wrote several books, both as an author as well as an editor, and has been a lecturer at the University of Pennsylvania about 3D graphics. And then he's also the CEO and co-founder of Cesium, and we'll learn more about Cesium during this episode. It's a world leading framework for 3D geospatial visualization, and then got acquired by Bentley Systems, where Patrick is now chief platform officer. Yeah, Patrick, again, great to have you here. And my first question would be when we think about the digital twins, right? I remember you are working with massive geospatial data, but also integrating other data sources into these visualizations and CSIM doing really amazing work. What fascinates you about digital twins, or maybe I should say the ultimate digital twin of the entire world that you are working on?

Patrick Cozzi: 1:24

Well, Max, first look, thanks for all your contributions to the 3D community, both the technical work that you've done for GLTF and then also what you're just doing for the community, like with this podcast here. Yeah, I'm quite interested in digital twins and digital twins at scale, coming from lots of different data. I did my master's thesis at University of Pennsylvania, which was focused on city rendering. And that's when we learned techniques like a hierarchical level of detail and all the viewfrost and calling and all the cash management that went into that. A lot of that knowledge went into 3D tiles and went into cesium, where now we're trying to do a full digital, customizable digital twin of the world coming from lots of different data, certainly 3D buildings, but also terrain, subsurface, undersea, and so on. Certainly the technical challenges, they've always fascinated me and they always will. But what I really like is what the community ends up building when you give them an open platform like this. So CCM started in Aerospace because it's open source. Like you can imagine what we thought like when Red Bull picked it up to do a digital twin at the XALPS power gliding and hiking race. And you're using the train and imagery layer, and they're showing the athletes and where they are. That's just so cool to see. And now we recently released CSEM Moon, which gives you a train and imagery based on open data for the moon, where you can do your simulations. And then we just released Cesium Mars, which was open data from NASA and European Space Agency, again, let me envision what it would be like to build, for example, infrastructure on Mars.

Max Limper: 2:58

So Cesium is an amazing platform, as you said. It's not just open source software, it's an entire platform, an entire ecosystem. And it's really cool to see how it joined the Bentley Systems family. And we'll talk about Bentley's iTunes platform in a bit. But how is the connection? How does Cesium integrate with Bentley Systems offering and the strategy?

Patrick Cozzi: 3:23

Yeah, so Cesium, we joined Bentley just over a year ago, and it's been everything that I thought it would be. When you look at the Cesium journey, right, we started in aerospace, we made it open source, then just guided by the open source community and really engaging with everyone on GitHub, on the forum, at conferences. We found our way from aerospace to geospatial, and then we started getting involved in the built environment. So think AEC, architecture, engineering, construction, and think infrastructure, roads, bridges, rails, tunnels, electric grid. So our roadmap in some ways started to you started to see what Bentley was doing. You go, wow, we could do a lot together. And then in parallel to that journey, we've known Bentley systems for about 10 years. So CSIM was open source, and then the iTwin capture team, Bentley, that does the reality captures or taking 3D photos or photos and then generating 3D meshes, or now generating Gaussing splats. They found the CSEM project, and one of the code examples had Exton, Pennsylvania as the location on the globe. And Bentley is headquartered in Exton, Pennsylvania.

Max Limper: 4:33

Oh my god, I didn't even know that. Yeah, so you're really neighbors.

Patrick Cozzi: 4:37

And we were part of the aerospace company AGI, and we were literally 15 minutes from each other, but it took all the computer vision experts based out of Paris, France, to bring us together. Um, I tell that story because Cesium, we've known Bentley a long time. So not just having that product roadmap alignment, but the philosophical alignment. Bentley supported 3D tiles very early before anyone knew what it was. Uh Bentley funded a lot of the Cesium JS work, like going back to 2015 or so. So in us coming together, like we knew both the product and platform vision would be really good, but then the philosophy vision would be really good. And we've seen that again over the last year. We had this amazing users conference in Philadelphia. We had over 400 people come from around the world. Out of the 100 presenters, 87 of them were from the community. It was organized by a vendor, but it was much more like an industry neutral event, if you will. And then we're continuing to support the community with ecosystem grants, a certification program. We're even starting to do some venture investments through convertible notes. So yeah, it's been super cool to come together and it's almost an accelerator for CSM.

Max Limper: 5:46

That sounds amazing. And it really seems like openness is a key part of the whole like CSIM philosophy, but also fits very well with Bentley, as you said. So openness in the sense of also really early on doing everything open source, 3D tiles system, right? Like the 3D tiles format in a way. Is it okay to call it a format? I think it's a format, right? Or is it more than that?

Patrick Cozzi: 6:10

Yeah, I think it's a format and then it's an open standard through OGC, the open geospatial consortium.

Max Limper: 6:16

So it's it's the maybe the connection between open source and standards, right, that really helped to accelerate this community. That's what I understood. Especially in a geospatial world, like open standards matter a lot, right? There are own consortiums for that. Does that have to do with the fact that you're integrating a lot of data? Let's say I have a geospatial model, elevation model, and then I add a city with buildings, and then I add maybe some vehicles or persons and some point cloud scans, and that's all data from different sources, and you all integrate it into one framework. So you gotta be very, very flexible and open, I guess, about like how you ingest the data.

Patrick Cozzi: 6:55

Yeah, I think you're exactly right. And certainly the Cesium philosophy has always been that any data that we bring into Cesium, we want to do so through open standards, and then when there's use cases and perhaps an open standard doesn't exist, then we would work to create one to meet their needs, hence our work with GLTF, which originally came from we needed to draw satellites, ground vehicles, those types of 3D models. And the only option at the time was Colada, which I just didn't feel appropriate for the web. And then GLTF was just getting started at the time. It was called Web GLTF. That's how early it was. And then same for 3D tiles, right? Like we saw, okay, a train engine and then a massive point cloud engine, and then a 3D city engine. They're actually quite similar when you look at the fundamental concepts to do the hierarchical level detail, the streaming, and so on.

Max Limper: 7:50

And then this Red Bull use case wasn't quite aerospace, but it just worked because the technology was flexible enough and allowed for it. And then here I have heard from people who are interested in using 3D tiles for product visualization, right? Let's say you don't have a mountain or a city or anything like that. You have something like a sofa, and it still can make sense, right? Somehow to use 3D tiles. And that's really fascinating and amazing to see. And so congrats to that level of the doctrine. I think you're really like pushing the boundaries here.

Patrick Cozzi: 8:22

Yeah, thanks, Max. And I think a lot of this is we just try to be as open as possible about the software and then the standards. And then that allows these use cases that maybe I would have never thought of or was not originally envisioned that they can be solved very well. And then, yeah, 3D tiles is interesting because the vast majority of it is not geospatial specific, right? It's just fundamental concepts and organizing massive amounts of data such that you can efficiently access just a small subset based on what the viewer or user needs. So it's great to see it being used outside of Geo.

Max Limper: 8:57

And like as said early on, like we're here, we're talking all about digital twins for the real world, right? And we can bring all kinds of real world data together, as you said, with like 3D scanning, converting data to GLTF, like for vehicle models and that kind of stuff. Then we have terrains and elevation models where you have standards for those. And Bentley Systems has this platform that's called iTwin, and it's all about digital twins. So I don't want to talk too much about because I think you can pitch it much better. In short, what does iTwin X actually do? And how is it now connected with Cesium? Like, where do you see the synergies going forward, especially?

Patrick Cozzi: 9:36

Yeah, I appreciate you asking. So iTwin stands for infrastructure digital twin. It's something I've had my eye on for a long time, even before us joining Bentley. And our vision is really to bring together CSEM and iTwin. So we have an open platform for both the built and natural environment. So when you think about the built environment, roads, bridges, rails, tunnels, having a digital twin for them, it's it is the geometry, right, and the 3D viz that you and I would know, but there's so much more behind, right? Like the amount of metadata and the semantics of that metadata. And if you think of just a building, it has structures, it has the electrical, it has HVAC, and having all of that information, and then even the history of that, maybe past inspections, or if you're gonna have a building plan, in what order do you build everything? So the iTwin platform, it's developer APIs to do full, highly detailed infrastructure digital twins. Bentley, we build our products, our next generation of products are built on the iTwin platform. And then with CSEM joining Bentley, we're bringing the iTwin and CSIM platform together. What I like is again the technology, but also how we're doing it and embracing openness. So, for example, we've released a lot of tutorials on how to use these platforms side by side, and then we're asking the community, and we're doing it on social media, we're doing it on the community forum, we're asking for input on hey, how would you like to guide the roadmap? So in CSEM today, you can look at the built environment. But if you want to be able to do change detection among two different models, how important is that to you compared to other things we could do? So the community is going to have a huge influence on where we take this.

Max Limper: 11:27

Well, thanks. That sounds really exciting, actually. And I love this approach in general of building something in an open way and involving the community. And I think it's really exciting to see where this is all heading. It's clearly advancing the state of the art on a completely different note in a way. You are also in your free and private time investing time into advancing knowledge around 3D and doing this for people where it can really have an impact. So doing charity work related to 3D tooling and 3D technology. Maybe can you share a little bit more about that? Sure.

Patrick Cozzi: 12:02

Yeah. So I started a non-profit called Create Access with my partner Liz Daly about three years ago. So my father passed away of pancreatic cancer when I was in high school, and then my mother passed away coming up on four years ago, and I wanted to do something right to honor them. And I was thinking about what might be helpful right to the world. And I landed on this idea of a 3D creation or 3D technology is a great field. And that we're going to need more and more talented people in it. But there might be a little bit of an access gap, right? Uh, kids or young people, they might have like phones and consoles, but they might not actually have a computer at home. Or the high school probably doesn't have, or some of them, especially in our Philadelphia area, don't necessarily have computer science programs. So I thought, geez, maybe we could start a nonprofit to do exposure education to young people. It could be at the high school level, or even we do some middle school as well, just to show them the possible of becoming a 3D creator, which could ultimately lead to being a software developer, an artist, a game designer, breath, a breath of the fields, right, that you and I work in with the intent of let's make very simple projects to get them up and running really quickly. Let's have them resonate with the participants, and then let's make everything online for free. So there's all these nice videos and tutorials that are up online. So then I mentioned this to my partner, Liz, and she goes, Okay, that sounds like a good idea. Okay, will you quit your job to run it? And she said, Yes. So Liz is really Liz and the team are really doing the bulk of the work. I've supported it financially, I've provided ideas and some introductions, but they're doing great work. They do a lot with UEFN on Real Editor for Fortnite and photogrammetry. No code to start, but you just you take out your phone, you do a 3D scan, like polycam, for example, bring it into UEFN to make a 3D experience. And then they've also introduced a CAD and 3D printing program. The young people really like that because they can print something they can touch.

Max Limper: 14:07

That is so cool. And honestly, this is really like teaching a new language. It's not just something that people can use to make video games, they can do amazing stuff in the real world, like capturing things or printing things and modifying them in between. When it look at 3D printing, it's so cool to see what people are coming up with and stuff you wouldn't be able to manufacture otherwise if you're not an expert or a craftsman. And so it's a great form of empowerment and a shout out to Liz and to you for creating access and thanks for sharing that. So that means basically you're already like addressing here, even outside the work at Bentley, a bottleneck, which is like know-how and talent, right? So for the adoption of 3Ds and digital twin technology so that we can leverage this language, as I called it, or this way of interacting with the world that is so powerful. People need to be aware that it's there, right? And they need to know how to use it. That is for sure one barrier. Do you see other barriers? What are maybe the biggest barriers that come to your mind when it comes to the adoption of 3D technology and digital twins, and how could we overcome those?

Patrick Cozzi: 15:19

I'm happy to share. So earlier in my career, there was a barrier that I don't think is there anymore, right? Where there weren't enough GPUs out there, or maybe the drivers weren't reliable enough. But now when you look at WebGL and Web GPU, for example, there really is 3D on every platform, on every device. It's robust, it's performant. That the fundamentals are there for us to have computing experiences that are 3D centric. Certainly from the create access perspective, some of the barriers that we do see, as I mentioned, are the maybe not having the hardware at home for the young people or not having the coursework early enough in the schooling, and also just trying to build the confidence. Like they're growing up, they're playing Fortnite, they're playing Roblox, right? And knowing, hey, not only can you use those experiences, but you can build those experiences, and it's closer, it's within reach. So I think that's one area. The other area when I look at Cesium and Bentley is there are software developers that can build with these 3D APIs. It's getting easier, and we want to make it even easier when you look at the use of AI to get you started. So this I didn't know it would work out like this, but with CSIM.js as open source, and then with pretty good reference documentation, a bunch of code examples, and like thousands and thousands of forum posts, we've actually been training LLMs without knowing that we can prompt ChatGPT and it will get you started writing a Cesium application. So I think leaning into this, whether it's at least to get folks started and over time, even doing more sophistic sophisticated agentic workflows, I think it's a pretty cool future.

Max Limper: 17:07

That sounds all great. And again, like congrats to like all the open source work and happy to see how it's paying off now, even with that, right? Like you can ask Chat GPT, hey, can you help me get started with a simple CSIM app that does this and this? That's really amazing. And it was only possible due to this approach of the openness and all the community, right? So that's super cool. And yeah, I'm happy we're advancing the field and overcoming these barriers. As you said, like back in the days, not any device had a GPU. And now, if we just look what happened within the past 20 years, it's amazing how fast the field has been advancing, and AI will probably just accelerate that, right? Thank you so much for sharing all of that. It was great speaking to you. And yeah, looking forward to following your guys' activities at Bentley and CSEM and maybe seeing you at the next convention then.

Patrick Cozzi: 18:02

Looking forward to it. Thanks, Max, and thanks everybody.