Flutter mixing project to open up the road to texture

Original link: https://xuyisheng.top/flutter_image_texture/

Flutter mixing project to get through the texture road

Flutter’s picture system is based on a set of architecture of Image, but the performance of this thing is really unflattering. It feels like it is still at the level of Native development at least 5 years ago. Although it is very simple to use, an Image.network takes the world, but no matter what. Whether it is decoding performance or loading speed, or memory usage and caching logic, it is far inferior to the native image library, especially Glide. Although Google has been planning to optimize the performance of Flutter Image, at this stage, the best way to experience image loading is to use Glide for loading through plug-ins.

Therefore, in the mixed environment, hosting Flutter’s image loading function to native is the most reasonable and best-performing solution.

Then for the bridging to native solution, there are two main directions, one is to pass the binary data stream of the loaded image through Channel, and then parse the binary stream in Flutter and then parse the image, and the other is to use external textures. , to share the image memory. Obviously, the second solution is a better solution. In terms of memory consumption and transmission performance, the external texture solution is the best choice for Flutter to bridge the Native image architecture.

Although the external texture scheme is better, there are not many researches on this scheme on the Internet. The more typical one is the video rendering scheme in Flutter’s official Plugins. The address is as follows.

https://github.com/flutter/plugins/tree/main/packages/video_player

This is our first-hand solution for researching external textures. In addition, Xianyu’s open source PowerImage is also implemented based on external texture solutions. At the same time, they also give a series of pre-research and Technical basic research, these are also the best ways for us to understand external textures. However, based on Ali’s consistent style, we do not dare to directly use PowerImage on a large scale to study external textures to realize a set of our own solutions. In fact, it is the most OK

https://www.infoq.cn/article/MLMK2bx8uaNb5xJm13SW

https://juejin.cn/post/6844903662548942855

Basic Concepts of External Textures

In fact, the above two Xianyu articles have explained the concept of external textures more clearly, so let’s briefly summarize them.

First of all, Flutter’s rendering mechanism is completely isolated from Native rendering. The advantage of this is that Flutter can fully control the drawing and rendering of Flutter pages, but the disadvantage is that when Flutter obtains some Native high-memory data, passing through Channel will lead to Waste and performance pressure, so Flutter provides external textures to handle this kind of scene.

In Flutter, the system provides a special Widget – Texture Widget. Texture is a special Layer in Flutter’s Widget Tree. It does not participate in the drawing of other Layers. Its data is all provided by Native. Native will write dynamic rendering data, such as pictures, videos and other data to PixelBuffer, while Flutter The Engine will get the corresponding rendering data from the GPU and render it into the corresponding Texture.

Texture in action

The process of loading images with the Texture solution is actually relatively long, involving the double-end cooperation between Flutter and Native. Therefore, we need to create a Flutter Plugin to complete the call of this function.

We create a Flutter Plugin, and Android Studio will automatically generate the corresponding plugin code and Example code for us.

Overall process

Between Flutter and Native, memory data is shared by external textures. The interrelated link between them is a TextureID. Through this ID, we can respectively associate with the memory data on the Native side, or associate with the Flutter side. Texture Widget, so all stories start from TextureID.

Flutter starts from the Texture Widget, when the Widget is initialized, it will request Native through the Channel, create a new TextureID, return the TextureID to Flutter, and bind the current Texture Widget to this ID.

Next, the Flutter side requests the Native through the Channel of the image Url to be loaded, the Native side finds the corresponding Texture through the TextureID, and uses the passed Url to load the image on the Native side through Glide, and writes the image resource to the Texture. At this time, Flutter The Texture Widget on the side can obtain the rendering information in real time.

Finally, when the Texture Widget on the Flutter side is recycled, the current Texture needs to be recycled to release this part of the memory.

The above is the implementation process of the entire external texture scheme.

Flutter side

First, we need to create a Channel to register several method calls mentioned above.

 class MethodChannelTextureImage extends TextureImagePlatform { @visibleForTesting final methodChannel = const MethodChannel('texture_image'); @override Future<int?> initTextureID() async { final result = await methodChannel.invokeMethod('initTextureID'); return result['textureID']; } @override Future<Size> loadByTextureID(String url, int textureID) async { var params = {}; params["textureID"] = textureID; params["url"] = url; final size = await methodChannel.invokeMethod('load', params); return Size(size['width']?.toDouble() ?? 0, size['height']?.toDouble() ?? 0); } @override Future<int?> disposeTextureID(int textureID) async { var params = {}; params["textureID"] = textureID; final result = await methodChannel.invokeMethod('disposeTextureID', params); return result['textureID']; } }

Next, go back to the Flutter Widget and encapsulate a Widget to manage Texture.

In this packaged Widget, you can adjust the size or manage the life cycle, but there is only one core, which is to create a Texture.

 Texture(textureId: _textureID),

Use the Channel created earlier to complete the loading of the process.

 @override void initState() { initTextureID().then((value) { _textureID = value; _textureImagePlugin.loadByTextureID(widget.url, _textureID).then((value) { if (mounted) { setState(() => bitmapSize = value); } }); }); super.initState(); } Future<int> initTextureID() async { int textureID; try { textureID = await _textureImagePlugin.initTextureID() ?? -1; } on PlatformException { textureID = -1; } return textureID; } @override void dispose() { if (_textureID != -1) { _textureImagePlugin.disposeTextureID(_textureID); } super.dispose(); }

In this way, the whole process on the Flutter side is completed – Create TextureID -> Bind TextureID and Url -> Recycle TextureID.

Native side

The processing on the Native side is concentrated in the registration class of the Plugin. When registering, we need to create a TextureRegistry, which is the entry provided by the system to use external textures.

 override fun onAttachedToEngine(@NonNull flutterPluginBinding: FlutterPlugin.FlutterPluginBinding) { channel = MethodChannel(flutterPluginBinding.binaryMessenger, "texture_image") channel.setMethodCallHandler(this) context = flutterPluginBinding.applicationContext textureRegistry = flutterPluginBinding.textureRegistry }

Next, we need to process the Channel and implement the three methods mentioned above.

 "initTextureID" -> { val surfaceTextureEntry = textureRegistry?.createSurfaceTexture() val textureId = surfaceTextureEntry?.id() ?: -1 val reply: MutableMap<String, Long> = HashMap() reply["textureID"] = textureId textureSurfaces[textureId] = surfaceTextureEntry result.success(reply) }

The core function of the initTextureID method is to create a surfaceTextureEntry from TextureRegistry, and textureId is its id attribute.

 "load" -> { val textureId: Int = call.argument("textureID") ?: -1 val url: String = call.argument("url") ?: "" if (textureId >= 0 && url.isNotBlank()) { Glide.with(context).load(url).skipMemoryCache(true).into(object : CustomTarget<Drawable>() { override fun onResourceReady(resource: Drawable, transition: Transition<in Drawable>?) { if (resource is BitmapDrawable) { val bitmap = resource.bitmap val imageWidth: Int = bitmap.width val imageHeight: Int = bitmap.height val surfaceTextureEntry: SurfaceTextureEntry = textureSurfaces[textureId.toLong()]!! surfaceTextureEntry.surfaceTexture().setDefaultBufferSize(imageWidth, imageHeight) val surface = if (surfaceMap.containsKey(textureId.toLong())) { surfaceMap[textureId.toLong()] } else { val surface = Surface(surfaceTextureEntry.surfaceTexture()) surfaceMap[textureId.toLong()] = surface surface } val canvas: Canvas = surface!!.lockCanvas(null) canvas.drawBitmap(bitmap, 0F, 0F, null) surface.unlockCanvasAndPost(canvas) val reply: MutableMap<String, Int> = HashMap() reply["width"] = bitmap.width reply["height"] = bitmap.height result.success(reply) } } override fun onLoadCleared(placeholder: Drawable?) { } }) } }

The load method is the Glide that we are familiar with. Use Glide to obtain the image data corresponding to Url, and then use SurfaceTextureEntry to create a Surface object, and write the data returned by Glide into Surface. Finally, return the width and height of the image to the surface. Pass it to Flutter for some subsequent processing.

 "disposeTextureID" -> { val textureId: Int = call.argument("textureID") ?: -1 val textureIdLong = textureId.toLong() if (surfaceMap.containsKey(textureIdLong) && textureSurfaces.containsKey(textureIdLong)) { val surfaceTextureEntry: SurfaceTextureEntry? = textureSurfaces[textureIdLong] val surface = surfaceMap[textureIdLong] surfaceTextureEntry?.release() surface?.release() textureSurfaces.remove(textureIdLong) surfaceMap.remove(textureIdLong) } }

The disposeTextureID method is to recycle the texture of dispose. Otherwise, the Texture has been applying for new memory, which will cause the native memory to rise and not be recycled. Therefore, after calling dispose on the Flutter side, we need to correspond to the corresponding TextureID. resources are recycled.

Above, we have completed the native processing. By cooperating with the Flutter side and with the efficient loading capability of Glide, we have completed a perfect image loading process.

Summarize

By loading images with external textures, we can have the following advantages.

  • Reuse Native’s efficient and stable image loading mechanism, including caching, encoding and decoding, performance, etc.
  • Reduce the memory consumption of multiple solutions and reduce the running memory of the App
  • Open up Native and Flutter, image resources can be shared in memory

However, the current solution is not “perfect”. It can only be said that the above solution is a “available” solution, but it is far from being “easy to use”. In order to better realize the solution of external textures , we still need to deal with some details.

  • Multiplexing, multiplexing, or TMD multiplexing, for pictures with the same Url and loaded pictures, a set of caching mechanisms should be implemented on both the Native and Flutter sides.
  • For the support of Gif and Webp, so far, we are dealing with static pictures, and we have not added dynamic content processing. Of course, this must be possible, but we have not yet supported it.
  • The Batch call of Channel, for a list, may generate a large number of picture requests in one frame at the same time, although the performance of Channel has been greatly improved, but if you can make a buffer for the call of Channel, then for special For frequent calls, the performance of some Channels will be optimized

So this is just the first article, we will continue to optimize the above problems later, please wait and see.

This article is reprinted from: https://xuyisheng.top/flutter_image_texture/
This site is for inclusion only, and the copyright belongs to the original author.

Leave a Comment

Your email address will not be published.