Unity Android Video Player Freezes but Audio Continues

UI Rendering is the act of generating a frame from your app and displaying it on the screen. To ensure that a user's interaction with your app is smooth, your app should render frames in under 16ms to achieve 60 frames per second (why 60fps?). If your app suffers from slow UI rendering, then the system is forced to skip frames and the user will perceive stuttering in your app. We call this jank.

To help you improve app quality, Android automatically monitors your app for jank and displays the information in the Android vitals dashboard. For information on how the data is collected, see Play Console docs.

If your app is experiencing jank, this page provides guidance on diagnosing and fixing the problem.

Identifying jank

Pinpointing the code in your app that is causing jank can be difficult. This section describes three methods for identifying jank:

• Visual inspection
• Systrace
• Custom performance monitoring

Visual inspection lets you quickly run through all the use-cases in your app in a few minutes, but it doesn't provide as much detail as Systrace. Systrace provides more details, but if you ran Systrace for all the use-cases in your app, you'd be flooded with so much data that it'd be difficult to analyze. Both visual inspection and systrace detect jank on your local device. If your jank cannot be reproduced on local devices, you can build custom performance monitoring to measure specific parts of your app on devices running in the field.

With visual inspection

Visual inspection helps you identify the use-cases that are producing jank. To perform a visual inspection, open your app and manually go through the different parts of your app and look out for UI that is janky. Here are some tips when performing visual inspections:

• Run a release (or at least non-debuggable) version of your app. The ART runtime disables several important optimizations in order to support debugging features, so make sure you're looking at something similar to what a user will see.
• Enable Profile GPU Rendering. Profile GPU Rendering displays bars on the screen that give you a quick visual representation of how much time it takes to render the frames of a UI window relative to the 16-ms-per-frame benchmark. Each bar has colored components that map to a stage in the rendering pipeline, so you can see which portion is taking the longest. For example, if the frame spends a lot of time handling input, you should look at your app code that handles user input.
• There are certain components, such as RecyclerView, that are a common source of jank. If your app uses those components, it's a good idea to run through those parts of the app.
• Sometimes, jank can be reproduced only when the app is launched from a cold start.
• Try running your app on a slower device to exacerbate the problem.

Once you've found use-cases that produce jank, you might have a good idea of what is causing the jank in your app. But if you need more information, you can use Systrace to drill down further.

With Systrace

Although Systrace is a tool that shows what the entire device is doing, it can be useful for identifying jank in your app. Systrace has minimal system overhead, so you will experience realistic jankiness during instrumentation.

Record a trace with Systrace while performing the janky use-case on your device. See the Systrace Walkthrough for instructions on how to use Systrace. The systrace is broken up by processes and threads. Look for your app's process in Systrace, which should look like figure 1.

Figure 1: systrace

The systrace in figure 1 contains the following information for identifying jank:

1. Systrace shows when each frame is drawn and color codes each frame to highlight slow render times. This helps you find individual janky frames more accurately than visual inspection. For more information, see Inspect UI frames and alerts.
2. Systrace detects problems in your app and displays alerts both in individual frames and the alerts panel. Following directions in the alert is your best option.
3. Parts of the Android framework and libraries, such as RecyclerView, contain trace markers. So, the systrace timeline shows when those methods are executed on the UI thread and how long they take to execute.

After looking at the systrace output, there might be methods in your app that you suspect are causing jank. For an example, if the timeline shows that a slow frame is caused by RecyclerView taking a long time, you can add Trace markers to the relevant code and re-run systrace for more information. In the new systrace, the timeline will show when your app's methods are called and how long they took to execute.

If systrace doesn't show you details about why UI thread work is taking for a long time, then you'll need to use Android CPU Profiler to record either a sampled or instrumented method trace. Generally, method traces are not good for identifying jank because they produce false-positive janks due to heavy overhead and they can't see when threads are running vs blocked. But, method traces can help you identify the methods in your app are taking the most time. After identifying those methods, add Trace markers and re-run systrace to see whether those methods are causing jank.

For more information, see Understanding Systrace.

With custom performance monitoring

If you can't reproduce jank on a local device, you can build custom performance monitoring into your app to help identify the source of jank on devices in the field.

To do this, collect frame render times from specific parts of your app with FrameMetricsAggregator and record and analyze the data using Firebase Performance Monitoring.

To learn more, see Use Firebase Performance Monitoring with Android Vitals.

Fixing jank

To fix jank, inspect which frames aren't completing in 16.7ms, and look for what is going wrong. Is Record View#draw taking abnormally long in some frames, or perhaps Layout? See the Common sources of jank below for these problems, and others.

To avoid jank, long running tasks should be run asynchronously outside of the UI thread. Always be aware of what thread your code is running on and use caution when posting non-trivial tasks to the main thread.

If you have a complex and important primary UI for your app (maybe the central scrolling list), consider writing instrumentation tests that can automatically detect slow render times and run the tests frequently to prevent regressions. For more information, see the Automated Performance Testing Codelab.

Common sources of jank

The following sections explain common sources of jank in apps and best practices for addressing them.

Scrollable lists

ListView and especially RecyclerView are commonly used for complex scrolling lists that are most susceptible to jank. They both contain Systrace markers, so you can use Systrace to figure out whether they are contributing to jank in your app. Be sure to pass the command line argument -a <your-package-name> to get trace sections in RecyclerView (as well as any trace markers you added) to show up. If available, follow the guidance of the alerts generated in the systrace output. Inside Systrace, you can click on RecyclerView-traced sections to see an explanation of the work RecyclerView is doing.

RecyclerView: notifyDataSetChanged

If you see every item in your RecyclerView being rebound (and thus re-laid out and re-drawn) in one frame, make sure that you're not calling notifyDataSetChanged(), setAdapter(Adapter), or swapAdapter(Adapter, boolean) for small updates. Those methods signal that the entire list content has changed, and will show up in Systrace as RV FullInvalidate. Instead, use SortedList or DiffUtil to generate minimal updates when content changes or is added.

For example, consider an app that receives a new version of a list of news content from the server. When you post that info to the Adapter, it's possible to call notifyDataSetChanged() as shown below:

Kotlin

fun onNewDataArrived(news: List<News>) {     myAdapter.news = news     myAdapter.notifyDataSetChanged() }            

Java

void onNewDataArrived(List<News> news) {     myAdapter.setNews(news);     myAdapter.notifyDataSetChanged(); }            

But this comes with a big downside – if it's a trivial change (maybe a single item added to the top), the RecyclerView isn't aware – it is told to drop all its cached item state, and thus needs to rebind everything.

It's much preferable to use DiffUtil, which will calculate and dispatch minimal updates for you.

Kotlin

fun onNewDataArrived(news: List<News>) {     val oldNews = myAdapter.items     val result = DiffUtil.calculateDiff(MyCallback(oldNews, news))     myAdapter.news = news     result.dispatchUpdatesTo(myAdapter) }            

Java

void onNewDataArrived(List<News> news) {     List<News> oldNews = myAdapter.getItems();     DiffResult result = DiffUtil.calculateDiff(new MyCallback(oldNews, news));     myAdapter.setNews(news);     result.dispatchUpdatesTo(myAdapter); }            

Just define your MyCallback as a DiffUtil.Callback implementation to inform DiffUtil how to inspect your lists.

RecyclerView: Nested RecyclerViews

It's common to nest RecyclerViews, especially with a vertical list of horizontally scrolling lists (like grids of apps on Play Store main page). This can work great, but it's also a lot of views moving around. If you see a lot of inner items inflating when you first scroll down the page, you may want to check that you're sharing RecyclerView.RecycledViewPools between inner (horizontal) RecyclerViews. By default, each RecyclerView will have its own pool of items. In the case with a dozen itemViews on screen at once though, it's problematic when itemViews can't be shared by the different horizontal lists, if all the rows are showing similar types of views.

Kotlin

class OuterAdapter : RecyclerView.Adapter<OuterAdapter.ViewHolder>() {      ...      override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder {         // inflate inner item, find innerRecyclerView by ID…         val innerLLM = LinearLayoutManager(parent.context, LinearLayoutManager.HORIZONTAL, false)         innerRv.apply {             layoutManager = innerLLM              recycledViewPool = sharedPool              }         return OuterAdapter.ViewHolder(innerRv)     }     ...            

Java

class OuterAdapter extends RecyclerView.Adapter<OuterAdapter.ViewHolder> {     RecyclerView.RecycledViewPool sharedPool = new RecyclerView.RecycledViewPool();      ...      @Override     public void onCreateViewHolder(ViewGroup parent, int viewType) {         // inflate inner item, find innerRecyclerView by ID…         LinearLayoutManager innerLLM = new LinearLayoutManager(parent.getContext(),                 LinearLayoutManager.HORIZONTAL);         innerRv.setLayoutManager(innerLLM);              innerRv.setRecycledViewPool(sharedPool);              return new OuterAdapter.ViewHolder(innerRv);      }     ...            

If you want to optimize further, you can also call setInitialPrefetchItemCount(int) on the inner RecyclerView's LinearLayoutManager. If for example you'll always have 3.5 items visible in a row, call innerLLM.setInitialItemPrefetchCount(4);. This will signal to RecyclerView that when a horizontal row is about to come onscreen, it should attempt to prefetch the items inside, if there's spare time on the UI thread.

RecyclerView: Too much inflation / Create taking too long

The prefetch feature in RecyclerView should help work around the cost of inflation in most cases by doing the work ahead of time, while the UI thread is idle. If you're seeing inflation during a frame (and not in a section labelled RV Prefetch), be sure you're testing on a recent device (Prefetch currently only supported on Android 5.0 API Level 21 and higher) and using a recent version of the Support Library.

If you frequently see inflation causing jank as new items come on screen, verify that you don't have more view types than you need. The fewer the view types in a RecyclerView's content, the less inflation will need to be done when new item types come on screen. If possible, merge view types where reasonable – if only an icon, color, or piece of text changes between types, you can make that change at bind time, and avoid inflation (reducing your app's memory footprint at the same time).

If your view types look good, look at reducing the cost of your inflation. Reducing unnecessary container and structural Views can help – consider building itemViews with ConstraintLayout, which can make it easy to reduce structural Views. If you want to really optimize for performance, your items hierarchies are simple, and you don't need complex theming and style features, consider calling the constructors yourself – note though, that it's often not worth the tradeoff of losing the simplicity and features of XML.

RecyclerView: Bind taking too long

Bind (that is, onBindViewHolder(VH, int)) should be very simple, and take much less than one millisecond for all but the most complex items. It simply should take POJO items from your adapter's internal item data, and call setters on views in the ViewHolder. If RV OnBindView is taking a long time, verify that you're doing minimal work in your bind code.

If you're using simple POJO objects to hold data in your adapter, you can completely avoid writing the binding code in onBindViewHolder by using the Data Binding library.

RecyclerView or ListView: layout / draw taking too long

For issues with draw and layout, see the sections on Layout and Rendering Performance.

ListView: Inflation

It's easy to accidentally disable recycling in ListView if you aren't careful. If you see inflation every time an item comes on screen, check that your implementation of Adapter.getView() is using, re-binding, and returning the convertView parameter. If your getView() implementation always inflates, your app won't get the benefits of recycling in ListView. The structure of your getView() should almost always be similar to the implementation below:

Kotlin

fun getView(position: Int, convertView: View?, parent: ViewGroup): View {     return (convertView ?: layoutInflater.inflate(R.layout.my_layout, parent, false)).apply {         // … bind content from position to convertView …     } }            

Java

View getView(int position, View convertView, ViewGroup parent) {      if (convertView == null) {         // only inflate if no convertView passed         convertView = layoutInflater.inflate(R.layout.my_layout, parent, false)     }     // … bind content from position to convertView …     return convertView; }            

Layout performance

If Systrace shows that the Layout segment of Choreographer#doFrame is doing too much work, or doing work too often, that means you're hitting layout performance issues. The layout performance of your app depends on what portion of the View hierarchy has changing layout parameters or inputs.

Layout performance: Cost

If the segments are longer than a few milliseconds, it's possible that you're hitting worst-case nesting performance for RelativeLayouts, or weighted-LinearLayouts. Each of these layouts can trigger multiple measure/layout passes of its children, so nesting them can lead to O(n^2) behavior on the depth of nesting. Try avoiding RelativeLayout or the weight feature of LinearLayout in all but the lowest leaf nodes of the hierarchy. There are a few ways to do this:

• You can reorganize your structural views.
• You can define custom layout logic. See the optimize your layout guide for a specific example.
• You can try converting to ConstraintLayout, which provides similar features, without the performance drawbacks.

Layout performance: Frequency

Layout is expected to happen when new content comes on screen, for example when a new item scrolls into view in RecyclerView. If significant layout is happening on each frame, it's possible that you're animating layout, which is likely to cause dropped frames. Generally, animations should be run on drawing properties of View (e.g. setTranslationX/Y/Z(), setRotation(), setAlpha(), etc…). These can all be changed much more cheaply than layout properties (such as padding, or margins). It's also much cheaper to change drawing properties of a view, generally by calling a setter which triggers a invalidate(), followed by a draw(Canvas) in the next frame. This will re-record drawing operations for the view that is invalidated, and is also generally much cheaper than layout.

Rendering performance

Android UI does work in two phases – Record View#draw, on the UI thread, and DrawFrame on the RenderThread. The first runs draw(Canvas) on every invalidated View, and may invoke calls into custom views or into your code. The second runs on the native RenderThread, but will operate based on work generated by the Record View#draw phase.

Rendering performance: UI Thread

If Record View#draw is taking a long time, it's often the case that a bitmap is being painted on the UI thread. Painting to a bitmap uses CPU rendering, so you should generally avoid this when possible. You can use method tracing with the Android CPU Profiler to see if this is the problem.

Painting to a bitmap is often done when an app wants to decorate a bitmap before displaying it. Sometimes a decoration like adding rounded corners:

Kotlin

val paint = Paint().apply {     isAntiAlias = true } Canvas(roundedOutputBitmap).apply {     // draw a round rect to define shape:     drawRoundRect(             0f,             0f,             roundedOutputBitmap.width.toFloat(),             roundedOutputBitmap.height.toFloat(),             20f,             20f,             paint     )     paint.xfermode = PorterDuffXfermode(PorterDuff.Mode.MULTIPLY)     // multiply content on top, to make it rounded     drawBitmap(sourceBitmap, 0f, 0f, paint)     setBitmap(null)     // now roundedOutputBitmap has sourceBitmap inside, but as a circle }            

Java

Canvas bitmapCanvas = new Canvas(roundedOutputBitmap); Paint paint = new Paint(); paint.setAntiAlias(true); // draw a round rect to define shape: bitmapCanvas.drawRoundRect(0, 0,         roundedOutputBitmap.getWidth(), roundedOutputBitmap.getHeight(), 20, 20, paint); paint.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.MULTIPLY)); // multiply content on top, to make it rounded bitmapCanvas.drawBitmap(sourceBitmap, 0, 0, paint); bitmapCanvas.setBitmap(null); // now roundedOutputBitmap has sourceBitmap inside, but as a circle            

If this is the sort of work you're doing on the UI thread, you can instead do this on the decoding thread in the background. In some cases like this one, you can even do the work at draw time, so if your Drawable or View code looks something like this:

Kotlin

fun setBitmap(bitmap: Bitmap) {     mBitmap = bitmap     invalidate() }  override fun onDraw(canvas: Canvas) {     canvas.drawBitmap(mBitmap, null, paint) }            

Java

void setBitmap(Bitmap bitmap) {     mBitmap = bitmap;     invalidate(); }  void onDraw(Canvas canvas) {     canvas.drawBitmap(mBitmap, null, paint); }            

You can replace it with this:

Kotlin

fun setBitmap(bitmap: Bitmap) {     shaderPaint.shader = BitmapShader(bitmap, Shader.TileMode.CLAMP, Shader.TileMode.CLAMP)     invalidate() }  override fun onDraw(canvas: Canvas) {     canvas.drawRoundRect(0f, 0f, width, height, 20f, 20f, shaderPaint) }            

Java

void setBitmap(Bitmap bitmap) {     shaderPaint.setShader(             new BitmapShader(bitmap, TileMode.CLAMP, TileMode.CLAMP));     invalidate(); }  void onDraw(Canvas canvas) {     canvas.drawRoundRect(0, 0, width, height, 20, 20, shaderPaint); }            

Note that this can also often be done for background protection (drawing a gradient on top of the Bitmap), and image filtering (with ColorMatrixColorFilter), two other common operations done modifying bitmaps.

If you're drawing to a bitmap for another reason, possibly using it as a cache, try and draw to the hardware accelerated Canvas passed to your View or Drawable directly, and if necessary, consider calling setLayerType() with LAYER_TYPE_HARDWARE to cache complex rendering output, and still take advantage of GPU rendering.

Rendering performance: RenderThread

Some canvas operations are cheap to record, but trigger expensive computation on the RenderThread. Systrace will generally call these out with alerts.

Canvas.saveLayer()

Avoid Canvas.saveLayer() – it can trigger expensive, uncached, off-screen rendering each frame. Though performance was improved in Android 6.0 (when optimizations were made to avoid render target switching on the GPU), it's still good to avoid this expensive API if possible, or at minimum, ensure you're passing the Canvas.CLIP_TO_LAYER_SAVE_FLAG (or calling a variant that doesn't take flags).

Animating large Paths

When Canvas.drawPath() is called on the hardware accelerated Canvas passed to Views, Android draws these paths first on CPU, and uploads them to the GPU. If you have large paths, avoid editing them from frame to frame, so they can be cached and drawn efficiently. drawPoints(), drawLines(), and drawRect/Circle/Oval/RoundRect() are more efficient – it's better to use them even if you end up using more draw calls.

Canvas.clipPath

clipPath(Path) triggers expensive clipping behavior, and should generally be avoided. When possible, opt for drawing shapes, instead of clipping to non-rectangles. It performs better and supports anti-aliasing. For example, the following clipPath call:

Kotlin

canvas.apply {     save()     clipPath(circlePath)     drawBitmap(bitmap, 0f, 0f, paint)     restore() }            

Java

canvas.save(); canvas.clipPath(circlePath); canvas.drawBitmap(bitmap, 0f, 0f, paint); canvas.restore();            

Can be instead expressed as:

Kotlin

paint.shader = BitmapShader(bitmap, Shader.TileMode.CLAMP, Shader.TileMode.CLAMP) // at draw time: canvas.drawPath(circlePath, mPaint)            

Java

// one time init: paint.setShader(new BitmapShader(bitmap, TileMode.CLAMP, TileMode.CLAMP)); // at draw time: canvas.drawPath(circlePath, mPaint);            

Bitmap uploads

Android displays bitmaps as OpenGL textures, and the first time a bitmap is displayed in a frame, it's uploaded to the GPU. You can see this in Systrace as Texture upload(id) width x height. This can take several milliseconds (see Figure 2), but it's necessary to display the image with the GPU.

If these are taking a long time, first check the width and height numbers in the trace. Ensure that the bitmap being displayed isn't significantly bigger than the area on screen it's showing in. If it is, that wastes upload time, and memory. Generally Bitmap loading libraries provide easy means of requesting an appropriately sized Bitmap.

In Android 7.0, bitmap loading code (generally done by libraries) can call prepareToDraw() to trigger an upload early, before it's needed. This way the upload happens early, while the RenderThread is idle. This can be done after decoding, or when binding a bitmap to a View, as long as you know the bitmap. Ideally, your bitmap loading library will do this for you, but if you're managing your own, or want to ensure you don't hit uploads on newer devices, you can call prepareToDraw() in your own code.

Figure 2: An app spends significant time in a frame uploading a large bitmap. Either reduce its size, or trigger it early when decoded with prepareToDraw().

Thread scheduling delays

The thread scheduler is the part of the Android operating system in charge of deciding which threads in the system should run, when, and for how long. Sometimes, jank occurs because your app's UI Thread is blocked or not running. Systrace uses different colors (see figure 3) to indicate when a thread is Sleeping (gray), Runnable (blue: it could run, but the scheduler hasn't picked it to run yet), Actively running (Green), or in Uninterruptible sleep (Red or Orange). This is extremely useful for debugging jank issues that are caused by thread scheduling delays.

Figure 3: highlights a period when the UI Thread is sleeping.

Often long pauses in your app's execution are caused by binder calls, the inter-process communication (IPC) mechanism on Android. On recent versions of Android, it's one of the most common reasons for the UI Thread to stop running. Generally, the fix is to avoid calling functions that make binder calls; if it's unavoidable, you should cache the value, or move work to background threads. As codebases get larger, it's easy to accidentally add a binder call by invoking some low level method if you aren't careful – but it's just as easy to find and fix them with tracing.

If you have binder transactions, you can capture their call stacks with the following adb commands:

          $ adb shell am trace-ipc start … use the app - scroll/animate ... $ adb shell am trace-ipc stop --dump-file /data/local/tmp/ipc-trace.txt $ adb pull /data/local/tmp/ipc-trace.txt                  

Sometimes innocuous seeming calls like getRefreshRate() can trigger binder transactions, and cause big problems when they're called frequently. Tracing periodically can help you quickly find and fix these issues as they show up.

Figure 4: shows the UI Thread sleeping due to binder transactions in a RV fling. Keep your bind logic simple, and use trace-ipc to track down and remove binder calls.

If you aren't seeing binder activity, but still aren't seeing your UI Thread run, be sure that you're not waiting on some lock or other operation from another thread. Typically, the UI thread shouldn't have to wait on results from other threads – other threads should post information to it.

Object allocation and garbage collection

Object allocation and garbage collection (GC) have become significantly less of an issue since ART was introduced as the default runtime in Android 5.0, but it's still possible to weigh down your threads with this extra work. It's fine to allocate in response to a rare event that doesn't happen many times a second (like a user clicking a button), but remember that each allocation comes with a cost. If it's in a tight loop that's called frequently, consider avoiding the allocation to lighten the load on the GC.

Systrace will show you if GC is running frequently, and the Android Memory Profiler can show you where allocations are coming from. If you avoid allocations when you can, especially in tight loops, you shouldn't have a problem.

Figure 5: shows a 94ms GC on the HeapTaskDaemon thread

On recent versions of Android, GC generally runs on a background thread named HeapTaskDaemon. Note that significant amounts of allocation can mean more CPU resources spent on GC as shown in figure 5.

fisherexcurce.blogspot.com

Source: https://developer.android.com/topic/performance/vitals/render

Share this post