Im using the MLKit combined with the CameraX Image-Analysis Use Case to build an App, that lets you control a cursor, which gets drawn onto an overlay based on the position of the face (head based pointing essentially).

I have two activities:

Activity 1 previews the camera feed and draws the facial landmarks (in my case, just the nosetip) onto the overlay as well as the cursor. As I said before, the cursor position is being calculated based on the nosetip position in the image -> No problems here

Activity 2 runs a very simplistic game in a separate thread, which spawns obstacles that need to be avoided with the cursor. No live camera feed, "just" the face detection, the face -> cursor screen mapping, a custom gameView and the overlay for the cursor ontop.

My problem: After some time, the face detector just stops detecting faces (thus the cursor gets "stuck") without any errors and oddly enough, the game incrementally speeds up (but still registers collision with the cursor).

I already followed the instructions for optimizing face detection in real-time by only enabling landmark-mode, enabled tracking and setting the performance mode to fast.

My guess was that running the face detection, mapping the cursor to a point on the screen, drawing onto the overlay for the cursor and having a game running at the same time might be just too much work, but I read somewhere that MLKit runs its detection on its own thread.

It must have something to do with my game, because as already mentioned above, in Activity 1 (Live camera preview + calculating cursor position + drawing landmarks & cursor on the overlay canvas) I never faced (no pun intended) a crash.

My custom GameView:

public class GameView extends SurfaceView implements SurfaceHolder.Callback{

private final int difficultyTime = 5;
private GameThread thread;
private long startTime;
private long timer;
private long difficultyTimer;
private long lastSpawned = 0;
private Paint textPaint;
private Random random;
private Pointer pointer;
private GameManager game;

public GameView(Context context) {
    super(context);
    init();
}

public GameView(Context context, AttributeSet attrs) {
    super(context, attrs);
    init();
}

public GameView(Context context, AttributeSet attrs, int defStyle) {
    super(context, attrs, defStyle);
    init();
}

public void init() {
    getHolder().addCallback(this);
    thread = new GameThread(getHolder(), this);
    game = new GameManager();
    setFocusable(true);

    textPaint = new Paint();
    textPaint.setColor(Color.RED);
    textPaint.setTextSize(70);
    Paint collisionPaint = new Paint();
    collisionPaint.setColor(Color.BLUE);
    Paint pointerPaint = new Paint();
    pointerPaint.setColor(Color.GREEN);

    startTime = System.currentTimeMillis();
    lastSpawned = System.currentTimeMillis();
    difficultyTimer = System.currentTimeMillis() + difficultyTime * 1000;
    random = new Random();
}

public void update() {
    timer = (System.currentTimeMillis() - startTime) / 1000;
    long currentTimeMS = System.currentTimeMillis();
    
    if(currentTimeMS > lastSpawned + (game.getCooldown() * 1000L)) {
        lastSpawned = System.currentTimeMillis();
        game.setMayDraw(true);
    }
    
    if(checkCollision()) {
        gameover();
    }
    
    // Raise difficulty
    if(difficultyTimer <= System.currentTimeMillis()) {
        difficultyTimer = System.currentTimeMillis() + difficultyTime * 1000;
        game.advanceDifficulty();
    }
}

@Override
public void draw(Canvas canvas) {
    super.draw(canvas);
    int bgColor = Color.WHITE;
    canvas.drawColor(bgColor);

    // Spawn elements
    if(game.getMayDraw()) {
        game.setMayDraw(false);
        spawnObstacles(canvas);
    }
    // Update existing elements
    game.updateObstacles(canvas);
    
    // Draw the timer
    drawTimer(canvas);

    // Draw difficulty text
    drawDifficulty(canvas);
}

private void drawTimer(Canvas canvas) {
    canvas.drawText(Long.toString(timer), (getWidth() / 2), 80, textPaint);
    invalidate();
}

private void drawDifficulty(Canvas canvas) {
    canvas.drawText("Difficulty: " + game.getCurrentDifficultyIndex(), (getWidth() / 2) + 250, 80, textPaint);
}
private void spawnObstacles(Canvas canvas) {
    int random_number;
    int last_random = 0;

    for(int i = 0; i < game.getMAX_SPAWN(); i++) {

        do {
            random_number = random.nextInt(4) + 1;
        }while(random_number == last_random);


        switch(random_number) {
            case 1:
                game.spawnTop(canvas);
                break;
            case 2:
                game.spawnBottom(canvas);
                break;
            case 3:
                game.spawnLeft(canvas);
                break;
            case 4:
                game.spawnRight(canvas);
                break;
        }
        last_random = random_number;
    }
}

public void gameover() {
    startTime = System.currentTimeMillis();
    game.getObstacles().clear();
    game.resetDifficulty();
    difficultyTimer = System.currentTimeMillis() + difficultyTime * 1000;
}

public boolean checkCollision() {
    List<Pair<GameManager.Direction, Rect>> collidables = game.getObstacles();

    for (Pair<GameManager.Direction, Rect> r : collidables) {
        if(intersects(r.second, pointer.getCollisionRect()) || intersects(pointer.getCollisionRect(), r.second)) {
            return true;
        }
    }
    return false;
}

public static boolean intersects(Rect rect, Rect otherRect) {
    return rect.left <= otherRect.right && otherRect.left <= rect.right
            && rect.top <= otherRect.bottom && otherRect.top <= rect.bottom;
}

@Override
public void surfaceCreated(@NonNull SurfaceHolder surfaceHolder) {
    thread.setRunning(true);
    thread.start();
}

@Override
public void surfaceChanged(@NonNull SurfaceHolder surfaceHolder, int format, int width, int height) {
    System.out.println("Surface changed - Format: " + format + " W/H: ("+width +"/" +height+")");

}

@Override
public void surfaceDestroyed(@NonNull SurfaceHolder surfaceHolder) {
    boolean retry = true;
    while (retry) {
        try {
            thread.setRunning(false);
            thread.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        retry = false;
    }
}

@Override
protected void onSizeChanged(int w, int h, int oldw, int oldh) {
    super.onSizeChanged(w, h, oldw, oldh);
}

GameThread:

public class GameThread extends Thread{

private final SurfaceHolder surfaceHolder;
private final GameView gameView;
private boolean running;
public static Canvas canvas;

public GameThread(SurfaceHolder surfaceHolder, GameView gameView) {
    super();
    this.surfaceHolder = surfaceHolder;
    this.gameView = gameView;
}

@Override
public void run() {
    long startTime;
    long timeMillis;
    long waitTime;
    long totalTime = 0;
    int frameCount = 0;
    int targetFPS = 15;
    long targetTime = 1000 / targetFPS;

    while (running) {
        startTime = System.nanoTime();
        canvas = null;

        try {
            canvas = this.surfaceHolder.lockCanvas();
            synchronized(surfaceHolder) {
                this.gameView.update();
                this.gameView.draw(canvas);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        finally {
            if (canvas != null)            {
                try {
                    surfaceHolder.unlockCanvasAndPost(canvas);
                }
                catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }

        timeMillis = (System.nanoTime() - startTime) / 1000000;
        waitTime = targetTime - timeMillis;

        try {
            if(waitTime > 0) {
                this.sleep(waitTime);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

        totalTime += System.nanoTime() - startTime;
        frameCount++;
        if (frameCount == targetFPS)        {
            double averageFPS = 1000 / ((totalTime / frameCount) / 1000000);
            frameCount = 0;
            totalTime = 0;
            Log.d("GameFPS", "" + averageFPS);
        }
    }

}
public void setRunning(boolean isRunning) {
    running = isRunning;
}

FaceDetector:

public class Detector {


private final FaceDetector faceDetector;
private final GraphicOverlay overlay;
private final Pointer pointer;
private final ClickHandler clickHandler;
private final boolean drawFaces;

public Detector(GraphicOverlay overlay, Context context, boolean drawFaces) {

    this.overlay = overlay;
    this.drawFaces = drawFaces;

    pointer = new Pointer(overlay, context);
    clickHandler = new ClickHandler(context, this.pointer);

    FaceDetectorOptions options =
            new FaceDetectorOptions.Builder()
                    //.setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
                    .setLandmarkMode(FaceDetectorOptions.LANDMARK_MODE_ALL)
                    /* disable for better performance *///.setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL)
                    .enableTracking()
                    .setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
                    .build();

    faceDetector = FaceDetection.getClient(options);
}


public void detectFace(ImageProxy image) {
    if(image == null) {
        return;
    }
    InputImage inputImage = getInputImage(image);

    int rotationDegrees = image.getImageInfo().getRotationDegrees();
    
    if(rotationDegrees == 0 || rotationDegrees == 180) {
        overlay.setImageSourceInfo(image.getWidth(), image.getHeight(), true);
    }else{
        overlay.setImageSourceInfo(image.getHeight(), image.getWidth(), true);
    }

    if(inputImage == null) {
        return;
    }

    Task<List<Face>> result =
            this.faceDetector.process(inputImage)
                    // When detection succeeded
                    .addOnSuccessListener(
                            faces -> {
                                overlay.clear();
                                if(faces.size() > 0) {
                                    for(Face face: faces) {
                                        if(drawFaces) {
                                            // Adding faces to overlay
                                            FaceGraphic g = new FaceGraphic(overlay, face);
                                            overlay.add(g);
                                        }
                                        // Update Pointer/Cursor
                                        pointer.update(face);
                                    }
                                }
                            })
                    // When detection failed
                    .addOnFailureListener(
                            e -> Log.d("Processing failed", e.getMessage()))
                    // When detection is completed
                    .addOnCompleteListener(
                            task -> image.close()
                    );

}

@SuppressLint("UnsafeOptInUsageError")
private InputImage getInputImage(ImageProxy image) {
    int rotation = image.getImageInfo().getRotationDegrees();
    Image mediaImage = image.getImage();

    if(mediaImage != null) {
        return InputImage.fromMediaImage(mediaImage, rotation);
    }else{
        return null;
    }
}

GameActivity:

public class GameActivity extends AppCompatActivity {

private GraphicOverlay mOverlay;
private GameView gameView;
private Camera camera;
private Pointer pointer;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
            WindowManager.LayoutParams.FLAG_FULLSCREEN);
    this.requestWindowFeature(Window.FEATURE_NO_TITLE);

    setContentView(R.layout.activity_game);

    gameView = findViewById(R.id.gameView);
    mOverlay = findViewById(overlay_game);

    initCamera();
}


private void initCamera() {
    camera = new Camera(this, mOverlay);
    camera.setDrawFaces(false);
    camera.setAnalysisUseCase();
    camera.setAnalyzer();
    camera.build();

    pointer = camera.getDetector().getPointer();
    gameView.setPointer(pointer);
}

@Override
protected void onResume() {
    super.onResume();
}

@Override
protected void onPause() {
    super.onPause();
}

The face detection "crash" can occur anywhere from 1 second to 5 minutes playing the game. I would gladly appreciate any hints or suggestions. Also im fairly new to java and mobile-app development, so please excuse my bad code.

If you need any further information/code i'll happily provide it.

Thanks in advance