Unreal Engine Plugin

Introduction

Band provides an official plugin for Unreal Engine. We support both Blueprint (visual scripting language of the UE) and C++ based interfaces.

Example

We first demonstrate how to use the plugin with a simple example. Below code snippet is part of the mobile augmented-reality classification example built with the C++ interface of the plugin.

Link to Example

C++

The main workflow of the example is as follows: Initialize the plugin and load the model.

  • Preallocate the input and output tensors.
  • Create a widget to display the camera image and the classification result.
  • Start the camera (we use our own camera plugin to access the Android camera).
// Called when the game starts or when spawned
void ADetectionActor::BeginPlay() {
  DetectorInputTensors = DetectorModel->AllocateInputTensors();
  DetectorOutputTensors = DetectorModel->AllocateOutputTensors();

  TSubclassOf<UBandUIBase> WidgetClassType = WidgetClass.LoadSynchronous();
  Widget = CreateWidget<UBandUIBase>(GetWorld(), WidgetClassType);
  Widget->AddToViewport();

  CameraImage = Cast<UImage>(Widget->GetWidgetFromName("CameraImage"));
  GetGameInstance()->GetSubsystem<UBandSubSystem>()->OnEndInvoke.AddUObject(
      this, &ADetectionActor::OnEndRequest);

  AndroidCamera->StartCamera(640, 640, 30);
  AndroidCamera->OnFrameAvailable.AddUObject(
      this, &ADetectionActor::OnFrameAvailable);
  AndroidCamera->OnTextureAvailableDynamic.AddDynamic(
      this, &ADetectionActor::OnTextureAvailable);

  Super::BeginPlay();
}

When a new frame is available, we feed the frame to the model and request the inference.

void ADetectionActor::OnFrameAvailable(const UAndroidCameraFrame *Frame) {
  DetectorInputTensors[0]->FromCameraFrame(Frame, 0.f, 1.f);
  FBandModule::Get().RequestAsync(DetectorModel, DetectorInputTensors);
}


void ADetectionActor::OnTextureAvailable(UTexture2D *Texture) {
  CameraImage->SetBrushFromTexture(Texture, true);
}

When the inference is done, we get the result and update the widget.

void ADetectionActor::OnEndRequest(int32 JobId, EBandStatus Status) {
  if (Status == EBandStatus::Ok) {
    UBandBlueprintLibrary::GetOutputs(JobId, DetectorOutputTensors);
    TArray<FBandBoundingBox> BoundingBoxes =
        UBandBlueprintLibrary::GetDetectedBoxes(
            DetectorOutputTensors, EBandDetector::SSDMNetV2, Label);
    Widget->BoundingBoxes = BoundingBoxes;
  } else {
    UE_LOG(LogTemp, Error, TEXT("Detection failed"));
  }
}
← Java