Repository navigation
Can't summon soft_input with egui + winit + wgpu #44
Description
Activity
Hi,
Good Android IME support is something that still needs work in Egui, Winit and android-activity unfortunately but is something I'd like to make some progress on.
You can see an initial attempt I made at enabling input method support in
android-activityhere: https://github.com/rib/android-activity/pull/24 but I hit some odd input issues at the time which I didn't have a chance to debug.More recently I also looked at creating initial branches for Winit + Egui to experiment with this too:
https://github.com/rib/winit/tree/android-activity-ime-events
https://github.com/rib/egui/tree/android-winit-ime-supportI would probably recommend first looking at the
GameActivitybackend if you are looking to enable soft keyboard support since we should be able to utilize theGameTextbindings in that backend to also support the InputMethod protocol.Maybe you could take a look over some of the above branches to see if they are a helpful starting point.
At least in terms of using JNI to access the
Activityyou can use thendk_contextcrate to get to theActivity, e.g. something like:let ctx = ndk_context::android_context(); let jvm_ptr = ctx.vm(); let jvm = unsafe { jni::JavaVM::from_raw(jvm_ptr.cast()).expect("Expected to find JVM via ndk_context crate") }; let env = jvm.attach_current_thread_permanently().unwrap(); let activity_ptr = ctx.context(); let activity = unsafe { jni::objects::JObject::from_raw(activity_ptr as jni::sys::jobject) };
ndk_contextprovides a general (also usable for native Rust libraries that don't know whether they are part of a standalone Rust application) way to access aJavaVMon Android and anActivity/Contextreference.android-activityinitializes the state forndk_contextbefore yourandroid_main()runs.There is an implement in miniquad/miniquad, use a normal Activity.
not-fl3/macroquad#526
https://github.com/not-fl3/miniquad/search?q=showKeyboard@tkkcc if you just want to show the keyboard you should be able to use https://docs.rs/android-activity/0.4.0/android_activity/struct.AndroidApp.html#method.show_soft_input which should be equivalent to what macroquad does.
...but, unfortunately, that won't be very useful without also having input method support too.
macroquad has a custom subclass of the
Activityclass which effectively lets you call theInputMethodManager::showSoftInput(https://developer.android.com/reference/android/view/inputmethod/InputMethodManager#showSoftInput(android.view.View,%20int)) API via JNIIn
android-activitytheshow_input_methodAPI should do be equivalent to what macroquad does:When android-activity is built for
NativeActivityIt calls an NDK API calledANativeActivity_showSoftInputwhich will end up calling an internalandroid_NativeActivity_setWindowFlags(https://github.com/aosp-mirror/platform_frameworks_base/blob/ffeb11af1c22a1b8e2af57aac338e1fb57261ec5/core/jni/android_app_NativeActivity.cpp#L202) and using some IPC to invoke this: https://github.com/aosp-mirror/platform_frameworks_base/blob/ffeb11af1c22a1b8e2af57aac338e1fb57261ec5/core/jni/android_app_NativeActivity.cpp#L249 and that will call ashowImemethod forNativeActivity.java- similar to the macroquad subclass ofActivity: https://android.googlesource.com/platform/frameworks/base.git/+/master/core/java/android/app/NativeActivity.java#329When android-activity is built for
GameActivitytheshow_soft_inputwill call an internalGameActivity_showSoftInputthat uses IPC like in NativeActivity to send aCMD_SHOW_SOFT_INPUTcommand that then callsGameTextInput_showImewhich then calls a JNI method calledsetSoftKeyboardActivethat also does the same thing as macroquad: https://android.googlesource.com/platform/frameworks/opt/gamesdk/+/refs/heads/master/game-text-input/src/main/java/com/google/androidgamesdk/gametextinput/InputConnection.java#113I've at least smoke tested that
show_soft_inputhas worked for me in the past when working with experimental branches to start enabling better IME support so I'm not currently sure why nothing was happening for @inferrna when callingshow_soft_inputvia the linked example code.Reacted by bilabilaI have a similar issue, I do use
show_soft_input(in a native activity) but nothing happens. I just get the following message:
InputMethodManager: Ignoring showSoftInput() as view=android.app.NativeActivity$NativeContentView{a4db4 V.ED..... ......ID 0,0-1080,2084} is not served.agdk-eframe demo is able to show and hide input method via button
code
use eframe::egui; use eframe::{NativeOptions, Renderer}; #[cfg(target_os = "android")] use winit::platform::android::activity::AndroidApp; struct DemoApp { app: AndroidApp, } impl eframe::App for DemoApp { fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) { egui::CentralPanel::default().show(ctx, |ui| { if ui.button("ime show").clicked() { self.app.show_soft_input(false); } if ui.button("ime hide").clicked() { self.app.hide_soft_input(false); } }); } } fn _main(mut options: NativeOptions, app: AndroidApp) { let demo_app = DemoApp { app }; eframe::run_native("My egui App", options, Box::new(|_cc| Box::new(demo_app))); } #[cfg(target_os = "android")] #[no_mangle] fn android_main(app: AndroidApp) { use winit::platform::android::EventLoopBuilderExtAndroid; android_logger::init_once(android_logger::Config::default().with_min_level(log::Level::Info)); let mut options = NativeOptions::default(); let app2 = app.clone(); options.event_loop_builder = Some(Box::new(move |builder| { builder.with_android_app(app); })); _main(options, app2); }
i don't know if clone an AndroidApp will cause something wrong. i don't know how to access options.event_loop_builder from ui update fn.
tmp.mp4
now i can achieve some level (very buggy) of input experience. based on agdk-eframe demo, picking the last commit of
- https://github.com/rib/winit/tree/android-activity-ime-events
- https://github.com/rust-mobile/android-activity/tree/ime-support
and in
game_activity/mod.rs, i put the check oftextInputStatebefore the check ofandroid_app_swap_input_buffers. because the second is null, when the first is valid.pub fn input_events<F>(&self, mut callback: F) where F: FnMut(&InputEvent) -> InputStatus, { // eprintln!("555 {}", (*app_ptr).textInputState); unsafe { let app_ptr = self.native_app.as_ptr(); if (*app_ptr).textInputState != 0 { let state = self.text_input_state(); callback(&InputEvent::TextEvent(state)); (*app_ptr).textInputState = 0; } } let buf = unsafe { let app_ptr = self.native_app.as_ptr(); let input_buffer = ffi::android_app_swap_input_buffers(app_ptr); if input_buffer.is_null() { return; } InputBuffer::from_ptr(NonNull::new_unchecked(input_buffer))
issues:
- the logic has some mistake. like the end of record, after one key, the long deleted text shows up.
- existing string should be passed to gameactivity, may be using settextinputstate.
- ime support should be transparent to lib user.
- ime language is not system default. it can only be english.
- hardward keyboard not work.
Reacted by ardocratReacted by Lucas MeurerReacted by William CasarinI'm also looking into keyboard support on android (and iOS). Based on @rib s work I have been able to get pretty decent support working on android 🥳
Screen_Recording_20230708_144153.mp4
(opening and closing the keyboard also works, even if it's not in the video)
Unfortunately for this to work, I had to add completely new events to winit, basically passing the TextInputState struct from android activity all the way through to egui, and then handling the logic there.
This is not ideal, but I'm not sure how much we could improve on this, with the way the GameTextInput works.The most difficult part was finding the right places in egui to send the correct state updates when selecting a new text field or updating the selection.
If you'd like to try this you can use my relevant branches by adding this to your Cargo.toml (your project must be using game-activity for keyboard input to work):
[patch.crates-io] winit = { git = "https://github.com/lucasmerlin/winit", branch = "v0.28.x_ime_support" } egui = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"} eframe = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"} egui-wgpu = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"} android-activity = { git = "https://github.com/lucasmerlin/android-activity", branch = "ime_support"}
These are the relevant branches:
https://github.com/lucasmerlin/winit/tree/mobile_ime_support
https://github.com/lucasmerlin/egui/tree/mobile_ime_support
https://github.com/lucasmerlin/android-activity/tree/ime_support (based on the work of @rib)For this to feel native on android what's still missing would be egui showing the composition range with underlined text instead of the blue selection highlight. But I think that should be pretty straightforward.
Also missing of course is to update the insets / resize the window when the keyboard opens.
Next, I'll look into iOS input support, I already have opening the keyboard and typing working but I'll also try to get autocomplete and suggestions.
Reacted by Piotr Podusowski, bilabila and Robert BraggReacted by ardocrat and Robert BraggReacted by William Casarin, Snail with tea and Laudashow_soft_input() is still not working in NativeActivity with winit 0.29.10 (android-activity 0.5.0), nothing happens, no error on Android 8.1 or Android 11. I call it in android_main and after a touch event.
On Android 13 I get:
InputMethodManager: Ignoring showSoftInput() as view=android.app.NativeActivity$NativeContentView{454c3ca V.ED..... ......ID 0,0-1080,2023} is not served.I suspect this has to do with using a GPU surface.
I will see if the window flags are correct (if I manage to access them).https://github.com/rib/android-activity/pull/24 is a dead link.
GameActivity examples (rust-android-examples) immediately crash on my devices (Android 8.1, Android 11 and Android 13) with various errors, so I can't consider that for production.
On Android 13 the error is undefined symbol: ANativeActivity_onCreate; this may have to do with cargo apk/xbuild, they're quite limited.I'll try to do it through JNI.
https://github.com/rib/android-activity/pull/24 is a dead link.
Since this repository was migrated/transferred to the
rust-mobileorganization (whew you created the issue), then Rib forked the repo again for personal contributions, GitHub no longer redirects. Instead, replaceribwithrust-mobileto find the right link: #24Reacted by TÖRÖK AttilaOk, thanks.
It looks like a View is needed (created in onCreate) to request focus first, as done here:
https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/app/NativeActivity.javaShouldn't ANativeActivity_onCreate simply call it's super.onCreate?
Shouldn't ANativeActivity_onCreate simply call it's super.onCreate?
Why would that be required, given that
NativeActivityexists to not have to write Java/JNI (to at least get a basic app going)?The Java layer that powers
NativeActivitydoes callsuper.onCreate:It's unclear what line your link is supposed to link to.
Ok, I see.
I managed to get GameActivity to work with soft input (but not the latest winit/wgpu versions) on Android 8.1.I guess you don't get soft input on a window surface; you'd need a view.
It might be possible to achieve this with NativeActivity if you create a SurfaceView in the same way that is done for GameActivity.
But I guess that's kind of deprecated and people will just have to use GameActivity for soft input on a GPU surface.
It works with NativeActivity if you use
getDecorView, here's an example which worked for me
https://github.com/mvvvv/StereoKit-rust/blob/master/src/tools/os_api.rs#L196
Or it may be a few lines of Java, something likepublic static void JNI_showKeyboard() { InputMethodManager im = (InputMethodManager)Main.getContext().getSystemService(Context.INPUT_METHOD_SERVICE); im.showSoftInput(window.getDecorView(), InputMethodManager.SHOW_IMPLICIT); }
Still, there's a number of issues which prevent it to be usable in current state:
- the window is not resized event with the
android:windowSoftInputMode="adjustResize"property - the keyboard doesn't react to changing locale or input methods withing one locale such as different ways to enter Japanese characters
- special characters are not handled properly, the ones which appear when you hold a key for awhile
- the window is not resized event with the
Although a few more general text input topics came up in the discussion here, I think the original issue here (
show_soft_input()wasn't working with the native-activity backend, due to issues withANativeActivity_showSoftInput) has been fixed by #178 (and tested with egui + winit)It's perhaps worth noting here that the NativeActivity backend is not expected to ever get full InputConnection / IME support and so there's quite a significant limitation on what kind of text input can be supported, but with some keyboards you can at least get physical KeyEvents for basic ascii input.
For
InputConnection/ IME support it's recommended to use thegame-activitybackend if possible.
Hi. I'm trying to display soft keyboard in simple app by calling
show_soft_input:https://github.com/inferrna/hello_world_android_egui/blob/try_keyboard/src/lib.rs#L357
but nothing happens, even logcat contains no messages related to soft input, keyboard, etc.
Googling I found a lot complains about ANativeActivity_showSoftInput()
There is another way to do it — via JNI call. But it requires access to native_activity, which is currently hidden.